Derecho CSLAM performance issue. #827
Replies: 12 comments 25 replies
|
Actually not sure that this is the root cause - my second run with this mod hung as well. |
|
Adding environment variable MPICH_COLL_SYNC=1 seems to have solved this problem and does not seem to adversely affect performance. |
|
Performance results are in the timing table. search for derecho. |
|
At ne30, I wouldn't expect any speedup from 4096 -> 5120, really. Each rank will be 1 or 2 elements (5400 elements max), and you'll be rate-limited by the ones with 2, no? I'd be more curious if we do have ~2x performance going from 2700 -> 5400 ranks. I expect some degradation from comms, but otherwise that's more well-mapped in terms of elements/rank, I'd think. |
CAM6 (ne30 L32) was ~2k. Increasing L32->L58 doubles this to ~4k. See timing files here showing a robust 4.5k number: |
|
Thanks - that's helpful. |
|
I see several differences in the atm_in namelist between my run and the one you pointed out, would you expect any of these to affect performance? |
|
Those differences reflect our "F tunings" for the coupled eval, which don't come out of the box (yet). This shouldn't effect timings, other than the cheyenne run is solving for prognostic momentum fluxes (clubb_l_predict_upwp_vpwp=T) which would actually be more expensive. It might be useful to print out timings for all CAM processes via ./xmlchange TIMER_LEVEL=10. Although it would be better to have these same timings from a cheyenne run to compare it against. |
|
In a comparison of cheyenne and derecho runs each using 1800 tasks I see a significant difference in time for the prim_advec_tracers_fvm |
|
These results are in /glade/work/jedwards/ESMF_Profile.summary.cheyenne1800 and /glade/work/jedwards/ESMF_Profile.summary.derecho1800 I am uncommenting timers found in fvm_consistent_se_cslam.F90 and rerunning. I am also going to try adding profile_barrier=.true. |
|
Cheyenne: Derecho: |
|
I wrote a standalone program to test this /glade/work/jedwards/derecho_perf/fvmextend_standalone.F90 But it seems that the derecho run is faster than the cheyenne run - any ideas? |
|
What's the priority of this work? Can it wait for Peter? I heard the ASD simulations are to start on Monday; is that the driving factor here? If this is about the ASD runs, and since the subroutine you've identified is specific to CSLAM, maybe we should be testing the var-res configuration, which doesn't use CSLAM. You could run |
|
It can wait for Peter. Not aware of any ASD code using this compset. |
|
I am adding numbers for PFS.ne0ARCTICne30x4_ne0ARCTICne30x4_mt12.FHIST.derecho_intel to the timing table now. I think that it verifies that the performance issue we are seeing is limited to the CSLAM code. |
|
agreed, the ARCTIC grid numbers are about what I get on cheyenne. |
|
Can you point me to your code base? I would like to add more timers. It does not look like FVM:reconstruction:extend_panel1 increased timings explain the 10x ish slow down overall in fvm_resontruction. |
|
~jedwards/sandboxes/cesm2_x_alpha/components/cam |
|
Where are you getting 10x? I see 3x. |
|
Derecho: Cheyenne: |
|
In your table above (repeated here): fvm:tracers_reconstruct 1800 1800 334428 41.2533 37.6388 1714 55.1831 198 Derecho: fvm:tracers_reconstruct 1800 1800 334428 748.8997 364.5543 1795 787.3841 76 If I understand it correctly fvm:tracers_reconstruct is ~750 on Derecho and ~41 on Cheyenne which is ~18x. Also, part 1-3 timings on Cheyenne more or less add up to fvm:tracers_reconstruct (which they should) but on Derecho they don't. |
|
I've broken it down further, here the number following epi is the value of cubeboundary on the call. |
|
I added a timer to the call and return and found all of the extra time: |
|
Interesting! Do you think the problem is that "call reconstruction" is in a loop over vertical levels and inside the reconstruction subroutine there are loops over tracers (hence the tracer data is not contiguous in memory/cache)? |
|
I'm not sure if it's that or if it is just creating copies of the data structures on the subroutine boundary. |
|
In other words, would it help to change: to along with associated changes in the reconstruction subroutine (I can code that if you think that is worth pursuing - the reason for passing the entire %c and ctracer arrays it to avoid array copying but maybe that is not a performance issue or no longer the case with newer compilers ...) |
|
My instinct says that would make it worse instead of better, but it might be worth trying if you want to code it up. |
|
Holy cow I think I fixed it. I'd better call it a weekend. Here is the change that I made in fvm_consistent_se_cslam.F90: |
|
Great! Thanks! (I was about to start coding ...) Do you know what is happening? As far as I understand you switched to using a pointer ... |
|
The pointer is outside the k loop, I think it's avoiding a copy. Still need to test for correctness. |
Uh oh!
There was an error while loading. Please reload this page.
I am running PFS.ne30pg3_ne30pg3_mg17.FLTHIST_v0c.derecho_intel
and have run into a number of cases with the code hanging in bndry_mod.F90 looking at that code I see a couple of cases of neighborhood collectives when MPI_VERSION >= 3. I copied that file to SourceMods and disabled that check so that the older collective is used. That seems to be avoiding the hang.
All reactions