Load-balance, active-box windowing, block-structured AMR - #1628
Open
sbryngelson wants to merge 650 commits into
Open
Load-balance, active-box windowing, block-structured AMR#1628sbryngelson wants to merge 650 commits into
sbryngelson wants to merge 650 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in (“default-off”) family of performance/diagnostic features (load-weight and SFC partition diagnostics, weighted init-time decomposition, rank timing), plus major simulation capabilities (active-box RHS windowing and block-structured AMR) and corresponding post-processing support and documentation/validation updates.
Changes:
- Adds new runtime parameters and toolchain metadata/validation hooks for the experimental performance/AMR feature family.
- Extends the simulation code with new modules for active-box restriction, load-weight diagnostics, SFC partition reporting, rank timing, and AMR integration points (including restart/output plumbing).
- Updates post_process to read/write AMR fine-block overlays and adds/updates golden metadata plus documentation/indexing.
Reviewed changes
Copilot reviewed 82 out of 94 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/params/descriptions.py | Adds user-facing descriptions for new experimental/performance parameters. |
| toolchain/mfc/params/definitions.py | Registers new parameters (AMR, hybrid sensors, load-balance diagnostics) and target applicability. |
| toolchain/mfc/lint_docs.py | Treats new validator checks as non-physics doc checks. |
| tests/F980C769/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ECABA006/golden-metadata.txt | Adds golden metadata for active-box test coverage. |
| tests/DD4CD8F3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/CC4213FD/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BD21A5C0/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BCBA6E74/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ACE05393/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/987D9025/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/852CCB81/golden-metadata.txt | Adds golden metadata for AMR-related golden tests. |
| tests/65C375B4/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/4DADE04B/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/454C565F/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/3A474BEE/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/2FC423D3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/13945217/golden-metadata.txt | Adds golden metadata for moving-IB under AMR test coverage. |
| src/simulation/m_viscous.fpp | Clamps FD coefficient indexing to avoid ghost-region coefficient OOB in IB drag gradient evaluation. |
| src/simulation/m_time_steppers.fpp | Integrates active-box bounds into RK update loops and interleaves AMR fine-stage/subcycle operations. |
| src/simulation/m_start_up.fpp | Wires up new modules (rank timing, active-box, load-weight, SFC partition, AMR) into init/timestep/finalize and restart I/O. |
| src/simulation/m_sfc_partition.fpp | Adds analysis-only SFC tiling + weighted partition prediction and reporting. |
| src/simulation/m_rank_timing.fpp | Adds per-rank wall-time imbalance measurement helpers and reporting. |
| src/simulation/m_load_weight.fpp | Adds per-cell load-weight field construction and rank-level imbalance reporting. |
| src/simulation/m_hypoelastic.fpp | Refactors FD coefficient setup into a callable update routine (supporting AMR grid swaps). |
| src/simulation/m_global_parameters.fpp | Adds AMR working-state mirrors and slot selection helper plus defaults for new parameters. |
| src/simulation/m_data_output.fpp | Adds output/report hooks for load-weight, SFC partition, and rank-time diagnostics. |
| src/simulation/m_checker.fpp | Adds input validation/prohibits for active-box, hybrid sensors, load-balance, and AMR configurations. |
| src/simulation/m_active_box.fpp | Adds active-box initialization/growth and debug envelope checking. |
| src/simulation/m_acoustic_src.fpp | Adds AMR-aware handling of acoustic source support (bounding boxes and overlap abort). |
| src/post_process/m_start_up.fpp | Calls AMR fine-data reader and AMR overlay writer when amr is enabled. |
| src/post_process/m_global_parameters.fpp | Adds default-off amr flag for post_process overlay behavior. |
| src/post_process/m_data_output.fpp | Implements AMR fine-block overlay mesh/variables output (Silo/binary) and multimesh registration. |
| src/common/m_phase_change.fpp | Exposes per-cell Newton iteration count and threads it through relaxation to support load-weighting. |
| src/common/m_global_parameters_common.fpp | Adjusts start_idx lifecycle/allocation and makes load_weight_wrt visible to GPU macros. |
| src/common/m_derived_types.fpp | Introduces a simple t_box type used by new partitioning infrastructure. |
| src/common/m_box.fpp | Adds box/partition arithmetic helpers (equal/weighted splits, box-from-splits). |
| src/common/m_boundary_common.fpp | Skips BC buffer population during AMR fine advance to rely on coarse-driven ghost fill. |
| docs/module_categories.json | Registers new modules under documentation categories. |
| docs/documentation/readme.md | Adds AMR section link to the documentation index. |
| .typos.toml | Adds project-specific abbreviations to the spelling allowlist. |
| D = ((gs_min(lp) - 1.0_wp)*cvs(lp))/((gs_min(vp) - 1.0_wp)*cvs(vp)) | ||
|
|
||
| #ifdef MFC_SIMULATION | ||
| if (relax .and. load_weight_wrt) then |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1628 +/- ##
==========================================
+ Coverage 60.77% 61.98% +1.21%
==========================================
Files 83 93 +10
Lines 20872 25637 +4765
Branches 3101 4206 +1105
==========================================
+ Hits 12685 15892 +3207
- Misses 6121 6993 +872
- Partials 2066 2752 +686 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
Upstream latent gap found during the MHD+AMR investigation (independent of this PR): Also for the record: MHD+AMR was attempted and re-gated on measured evidence rather than assumption — the coarse/fine seam is a continuous O(1) div(B) source that cleaning spreads but cannot remove (details in the amr.md support matrix row and commit ac203b1). |
sbryngelson
changed the title
Member
Author
|
… bounded ovl storage sized to max overlap (exascale prep #3)
…lidate vs amr_block_owner (exascale prep #2)
…iles are not yet in the SFC cut; avoids a false abort on regrid-coexist)
…lance-on np=2 test (unification prep)
…s cartesian) (unification #1)
…wners in coexist (unification #2)
…fication #3) s_l0_tiles_init asserted that each tile's SFC compute owner equals its cartesian L0-storage owner, aborting with 'routed initial fill not implemented for this decomposition' otherwise, because the first fill (s_l0_copy_coarse_to_tiles) was a LOCAL L0->tile copy. Whether the two agree depends on how the cartesian split direction lines up with Morton order, so it is a property of the grid SHAPE: a 2:1 grid at np=2 splits in y and agrees, a square grid tie-breaks to x and diverges. Every existing L0 golden uses n=31 against m=63, so the whole suite sits on the passing side and CI cannot see it. Measured on a square 64x64 grid: np=2 aborted at l0_ntile>=2, np=4 aborted at every tile count. s_l0_copy_coarse_to_tiles now routes the seed when compute owner != storage owner: the L0-storage owner device-packs its chunk and sends it, the compute owner unpacks into its tile slot. Exact reverse of s_l0_scatter_tiles_to_coarse, and sound because a tile is built by subdividing ONE rank's cartesian chunk, so it never spans two L0-storage ranks. That alone segfaulted on a null address: s_l0_build_tile_slot ran inside the cartesian loop under 'if (r /= proc_rank) cycle' while the SFC cut overwrites amr_block_owner afterwards, so the compute owner had no slot to unpack into. The abort had been masking it. Slot construction reads only replicated region metadata and the global amr_g?cb, so it moves after the cut and keys on amr_block_owner. When the cut agrees with the cartesian order this allocates exactly the same set as before, just later. The compute==storage path is untouched (same call, same slot-extent arguments), so every previously working configuration is bit-identical. Validated on hpcfund (MI250X, amdflang AFAR 23.2.0, OpenMP offload): square 64x64 np=2 at l0_ntile 2 and 4 now run and are byte-identical to monolithic (restart md5 52a7aab258945073); 56 AMR goldens + 6 L0/coexist goldens pass; precheck clean. New golden 93093920 covers the square-grid np=2 case, which aborts before this change. Cray not exercised locally.
…iling, add batching design note amr.md was last updated 2026-07-04, before the l0-amr arc landed. Three claims had drifted from the code: - 'All amr_max_blocks slots are allocated at init at maximum size' (stated twice) is wrong: amr_slot_live drives lazy owned-only allocation, so a rank's fine memory is roughly 1/num_procs of the pool, not all of it. - 'Level-0 output only ... fine-resolution output is future work' is wrong: post_process overlays the refined fine blocks as separate SILO domains when amr = T in its input (default off). - The load-balance section claimed the coarse split is static after startup with no mention of l0_ntile, which did not appear anywhere on the page. In-run base-grid rebalancing exists: l0_ntile tiles the base grid into rr=1 blocks (byte-identical to untiled) and l0_rebalance_interval periodically re-cuts the SFC partition from measured per-tile cost and migrates tiles. Adds amr_block_batching.md, the design record for the per-block advance cost. Measured on MI250X with OpenMP offload via the l0_ntile sweep (fixed grid, fixed flops, byte-identity verified): cost is linear in block count at ~1.05x a full monolithic step per block, and does not amortize (still 15x at 512^2 with 16 blocks). Per-block cost is independent of block size, so it is fixed overhead, not work. That rules out the lever amr.md implies: per-slot state removes the swap traffic but not the per-block kernel launch count, and per-lane duplication multiplies the O(N^d) working set without reducing launches. The note records the state inventory, the three increments, and the measurement recipe. Also notes why this gates the coexist arc: relaxing the amr_max_level clause of the l0_ntile/amr checker gate fails on 'the nested level-2 block exceeds the per-rank scratch cap', which is the same base-subdomain solver scratch the batching work targets. Docs only; no behavior change. precheck clean.
… not work Traced the hot path: exactly one s_amr_swap_to_fine / s_amr_restore_coarse pair per block per RK stage, wrapping s_compute_rhs in s_amr_fine_stage_rhs. The RK pass and the fill phase do not swap. So a timestep costs 3*nblocks swap round-trips, each carrying s_amr_sync_grid_state_to_device plus s_amr_recompute_weno_coefs on nonuniform grids. Also records the obvious optimization that is NOT available: hoisting the restore so consecutive blocks swap fine->fine fails because m_time_steppers calls s_amr_p2p_reflux_faces and s_amr_apply_reflux between blocks inside the advance loop, both in the coarse frame. Any batching design has to move the reflux out of the per-block loop or make it frame-independent first. Docs only. precheck clean.
…erequisite) The fine stage advanced and refluxed each block in one loop: s_amr_fine_stage_advance runs with the block's grid globals swapped in, then s_amr_p2p_reflux_faces / s_amr_apply_reflux write the coarse rhs_vf at coarse indices. Interleaving the two frames forces a swap/restore round trip per block per RK stage, and it is what prevents consecutive blocks from swapping fine->fine. Phase 3 now advances every block; phase 4 refluxes the level-1 blocks afterwards in the coarse frame. Behavior-preserving: no block's advance reads rhs_vf, the regrid merge invariant keeps blocks >= buff_size apart so their c/f corrections are disjoint, and every rank still visits the same slots in the same order, preserving the collective ordering of s_amr_p2p_reflux_faces. level>=2 blocks still skip L0 reflux (they apply a state reflux into their parent after the stage loop). This does not by itself reduce the measured per-block cost - it removes the structural obstacle to doing so, which docs/documentation/amr_block_batching.md names as the prerequisite for batching the advances. Validated on hpcfund (MI250X, amdflang AFAR 23.2.0, OpenMP offload): the full 56-case AMR golden set passes, including the multi-block, multi-level, np=2 and reflux-exactness cases, plus the 6 L0/coexist goldens. precheck clean. Cray not exercised locally.
…p sites The note claimed the reflux frame was the prerequisite for hoisting the restore. That was incomplete. Traced s_amr_fine_stage_rk: it reads no grid globals itself (slot arrays plus dt and feature flags), but it calls s_amr_pressure_relax_fine, s_amr_ib_correct_fine and s_amr_update_mib_fine, and each opens its OWN s_amr_swap_to_fine / s_amr_restore_coarse pair assuming the coarse frame on entry. With the restore deferred they would swap while already swapped and trip the nested-swap assert. So there are two blockers. The reflux frame is resolved (advance and reflux loops are now split). The nested swap sites are open and are the larger of the two: three of the five swap/restore sites in m_amr.fpp nest inside the RK pass, which makes hoisting the restore a swap-CONTRACT change (make the swap re-entrant, or make those three routines no-op when already on their own slot) rather than a call-site change. Also records that any swap-contract change needs a Cray GPU run before it can be trusted, per the CCE-acc-only stale-device class in .claude/rules/common-pitfalls.md. Docs only. precheck clean.
…e' curve) Measured on hpcfund (MI250X, amdflang, OpenMP offload), 2D 256^2, static single-level AMR, 20 steps: np compact block (32x32) wide block (128x32) 1 0.1009 s (1.00x) 0.1009 s (1.00x) 2 0.0980 s (1.03x) 0.1190 s (0.85x) 4 0.0983 s (1.03x) 0.1267 s (0.80x) AMR does not strong-scale, and wide blocks anti-scale. Both cases hold exactly one fine block (a static block cannot exceed amr_maxc, so tiling never triggers), and single-owner distribution puts all its work on one rank whatever np is. The compact case is flat because extra ranks only split the coarse grid, which is not the bottleneck. The wide case is worse than flat: the block spans more ranks' coarse subdomains, so each added rank buys more coarse<->fine P2P gather/scatter with no fine parallelism to offset it. Caveat recorded in the note: this is the single-block regime and does not test multi-block distribution, which needs dynamic regrid to reach several blocks. That is the curve batching should move; these numbers are the floor it has to beat. Docs only. precheck clean.
s_amr_swap_to_fine asserted .not. amr_swapped, forbidding nesting outright. That assert is what makes hoisting the restore across blocks a contract change rather than a driver change: s_amr_fine_stage_rk calls s_amr_pressure_relax_fine, s_amr_ib_correct_fine and s_amr_update_mib_fine, and each opens its own swap/restore pair, so a deferred restore would trip it three times per block. Replaces the paired-swap logical with a depth counter. Only the outermost swap saves the coarse state into the sw_* bounce buffers (every save is now depth-guarded; the installs are not, since re-installing the same slot is idempotent), and only the matching outermost restore puts it back - an inner restore returns early, leaving the enclosing frame's slot installed. Sound because every nested swap site swaps to the SAME slot, amr_cur. Behavior-preserving by construction: no nesting occurs today (the RK pass's helpers run after the RHS pass has already restored), so the depth never exceeds 1 and every path is exactly as before. The driver is untouched. What this buys is that hoisting the restore is now a change to m_time_steppers alone. Validated on hpcfund (MI250X, amdflang AFAR 23.2.0, OpenMP offload): 56 AMR goldens and 5 L0/coexist goldens pass, precheck clean. Cray not exercised locally - per .claude/rules/common-pitfalls.md the stale-device class around this contract is CCE-acc-specific, so the Frontier lanes are the real gate.
…-and-seed in one routine Correcting an overclaim: with the swap now re-entrant I said hoisting the restore was a driver-level edit. It is not. s_amr_swap_to_fine ends with 'if (igr) call s_amr_igr_swap_sigma()', which both SAVES the coarse jac/jac_old into sw_jac/sw_jac_old and SEEDS the block's sigma from the parent, reading sw_idwbuff for the coarse extent. Once nesting is actually used an inner swap re-runs it and overwrites sw_jac with fine state, so it must be split into a depth-guarded save and a per-block seed first. Marks prerequisite 2 (re-entrant swap) resolved and records this as prerequisite 3. Also records that this routine already caused an OpenACC-only crash - its own comment notes the sw_* host-only state makes OpenACC's present lookup fail while OpenMP's implicit map(to) tolerates it - as a worked example of why an OpenMP-offload pass does not validate this area. Docs only. precheck clean.
…te 3 of 3) s_amr_igr_swap_sigma both saved the coarse jac/jac_old into sw_jac/sw_jac_old and seeded the block's sigma from the parent. Under nesting an inner swap would re-run the save and put FINE state into sw_jac, which both the seed and s_amr_igr_restore_sigma then work from. The save loop is now depth-guarded; the seed loop is not. The seed reads sw_jac, which still holds the coarse state, so every nested block seeds from the correct parent - guarding only the save is both necessary and sufficient. Behavior-preserving today: no nesting occurs, so the depth is always 1 and the path is exactly as before. This was the last of the three obstacles to hoisting the restore across blocks. With the reflux split (advance and reflux loops separated) and the re-entrant depth-counted swap already in, hoisting is now a change to m_time_steppers alone: open a swap frame across the phase-3 loop, let each block's inner swap install its own slot, restore once after. The swap's recomputes need no guard - they are idempotent for whichever grid is installed. Validated on hpcfund (MI250X, amdflang AFAR 23.2.0, OpenMP offload): the two IGR+AMR goldens that exercise this path (6C20B752, 660FFBFE) pass, as does the full 56-case AMR set. precheck clean. Cray not exercised locally, and this routine has a documented OpenACC-only failure history, so the Frontier acc lane is the real gate.
Follow-up to 2bffefd, whose doc hunk did not apply. Records that the IGR sigma save is now depth-guarded while the seed is not, and that guarding only the save is necessary and sufficient because the seed reads sw_jac which still holds the coarse state. All three prerequisites are now resolved, so the note states plainly what is left: hoisting the restore is a change to m_time_steppers alone - open a swap frame across the phase-3 loop, let each block's inner swap install its own slot, restore once after. The swap's recomputes need no guard, being idempotent for whichever grid is installed. Docs only. precheck clean.
…ffect Built the hoist and measured it: added s_amr_swap_hold (save the coarse state without installing a slot) and bracketed the per-tile RHS loop with it, so consecutive tiles ran fine->fine with one coarse restore after the loop instead of one per tile. Output byte-identical. l0_ntile sweep, 2D 256^2, np=1, with vs without: 1.00/1.01/4.32/16.87 vs 1.00/1.01/4.40/16.75. No effect. The coarse round trip between blocks - its sw_* copies, its device sync, its coefficient rebuild - is not a measurable share of the per-block cost even at 16 blocks where the total penalty is ~17x. Reverted rather than carried as unused machinery. This sharpens the note's conclusion: the per-block cost really is kernel launch count inside s_compute_rhs, not swap traffic, so reducing swap traffic in any form should be expected to do nothing on its own. Only the batched-kernel increment addresses the measured cost. Recorded so the hoist is not retried. Docs only. precheck clean.
Widen the dual-pass/nc_iface allocations in m_riemann_solvers and the UVM host-pinned IGR fallbacks to the _alloc grid maxima: under AMR's fine-grid swap, m/n/p are one block's dims at allocation time, so sizing on them is an out-of-bounds write for a later, larger block. Make s_initialize_phasechange_module take its allocation upper bound as an argument, removing a stage ifdef from src/common. Refresh num_gps on the device after the moving-IB ghost-point recompute; skip level-0 tiles in the acoustic-source overlap check; use 64-bit arithmetic in the regrid clustering volume and index computations. New prohibits (Fortran checker + case_validator mirrors): AMR with characteristic (CBC) boundaries - s_cbc is gated only on the global bc codes with no amr_in_fine_advance guard, so it fires at interior fine-block edges; active_box with synthetic turbulence; HLLD hypoelasticity with CBC (the dual pass reads the pre-CBC Riemann fluxes). The AMR axisymmetric test used bc_y%end = -6 against its own closed-box conservation claim; switch it to reflective and regenerate the golden. The old golden encoded the interior-edge CBC pollution: a localized density/momentum error and an unphysical alpha > 1 at the static fine block's upper y-edge.
…; the AMR axisymmetric golden had blessed CBC pollution at an interior fine-block edge
s_compute_hypoelastic_interface_energy joint-gated the LEFT state's elastic energy on both moduli and both damage variables, while the RIGHT state had no damage cutoff at all. A damage-collapsed or fluid right state therefore suppressed the left side's stored energy, and a collapsed right modulus kept contributing energy through the reciprocal the cutoff exists to bound. Give each side its own gate: E_L on G_L and damage_L, E_R on G_R and damage_R. Regenerates the AMR 1D hypoelastic static block cont_damage golden (D731AB7A), the one test that exercises the changed regime. The deltas concentrate in the stress and damage fields near the damaged region (worst relative 2.3e-3 on the stress trace, densities at O(1e-6)); every other hypoelasticity and AMR golden is unchanged.
The prohibit added in c5b461f blocked an upstream test: the CBC uniform preservation case from the dual-pass HLLD work, whose own comment records that a pre-fix version of it drifted the volume fraction by O(1e-3) within five steps. That history proves the characteristic treatment is live and consequential under HLLD, contradicting the reading that its corrections are silently dropped. The reading was half right. The conservative rows read flux_n, which the dual pass finalizes before s_cbc corrects flux_rsx_vf, but the advection source consumes the corrected flux_src_rsx_vf, and the combination is deliberately supported and tested upstream with documented usage limits. Outlawing a shipped configuration on a partial static reading does not belong in this branch; the conservative-flux question belongs upstream.
…tracted as wrong; two convergence failures were -j cross-contamination
…mplete; the remaining work is correctness and merge hygiene
Three #ifdef MFC_SIMULATION blocks remained in src/common, against the invariant that stage-varying behavior is passed in rather than compiled out. They existed because their symbols were not declared in every target: load_weight_wrt and sfc_partition_wrt were registered for simulation and post_process only, while src/common/m_phase_change compiles into pre_process as well, and amr_in_fine_advance was declared in the simulation. Register the two writer flags for all three targets and give them their default in s_assign_common_defaults; declare amr_in_fine_advance in m_global_parameters_common, where it is .false. for pre_process and post_process. The guards then delete. Behavior is unchanged everywhere: the flags default false, so count_pc_iters is false exactly as the removed #else branch set it, and the boundary early-return never fires outside the AMR fine advance. Also fixes a latent gap: post_process declared both flags with no initializer and assigned no default, so they were defined only when the namelist happened to supply them.
…0 resolves to delete, and Phase 2b is not the next increment
…s 2.00x law measured independently The genuine W4 remedy is S3 local clustering with boundary reconciliation, already scheduled in endstate Phase 3: the union_gtag ALLGATHERV exists to make the bisection rank-invariant, so a neighbor-scoped exchange breaks that property rather than satisfying the invariant.
Upstream 55fb1b1 migrated input-only checks out of m_checker into case_validator and added a lint rule to enforce the split. Our merge resolved the conflicted region as ours, restoring 19 of them, and then allowlisted s_check_inputs so the new rule could not fire. Delete the 19, and with them the now-unused m_helper_basic import and the muscl_order_first_order import that had already been dead. One of the 19 was doing real work the validator did not replicate: the Fortran pign check fires on the dflt_real sentinel while the validator fired only on an absent key, so a case writing rburn%pign explicitly as the sentinel passed validation and then ignited everywhere from t = 0. The validator now rejects any non-positive pign, matching the pattern rburn%k and rburn%pref already use, so it is strictly stronger than either previous check. That fix lands before the Fortran check is removed. The allowlist entry stays, with accurate wording: removing it exposes 57 prohibits in s_check_inputs, only 4 of which read runtime state. The other 53 are the AMR/active_box block, mirrored in Fortran and the validator on purpose so a misconfigured run fails closed at startup as well as at validation. Migrating those drops the runtime guard, which is a behavioural change and is tracked separately rather than smuggled into a cleanup.
s_amr_assign_block_owners declared seven automatic arrays sized amr_num_blocks (about 48 bytes per block between them) and s_amr_regrid five more sized amr_max_blocks. At a million blocks that is tens of megabytes of stack per call, which overflows a default 8 MB stack well before the box count itself becomes the bottleneck - a crash rather than a slowdown, and one that arrives long before exascale. Heap-allocate all twelve. The regrid routine has three early returns; unsaved local allocatables are deallocated automatically on every return path, so no explicit cleanup is needed there. Gates: S0 np8 bitcmp byte-identical against the last accepted binary, AMR suite 10/10, precheck clean.
s_check_inputs held 57 @:PROHIBIT statements for the AMR, active_box, load_balance and sfc_partition features, mirrored in case_validator. Two copies of one constraint drift: this branch already shipped a reactive_burn pair where the Fortran tested a sentinel and the Python tested for an absent key, so a case setting rburn%pign to the sentinel passed validation and then ignited everywhere from t = 0. Express each constraint once, in Python, and keep in Fortran only what needs state the validator cannot see. 53 prohibits are deleted. The four that survive all read num_procs, which is MPI_Comm_size rather than a case-file value, and each now carries a '! lint: runtime-check' marker. With those the only ones left, s_check_inputs comes off RUNTIME_CHECKER_SUBROUTINES, so upstream's check_checker_input_constraints rule is armed here for the first time and a new input-only check added to the Fortran checker will be flagged. Nine validator checks had to be corrected first, because deleting their Fortran counterparts would otherwise have opened silent holes. cfl_dt and bodyForces are derived in s_read_input_file (from cfl_adap_dt/cfl_const_dt and from bf_spatial_support) but the validator read only the literal keys, so amr_subcycle with cfl_const_dt passed validation and aborted in the solver. time_stepper and model_eqns default to the dflt_int sentinel, which the Fortran rejects and the validator skipped when the key was absent. amr_tag_eps and amr_buf were over-strict in the opposite direction, erroring on unset keys whose Fortran defaults are valid. This trades a guarantee: the validator runs on the ./mfc.sh run path, so invoking a built binary directly against a hand-edited simulation.inp no longer gets these checks. It also removes undefined behaviour - s_check_inputs read num_dims, which is not assigned until s_initialize_parallel_io_common runs 25 lines later, and then sliced amr_block_beg(1:num_dims) with it.
s_amr_build_seam_pairs ran two nested loops over amr_num_blocks per regrid, once to count and once to fill, so the cost was 2*N^2 in the GLOBAL block count. At a million blocks that is 1e12 predicate evaluations per regrid. The sort in s_amr_sfc_cut carries a comment making exactly this argument about its own O(n^2) predecessor; the scan next door was never converted. It was never a search. f_amr_seam requires exact equality of region lo and hi on both transverse dims plus lo(d, yb) = hi(d, xb) + 1, and blocks are disjoint, so (level, lo) names a block uniquely and those conditions FIX the neighbour's lo corner given xb and d. Morton-sort the lo corners once with the same stable merge sort the SFC cut uses, then binary-search the single candidate per (block, dim) and verify the full predicate on it: O(N log N). Emission order is preserved exactly - xb ascending, and each block's at most three matches insertion-sorted into ascending yb - because the list drives paired MPI_SENDRECVs in s_amr_fine_fine_halo and a reordered list mismatches sends to receives rather than producing wrong numbers. The equal-key run is scanned and the full lo vector rechecked, since f_morton keeps 21 bits per dim and truncates above 2**21 cells. The l0 periodic-wrap branch is kept as a second probe at lo(d) = 0. Gates: S0 np8 bitcmp byte-identical; 17 AMR tests covering tiled seams, tiled L2, multi-block, three-level, np=2 multi-level and the coexist set that exercises the periodic branch; precheck clean.
…ve D-phase2 and D-l0 Also stops the endstate invariant table being trusted as live state: its 'today' column was a week stale when re-audited, so it now points at amr-bench/invariant_scorecard.py, which computes the same quantities from the counters the code already emits. A scorecard you have to remember to consult is one that stops being consulted - which is how the program spent five days on a wall-clock metric that is structurally blind to W4.
…epth assumption is unverified Negative-testing the migrated checks confirms they still reject bad cases, including the amr_subcycle with cfl_const_dt combination the old validator let through. Adds the standing rule that deleting a check requires a negative test, since a suite proving only that good cases pass cannot tell a working check from a deleted one. Re-orders S3 so the first increment is a measurement, not code: Berger-Rigoutsos splits at signature holes rather than midpoints, so the tree depth that the fusion design depends on is not balanced by construction and has never been measured.
…/S3.0b) S3's fusion design claims O(log nboxes) fused collectives per regrid, which needs a shallow clustering tree. BR splits at signature holes and Laplacian inflections rather than midpoints, so the tree is not balanced by construction and its depth has never been measured. Report it: carry a depth per stack entry in s_amr_cluster and emit [amr-tree] beside [amr-scale]. Two independent maxima are kept -- the deepest call with its leaf count, and the largest call with its depth -- because one running max with a tie-break would pair a deep tiny subtree with the wrong denominator and could fake either verdict. rbytes prices what the distributed form would ALLREDUCE, one fused reduction per node covering every splittable axis, sized on the untrimmed box; it is a lower bound, short by at most 3*num_dims integers per node because the trim also needs axes too thin to split. Named amr_cl_ rather than amr_br_: that prefix already means the batched bridge in this module. No behaviour change.
ffmt detects structure it cannot match and reports it as a warning on stderr while still exiting 0, and format.sh discarded that stderr. The combination meant malformed Fortran formatted clean and linted clean, so the compiler was the first thing that could say no: an unbalanced end do / end block passed both and only failed an 18-minute GPU build. Capture ffmt's stderr, always show it, and treat unmatched structure as fatal. Verified both directions: it fires on a reconstructed broken file and leaves all 303 clean files passing, so it cannot trip on well-formed code.
The level-1 tag ALLGATHERV is W4's largest term: every rank received the GLOBAL tagged-cell list, 92.77 MB per rank per regrid at np8 and doubling with every np-doubling. Delete it. s_amr_union_gtag is replaced by s_amr_local_tags, where each rank scans only its own disjoint interior, and s_amr_cluster gains a reduce mode that ALLREDUCEs one fused per-axis signature per tree node. Integer SUM is exact and order-independent, so every rank obtains a bit-identical signature and makes bit-identical trim, accept and split decisions. One reduction per node suffices for all three uses, which is why this is not three collectives: s_amr_trim_box took the per-axis MIN/MAX of the contained tags, and that IS the first and last nonzero of the signature; its emptiness test IS sum > 0; and s_amr_find_split only ever touched the tag list to build the signature. So s_amr_box_sig / s_amr_trim_from_sig / s_amr_find_split_sig replace the two tag-rescanning routines, and the node makes one pass where it previously made up to num_dims + 1. Sizing the signature on the UNTRIMMED box is what lets the same buffer serve the trim, since trim only shrinks to the tags' own bbox and no tag leaves the box. Per-axis reductions would be WRONG: find_split returns early on the first axis yielding a zero-run split, so a per-axis collective would make the collective COUNT data-dependent and therefore rank-divergent. A rank with no local tags must still walk the tree and enter every reduction, contributing zeros; the early return is now gated on .not. reduce, because returning early would deadlock the ranks that do have tags. Gates: restart output byte-identical at np8, the clustering tree reproduced exactly (lmax 8192, ldepth 31, nodes 66354 in both arms), ntag_bytes 185241240 -> 0, and 69/69 AMR goldens, which cover the 1D/2D, np=1 and multi-level paths a single np8 probe cannot. Adds a ppn=2 AMR+IB case, the suite's only distributed AMR+IB coverage. Measured after: the reduction grows 1.66x per np-doubling against the gather's 2.00x, so W4 is improved but NOT met; S3.2 and S3.3 remain.
… status, increment status The planning docs had drifted from the code in ways that would misdirect the next reader. amr_action_plan.md gains ledger (38) and leads with a CURRENT STATE block so live status precedes 2400 lines of history. amr_endstate.md's invariant table had a today column describing the collective S3.1 deleted; all seven rows are rewritten against verified numbers. amr_plan_based_exchange.md gains a STATUS block: landed I0/I1a/I1b/I2a/I3/I4a/I4b/I5, outstanding I2b/I5b/I6/I7/I8. Two corrections worth stating rather than silently fixing. W5 is not one increment: 19 of 41 AMR p2p call sites still tag per box, F1 retains an unconverted path passing amr_cur as the tag, and the subcycle sites are an explicit deferral to I8 -- so the ~28k-rank tag wall does not clear until I8. And m_amr.fpp claimed that term leaves with I7, which contradicts I7's own boundary that any family left per-box keeps its tables; the comment is corrected and the contradiction recorded. Also W1-W7 -> W1-W8, and W2 marked flat in P and therefore off the scaling critical path.
Lines of Code
|
…ee is rank-local (S3.2a) B0: amr_blocking_factor raises min_child from the algorithmic floor of 2, which is what stops the bisection over-generating. With the floor at 2 and amr_cluster_eff = 0.9 the recursion never converges on its own -- it splits until amr_max_blocks stops it (warning on every regrid at five different caps) and the min-separation merge then collapses the result back. Measured at np8, per regrid: bf=1 gives 38,255 nodes / 594 boxes; bf=8 gives 6,475 / 576; bf=16 gives 1,466 / 576. So 26x fewer tree nodes produce the SAME answer. Default bf=1 is exact identity, min_child = max(2,1) = 2. Deliberately NOT AMReX's blocking factor. AMReX coarsens the TAG LATTICE, which also shrinks its global tag gather; S3.1 already deleted that gather here, and coarsening a rank-local sparse list cannot dedup coarse cells straddling a rank boundary without an extra exchange -- lower-corner ownership would DROP tags rather than dedup them. A size floor is simpler and is the part that actually stops over-generation. S3.2a instrumentation: per tree node, does its box fit inside ONE rank's subdomain? If so that rank holds every tag in the subtree and the subtree needs no communication -- the property S3.2's design rests on. Counted separately for the reducing level-1 path ([amr-scope-r]), because the level-2 forest clusters inside a parent window and is rank-local by construction, which would inflate the fraction. rbytes is split the same way: rbytes_now counts only nodes that actually reduce, rbytes projects the finished design. Result at np8: 7 of 16,383 level-1 nodes per regrid are shared, 99.96 percent rank-local, shr_maxdep 3 = log2(P). It holds at bf=8 (99.7 percent), so it is not a saturation artifact. np16 gives shared 11, shr_maxdep 5, and shared BYTES growing 1.86x per doubling -- which refutes doing the shared phase as plain allreduces over COMM_WORLD and keeps the sparse per-depth scoping in the S3.2 design. Gated by bit-identity: at bf=1 the restart output is byte-identical to the S3.1 binary, the counters being read-only. The 69-test AMR suite passed on S3.1 and was re-running on this build when the node allocation expired; re-run it before relying on this beyond bf=1.
… retract the P^0.73 law Three corrections the plan needed. (1) The two-point 'rbytes ~ P^0.73' law is RETRACTED: the np8/16/32 ladder shows the forest term growing 1.97x/1.99x = exactly P while the cap-fixed level-1 term stays constant, so the exponent climbs 0.73 -> 0.87 toward 1.0. S3.1 is asymptotically O(P), a ~27x COEFFICIENT cut, and W4 is not met. Published on two points, refuted by the third. (2) S3.2's design contract, recorded before any of it is written, including the negative result that depth-fusion ALONE is still O(P) per rank: an allreduce delivers the whole buffer to every rank, so fusing a depth's nodes fixes latency and leaves the slope. The mechanism that works is a sparse per-depth exchange over the shallow shared phase, with the deep phase rank-local and free, and no MPI_Comm_split anywhere. (3) B1, a prerequisite found by reading every use of nacc: the min-separation merge consumes the accepted boxes IN INSERTION ORDER, fusing the first too-close pair and removing by swapping with the last element. So the merged box set is a function of traversal order, and S3.2 -- which completes local subtrees in parallel -- cannot be bit-identical while that holds. B1 canonicalises the merge input by a deterministic key. It is the ONLY planned change that moves goldens, so it lands alone; folding it into S3.2 would confound 'did the scoping break something' with 'did canonicalisation move the boxes'.
… that decides S3.2 The suite finished after the commit that flagged it as pending: 69 passed, 0 failed. Records B0's pricing (26x fewer tree nodes for the same box set, saturation gone at bf >= 4) and the finding that B0 is NOT a W4 lever -- it prunes leaves, which are already rank-local and free, leaving the shared set untouched. Also records why big-machine allocation should be held: the need for the scoping technique is established here by mechanism (the forest term grows exactly with P while the cap-fixed level-1 term is constant), so Frontier would refine where S3.1 breaks rather than whether. The discriminating question is whether shr_maxdep grows +1 or +2 per doubling, and np32 answers it on this machine.
…trumentation The min-separation merge scanned in list order and fused the FIRST too-close pair, so its output was a function of the order boxes were ACCEPTED -- the traversal order. Sort the accepted boxes by Morton of lo (stable insertion sort; accepted boxes are disjoint so their lo corners are distinct) before the merge, and remove a fused box by shifting down rather than swapping with the last entry. The merge now depends only on the box SET, which is the precondition for any scheme where ranks produce boxes in a different order -- both the scoped level-1 clusterer and the level-2 ownership split close with a gather whose arrival order is not deterministic. Gate: 69/69 AMR goldens pass, bit-identical -- no golden regeneration. Note this means the suite does NOT exercise the property: the order-dependence needs ambiguous chains of too-close boxes, which need many boxes (tests carry a handful, the benchmark 594-2405). The evidence B1 is needed is the reading of nacc's three uses, not the suite. Also adds amr_cl_me_nodes_r / amr_cl_me_rb_r ([amr-scope-me]): the shared clustering nodes whose box actually reaches into THIS rank's subdomain, max-reduced across ranks. Print-only and golden-neutral; it prices the sparse per-depth exchange against the allreduce form.
…t' was a cap artifact np32 closes the three-point sweep. gwin_bytes (the level>=2 window ALLGATHERV) is 360/719/1440 MB per rank at np8/16/32 -- ratios 1.997 and 2.001, exactly O(P), and larger than the 185 MB gather S3.1 deleted. Level-1's shared set grows only as O(log P) (7/11/15 nodes per regrid). Records the S3.3 design contract (ownership + Alltoallv + one box gather) and reorders S3.3 ahead of S3.2b. CORRECTION, prompted by auditing the input decks: the decks are correct (period-1 blob lattice, 8/16/32 blobs, constant 200 cells per unit, nboxes 594/1206/2405, restart files 6 vars x cells x 8 B) but amr_max_blocks = 8192 was FIXED across rungs and SATURATED in all three (lmax exactly 8192, capped warning 10/10, nodes = 2*8192-1). So 'level-1 collectives are flat in P' measured the CAP, not the clusterer. Confirmed directly: at bf=4 np8 gives 13,825 nodes and 0 capped warnings, np16 re-saturates at 16,383 with 5. gwin's O(P) survives (it is driven by tagged cells, ~40 MB/regrid at both bf=1 and bf=4); the claim that level 1 is asymptotically harmless does not. The ordering is marked provisional pending the uncapped arm. Also records B0b (amr_blocking_factor default 1 -> 4, because the cap is load-bearing at the shipped default and a private subtree cannot see global nacc), and S3.3c: S3.3a/b do NOT close W1 -- pass 1 still allocates a dense window per parent on every rank and runs an O(parents x global blocks) = O(P^2) scan.
…oint-to-point Every rank was clustering EVERY parent's nesting window. Pass 1 gathered all ranks' tagged cells with one MPI_ALLGATHERV per level -- gwin_bytes, measured at 360/719/1440 MB per rank at np8/16/32, ratios 1.997 and 2.001, i.e. exactly O(P) and larger than the 185 MB tag gather S3.1 deleted. Pass 2 then rebuilt each parent's window with 'do i = 1, ntot_g; if (gkb(i) /= kb) cycle' INSIDE 'do kb = plo, phi', an O(parents x global tags) rescan on every rank -- W1's pass-2 term. S3.3a: assign each parent an owner, powner(kb) = mod(kb - plo, num_procs), a pure function of kb so every rank agrees without communicating; pass 2 processes only its own parents. The global 'nboxes + 1 > amr_max_fine' guard cannot be evaluated per rank once children are produced in parallel, so children are emitted uncapped into a local list and replayed after ONE ALLGATHERV of the child BOXES (7 ints each, ~67 KB against the 144 MB per-cell gather). Each parent has exactly one owner emitting in order, so a stable counting sort by kb reproduces the serial append order exactly -- same box list, same truncation at amr_max_fine, same box_level. S3.3b: bucket the packed pairs by powner(skb(i)) and replace the ALLGATHERV with MPI_ALLTOALL + MPI_ALLTOALLV. Send volume becomes O(this rank's tagged cells), receive volume O(its assigned parents' tags). Pass 2's residual per-parent scan becomes O(parents/P x tags/P). Safe because pass 2 rebuilds a DENSE window and re-extracts in (k,j,i) order, so ctags does not depend on pair arrival order. Gate: 69/69 AMR goldens pass, bit-identical -- including the np=2 multi-level dynamic regrid case, which is the golden that actually exercises the cross-rank nesting path this touches. NOT closed by this: pass 1 still allocates a dense window per parent on every rank and runs 'do ob = 1, amr_num_blocks' inside 'do kb = plo, phi' (O(P^2) per rank). See the S3.3c and W1 root-cause sections in amr_action_plan.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An opt-in, default-off family of performance features and the measurement infrastructure they rest on. With all flags at their defaults the only touched production path is
s_mpi_decompose_computational_domain, refactored through the newm_boxmodule (byte-identical; covered by the existing suite).m_box(partition arithmetic),m_load_weight/load_weight_wrt(per-cell load-weight field + imbalance metric),m_sfc_partition/sfc_partition_wrt(Morton-SFC predicted-imbalance diagnostic),m_load_balance/load_balance(weighted static decomposition at init; AMR-fine-work-aware),m_rank_timing/rank_time_wrt(per-rank compute-time diagnostic).m_active_box/active_box: restricts reconstruction/Riemann/RK windows to a light-cone-grown box around non-ambient flow; strict-subset golden-tested.hybrid_wenoandhybrid_riemann(+hybrid_weno_eps,hybrid_smooth_flux): linear-optimal weights / central-or-Rusanov flux in smooth cells, full WENO/HLLC at flagged discontinuities (Jameson sensor, stencil-dilated, per-level under AMR).m_amr+m_amr_registers: two-level 2:1 refined block hierarchy; conservative restriction and conservative-linear prolongation with physics-specific closures; per-stage flux registers with Berger–Colella refluxing; Berger–Rigoutsos multi-block dynamic regrid; optional dt/2 subcycling; multi-rank (single-owner blocks assigned by Morton-SFC work balancing at each regrid, with migration; blocks may span rank seams via P2P coarse↔fine gather/scatter; same-level seam halo; distributed registers); restart (both IO modes, regridded-layout persistence); AMR-aware post-processing (fine blocks visualizable as Silo overlay domains); GPU-resident fine level on both OpenACC and OpenMP offload.Full algorithm and user documentation:
docs/documentation/amr.md(support matrix enforced at runtime by the checker — unsupported combinations abort with named messages, never silently).AMR physics support matrix (abridged; authoritative table in amr.md)
Supported and golden-tested: single- and multi-fluid (5-eq,
mpp_lim) · 6-eq with per-block pressure relaxation · viscous (refluxed) · phase change (relax) · chemistry incl. species diffusion · Euler–Euler bubbles (polytropic/non-polytropic, mono/polydisperse, QBMM incl. non-polytropic with per-blockpb/mvside-state; dynamic regrid + subcycle) · acoustic sources (coarse-grid support with regrid exclusion) · immersed boundaries (multi-body, static or prescribed-motion, incl. dynamic regrid with body-containment expansion and per-substage guards) · 2D axisymmetric (per-block WENO-coefficient recompute) · stretched grids (exact parent-bisection ghost coordinates + per-swap coefficient recompute) · hybrid WENO/Riemann sensors (per-level) · Lagrangian bubbles (cloud excluded from blocks; two-way coupling on the coarse grid; regrid clips around the moving cloud) ·active_box(blocks contained in the growing window; agrees with plain AMR to ~1e-14) · IGR (restriction-only coupling: fine sigma solve seeded/Dirichlet-bounded by the coarse solve; documented truncation-order seam, exact free-stream) · 1D MHD/RMHD (div(B)=0 by construction in 1D; HLL and HLLD, incl. relativistic).Gated with named aborts (documented rationale): surface tension (seam force imbalance is structural — three fixes attempted and diagnosed in amr.md) · 2D/3D MHD (attempted and measured: the c/f seam is a continuous O(1) div(B) source GLM cleaning cannot remove — needs constrained-transport-class B prolongation/reflux) · hyperelasticity · 3D cylindrical (global azimuthal filter) · force-driven IB (
moving_ibm=2) · STL bodies · Riemann-extrapolation BCs (bc=-4) ·amr_subcycleunder IGR · stretched grids with Lagrangian/IB-regrid (uniform-spacing index conversions).Validation evidence
Known issues (all non-gating or in progress)
continue-on-error): an intermittent post-detected NaN on the two Lagrangian+AMR goldens. Exhaustively unreproducible off GitHub's runners — the exact failing stack (NVHPC 24.3 SDK,-tp=px -Kieee, HPC-X MPI, and the CI docker image itself under apptainer) passes elsewhere, as do native/zen2 builds; 24.5+ green. Documented at the golden definitions.Review guide
The commit history is arc-ordered (active-box → load-weight → SFC → weighted decomposition → rank timing → hybrid → m_box → AMR rungs → physics envelope → CI/GPU hardening); reviewing by arc is much easier than by file. The AMR arc builds stepwise: static hierarchy → restriction/prolongation → fine advance → refluxing → regrid → subcycling → multi-rank → GPU → each physics rung with its own validation. Commit messages carry the validation evidence for their change (measured defects, golden UUIDs, repro details for CI fixes).
All parameters ship default-off with
case_validatorentries, runtime checker gates, andcase.md/amr.mddocumentation.