[sbryngelson]

Description

Two consistency fixes for the AMR fine-level advance, found while planning the #2117 follow-ups (#2247). Both make the fine blocks do what the coarse grid already does; level-0 results and amr = F results are unchanged.

1. Fine post-RK hooks ran pressure relaxation before the body force. The coarse stage applies the body force and then relaxes (6-equation model, m_time_steppers.fpp); the fine hooks (s_amr_batch_post_rk_hooks) did the opposite, although their comment says they follow the coarse order. The two do not commute: the body force changes momentum and total energy, and the relaxation's internal-energy correction reads both. Fix: swap the two calls.

2. Fine immersed-boundary markers lagged the body by one RK stage. s_amr_update_mib_fine (fine post-RK hook) ran inside s_amr_stage_fine, before s_propagate_immersed_boundaries moved the body to the stage's pose. So every fine block rebuilt its markers and corrected its ghost cells at the previous stage's pose, while the coarse grid used the current one. Fix: split the pose update out of s_propagate_immersed_boundaries (now s_advance_ib_pose; the marker rebuild s_update_mib stays where it was) and, under AMR only, advance the pose before the fine stage. Nothing between the two points reads the pose (forces come from the coarse RHS face sums, collisions from the unchanged coarse markers), so the coarse grid sees the same forces, poses and markers as before, bitwise. The fine containment check reads the pose and rotation matrix on the host, so the AMR branch also refreshes the rotation matrices (loop factored out of s_update_mib into s_update_ib_rotation_matrices) and syncs patch_ib to the host. With amr = F the sequence of operations is unchanged.

Also checked and not changed (no effect on results; details in #2247):

• s_compute_fd_gradient applies the domain-boundary one-sided closure at fine-block edges, but under AMR its output is read only by the cylindrical-axis stress, which AMR rejects. Before/after runs were bitwise identical at 1 and 2 ranks.
• The amr_xchg_coarse_ghosts margin uses + 1 where the quartic patch reaches 2 cells, but the fill gathers every patch cell from rank interiors point-to-point, so the flag only gates a redundant halo exchange.

Type of change

• Bug fix

Testing

Frontier, amdflang OpenMP offload (-c famd -m g), MI250X:

Check	Before (master 6e7dfd4)	After
Uniform two-fluid 6-eq state under gravity, periodic, static block (exact solution stays uniform): spread of the phase internal energies after 50 steps	1.2e-5	0 (exactly uniform)
Circle moving at Mach ~0.3 on a static block, 100 steps, vs. a uniform grid at the fine spacing (2x2-averaged), mean abs. error over the block's fluid cells: density / x-mom / y-mom / energy	5.24e-4 / 4.53e-4 / 3.83e-4 / 1.86e-3	3.60e-4 / 3.09e-4 / 2.53e-4 / 1.31e-3

Full suite on Frontier GPU (4 shards, -a): 862 passed, 3 failed before regeneration, all expected: the two moving-body AMR goldens and the new test's missing golden. The spinning-circle AMR case (7CA4FC48) is unchanged, since a circle rotating in place keeps the same markers.

Goldens regenerated: AMR -> 2D -> moving IBM circle (13945217, max change 2.4e-4) and ... -> two bodies (43AF9F25, max change 6.0e-3); both cases, the new test and 7CA4FC48 then pass. No other golden changes; level-0 and amr = F results are unchanged.

New test: AMR -> 1D -> two-fluid -> 6eq -> body force (EC935FC5).

./mfc.sh precheck passes.

This PR was drafted with Claude Code.

[github-actions]

Lines of Code

File	Lines	Diff
src/simulation/m_ibm.fpp	2867	+5
src/simulation/m_time_steppers.fpp	936	+5

Directory	Lines	Diff
simulation	39365	+10
total	59330	+10

[github-actions]

Claude Code Review

Head SHA: 0f0af60

Files changed:

• 4
• src/simulation/m_amr_advance.fpp
• src/simulation/m_ibm.fpp
• src/simulation/m_time_steppers.fpp
• toolchain/mfc/test/cases.py

Findings:

• medium src/simulation/m_ibm.fpp: The new s_update_ib_rotation_matrices was inserted between the doc comment of s_update_mib ("Resets the current indexes of immersed boundaries and replaces them after updating the position...") and s_update_mib itself. That comment now sits directly above the new routine's own !> line, so it describes the wrong subroutine. s_update_mib has no doc comment, and the new routine has two. Fix: move the new subroutine above that comment block, or move the two old comment lines back to just above impure subroutine s_update_mib. I'm confident about this; it is a comment-only problem.

Notes (no action needed; checked and consistent):

• The fine-hook reorder in m_amr_advance.fpp (body force, then 6-equation pressure relaxation) matches the coarse order in m_time_steppers.fpp (s_apply_bodyforces, then s_pressure_relaxation_procedure). The comment and the new "body force" test case agree with it.
• The AMR path now calls s_advance_ib_pose once per stage, before the fine stage. The coarse path still calls it once per stage, after the RK update, under .not. amr. s_update_mib runs in both cases, so there is no double advance. s_compute_ib_forces now runs before the coarse RK update in AMR runs. It reads the primitive state from this stage's RHS, so that appears equivalent.

[sbryngelson]

Independent GPU check: Frontier (OLCF), branch head 0f0af60

• Build: ./mfc.sh load -c famd -m g, amdflang OpenMP offload (--gpu mp), release; all 26 test-suite targets plus post_process (test --dry-run -a).
• Suite: ./mfc.sh test -v --max-attempts 3 --no-build -a -j 8 --gpu mp --shard i/4 -- -c frontier_amd, 4 shards on 1-node g1 jobs (MI250X, 8 GCDs each).
  • 862 passed, 3 failed, all expected: 13945217 and 43AF9F25 (moving-body AMR goldens; max change 2.4e-4 and 6.0e-3) and EC935FC5 (new test, no golden yet). 7CA4FC48 (spinning circle) passed unchanged.
  • Regenerated those three goldens on the same build (test --generate -o ...), then reran them plus 7CA4FC48: 4/4 passed.
• Before/after verification (master 6e7dfd4 build vs. this branch, same compiler):
  • Uniform two-fluid 6-eq state, gravity, periodic, static block, 50 steps: spread of the phase internal energies 1.2e-5 on master, 0 with the fix.
  • Circle at Mach ~0.3 on a static block, 100 steps, vs. a uniform grid at the fine spacing (2x2-averaged): mean error over the block's fluid cells drops ~30% (density 5.24e-4 to 3.60e-4, energy 1.86e-3 to 1.31e-3).
• ./mfc.sh precheck: passes.

NVHPC 26.9 (OpenACC + OpenMP builds, OpenACC suite on an A100 node) is running now; results will follow in another comment.

---

Run and posted with Claude Code.

[codecov]

Codecov Report

❌ Patch coverage is 71.42857% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.15%. Comparing base (9649d50) to head (0f0af60).
⚠️ Report is 12 commits behind head on master.

Files with missing lines	Patch %	Lines
src/simulation/m_time_steppers.fpp	71.42%	0 Missing and 2 partials ⚠️
src/simulation/m_amr_advance.fpp	0.00%	0 Missing and 1 partial ⚠️
src/simulation/m_ibm.fpp	83.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2249      +/-   ##
==========================================
+ Coverage   63.71%   65.15%   +1.44%     
==========================================
  Files          90      110      +20     
  Lines       24569    29562    +4993     
  Branches     3584     4478     +894     
==========================================
+ Hits        15653    19261    +3608     
- Misses       6135     6742     +607     
- Partials     2781     3559     +778     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sbryngelson]

Independent GPU check: NVHPC 26.9 on wingtip (NVIDIA A100)

• Commit: 0f0af606 (branch head)
• Compiler: NVHPC 26.9 (nvfortran) with the bundled MPI
• Builds: OpenACC (--gpu acc) and OpenMP offload (--gpu mp), both with -a (pre_process, simulation, post_process). Both built with no errors.
• Suite: ./mfc.sh test --max-attempts 3 --no-build -a --gpu acc -j 2 -g 2 3, on GPUs 2 and 3

Result: all tests passed. 865 passed, 0 failed, 39 skipped (904 total), in 304 minutes.

• Every AMR case passed, including:
  • the new EC935FC5 (6eq + body force);
  • the regenerated 13945217 and 43AF9F25 (moving IB). Their goldens were generated on Frontier, so they also match NVHPC.
• Two examples passed only on the second attempt: bc_patch_geometry (2B79A64D) and acoustic_support6_axisym (C0127FD0). The first attempt failed in execution, and the log does not keep the output of that attempt.
  • The two ran at the same time on the two GPUs, and wingtip had other users.
  • Neither uses AMR or touches code this PR changes. Earlier NVHPC runs on this machine did not need retries for them.
  • So I read these as transient failures, not a regression.
• The first launch was killed partway through by something outside the run. By then the OpenACC and OpenMP builds had both finished with no errors. I relaunched the remaining steps (rebuild check for OpenMP, then the OpenACC suite) on the same commit.

Run and posted with AI assistance (Claude Code).