Skip to content

perf(lcao): accelerate RI preparation and Libxc evaluation - #8193

Draft
Stardust0831 wants to merge 26 commits into
deepmodeling:developfrom
Stardust0831:experiment/lcao-hse-force-10x
Draft

Stardust0831 wants to merge 26 commits into
deepmodeling:developfrom
Stardust0831:experiment/lcao-hse-force-10x

Conversation

@Stardust0831

@Stardust0831 Stardust0831 commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Depends on #8169. The GPU path uses its LibRI/DDLA integration. The current diff includes prerequisite commits; the optimization changes can be reviewed separately.

What's changed?

Fresh LCAO HSE calculations can spend most of their initialization time rebuilding orbital products and evaluating radial integrals. This change batches GPU radial quadrature, reuses common integral terms, and corrects Libxc OpenMP scheduling that left available threads idle.

  • Evaluate the existing Simpson quadrature with FP64 GPU GEMM batches, preserving grids, weights, derivative recurrences and analytic origin values.
  • Share identical product-orbital transforms and omit their unused real-space origin derivative. Required integral gradients remain calculated.
  • Reuse radial interpolation, solid harmonics and two-center overlaps across magnetic components.
  • Distribute Libxc's existing 1024-point batches across OpenMP threads without applying a second 1024-iteration scheduling chunk.

Explicit radial GPU buffers and host staging are each bounded to 160 MiB per MPI rank and released before SCF. CUDA context and library allocations are outside this bound. Orbital-pair and radial dimensions are tiled; the reciprocal quadrature mesh is limited to 2,097,152 points. Product sharing reduces duplicate host storage; common species tables remain replicated across ranks.

The optimization adds no further INPUT option or dependency. GPU batching follows the RI backend selection from #8169. Shared integral reuse also benefits CPU execution, and the Libxc scheduling fix applies wherever these routines are used.

Linked Issue

Measurements and numerical checks: Stardust0831/abacus-develop#30. Extending full-table CPU preprocessing is outside this PR and tracked in Stardust0831/abacus-develop#31.

Unit Tests and/or Case Tests for my changes

Validation used V100 GPUs on SAI. At 1aaeb8ff4, a clean GPU executable build and the CPU/CUDA integral tests pass. Integral tests cover high angular momenta, full/partial batches, radial tile boundaries, origin/cutoff behavior, shared-product lifetime, magnetic components and finite-difference gradients. The existing Libxc test passes with one and four threads.

The main comparison (63ca5ef073 to 5156a5d4e) includes 77 fresh calculations and 16 matched comparisons: distorted Si8 at two k points, CsPbI3 at Gamma, CPU/GPU RI, displaced geometries, and one/four OpenMP threads. A further 12 calculations compare 5156a5d4e with 810d4909c, covering Si64, Cs/Pb/I and complex two-k-point Si8.

The dependency update (810d4909c to 1aaeb8ff4, based on #8169 at 3fa1f972b) is covered by six fresh calculations: distorted two-k-point Si8 with one OpenMP thread and Gamma-point CsPbI3 with four. Four paired checks compare the two GPU revisions and CPU/GPU RI within the current revision. All calculations converge; input and asset checks pass. All 18 upstream PR checks pass at 1aaeb8ff4.

All matched numerical checks pass. Maximum absolute differences:

Quantity Main changes Angular change Dependency update Acceptance limit
Energy (eV/atom) 4.002e-12 1.5e-12 7.276e-12 1e-8
Force component (eV/Angstrom) 6.0e-10 4.0e-10 1.0e-10 1e-6
Stress component (kbar) 9.7e-9 7.0e-10 7.0e-10 1e-5

Absolute energy/force finite differences still exceed 1e-4 eV/Angstrom for some inputs; the original implementation reproduces these residuals. The tests establish implementation consistency, not fully converged absolute force accuracy. Independent strain derivatives, long MD and multi-node execution remain untested.

Performance measurements cover the main optimizations (63ca5ef073 to 6283aea20). Both versions use GPU RI and cuSOLVER, one MPI rank and four OpenMP threads per GPU, and single-thread BLAS. Numerical settings and iteration counts match. Single-point and SCF-plus-force times include fresh initialization and process startup; RI initialization uses the rank-zero stage timer.

System GPUs Measurement Original version (s) Main optimizations (s) Speedup
Cs4Pb4I12 8 RI initialization, pair 1 993.36 24.99 39.75x
Cs4Pb4I12 8 RI initialization, pair 2 983.67 24.61 39.97x
Cs4Pb4I12 8 SCF + forces, pair 1 1098.82 92.61 11.87x
Cs4Pb4I12 8 SCF + forces, pair 2 1096.62 89.12 12.31x
Si128 4 Single point 230.95 148.06 1.56x
Si128 4 SCF + forces 288.25 203.03 1.42x

The eight-GPU comparisons reverse execution order within the same allocation: 64 logical CPUs corresponding to 32 physical cores, and 120 GiB host memory. Historical per-thread affinity was not recorded and the node hosted another job. These ratios measure software changes within the GPU implementation; the DDLA force contraction is unchanged.

Verification commands and additional coverage
export OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 BLIS_NUM_THREADS=1
cmake --build "$GPU_BUILD_DIR" --target MODULE_LCAO_center2orb_table MODULE_LCAO_center2orb_table_gpu MODULE_HAMILT_XCTest_VXC --parallel 12
ctest --test-dir "$GPU_BUILD_DIR" -V --output-on-failure -R '^MODULE_LCAO_center2orb_table(_gpu)?$'
cmake --build "$GPU_BUILD_DIR" --target abacus_std_gpu --parallel 12
for xc_threads in 1 4; do
  OMP_NUM_THREADS=$xc_threads ctest --test-dir "$GPU_BUILD_DIR" -V --output-on-failure -R '^MODULE_HAMILT_XCTest_VXC$'
done
  • Earlier regression coverage includes seven passing GPU RI reference cases, including spin, SOC, relaxation, HF and PBE0. The unsupported real-space EXX symmetry case retains the original rejection; its CPU reference passes.
  • GNU/Intel CPU builds pass in GitHub CI at 1aaeb8ff4. The CPU-only Si32 force comparison passed at 26d01a67d and has not been repeated after the angular change.
  • The 12 angular-change calculations pass convergence, input/assets and CPU/GPU placement checks. One CPU complex-k control takes 133/132 inner iterations near the unchanged convergence threshold and is excluded from timing comparisons.

Governance Notes

  • No documentation update required for the optimization: it changes internal evaluation and scheduling, with no additional INPUT or dependency change beyond feat(lcao): add optional LibRI GPU contractions through DDLA #8169.
  • Core impact: LCAO RI table construction and integral evaluation, plus shared Libxc scheduling. Functional definitions, screening and convergence thresholds are unchanged; FP64 operation regrouping can change rounding.
  • <memory> is required by the owning std::shared_ptr member. No governance exception is requested.
  • For the optimization diff, git diff --check 3fa1f972b..1aaeb8ff4 passes. python3 tools/03_code_analysis/agent_governance_check.py --base 3fa1f972b --head 1aaeb8ff4 --event-path "$PR_EVENT" --format text reports no errors and only the header warning explained above.
Contribution checklist
  • Read AGENTS.md and docs/developers_guide/agent_governance.md.
  • Link relevant issues and include focused test results and exact commands.
  • Describe INPUT behavior, core-module impact and validation limits.
  • Explain header dependencies; no governance exception requested.

Evaluate displacement-dependent harmonics and radial interpolation once for each Orb11 magnetic block, and reuse the vector path in Orb21 assembly. Extend scalar-reference and finite-difference coverage without changing numerical thresholds.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant