Skip to content

CUDA Managed Pointers integration for selected components - #1768

Draft
vcosmi wants to merge 8 commits into
UCL:masterfrom
vcosmi:feature-CUDA-managed-pointers
Draft

vcosmi wants to merge 8 commits into
UCL:masterfrom
vcosmi:feature-CUDA-managed-pointers

Conversation

@vcosmi

@vcosmi vcosmi commented Sep 30, 2026 •

Copy link
Copy Markdown

Changes in this pull request

This PR introduces an approach for using CUDA-Managed pointers in selected STIR components by modifying the underlying memory allocation mechanism. As a preliminary step, the work focused on image and projection data, with particular attention to the DiscretisedDensity and ProjDataInMemory classes and how memory allocation is handled within them.

The main changes are:

Implementations:

  • ProjDataInMemory.cxx: Updated buffer allocation to use CUDA unified memory via CuVec's custom allocator (CuAlloc). Added a workaround using std::swap inside create_buffer() to ensure memory addresses remain on the GPU during buffer initialization. Optimized the copy constructor and arithmetic operators to utilize cudaMemcpy and GPU-accelerated kernels now that projection data resides in managed memory.
  • VoxelsOnCartesianGrid.cxx: Added a temporary workaround to enforce CUDA unified memory allocation when instantiating grid data for projection operations.
  • DiscretisedDensity.inl: Updated image data allocation to use CuAlloc unified memory and implemented the corresponding copy constructor.
  • CudaGibbsPenalty.cuh: Updated kernel to accept pointers directly from unified memory allocations. Removed redundant host-to-device and device-to-host memory transfers.
  • algebraic_kernels.cu/h: Added definition and implementation of CUDA kernels and host wrappers for element-wise binary operations (+=, -=, *=, /=) and linear combinations (dst=a⋅x+b⋅y, via xapyb) on unified memory projection arrays, adapted from CuVec operations.

Headers:

  • DiscretisedDensity.h: Declared the CUDA-managed copy constructor added in the DiscretisedDensity class interface.

Build and testing

  • CMakeLists.txt: Added algebraic_kernels.cu to the target build sources for CUDA compilation.
  • stir_timings.cxx: Added timing coverage for copy_only and xapyb_operator benchmark groups.

Testing performed

The changes were tested using STIR utility stir_timings.

Related issues

Checklist before requesting a review

  • I have performed a self-review of my code
  • The code builds and runs on my machine

Contribution Notes

Please tick the following:

  • The content of this Pull Request (the Contribution) is intentionally submitted for inclusion in STIR (the Work) under the terms and conditions of the Apache-2.0 License.

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 227 complexity · 24 duplication

Metric Results
Complexity 227
Duplication 24

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@KrisThielemans KrisThielemans left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some hopefully helpful comments, mostly just clean-up recommendations for now.

Obviously, we'll have to guard lots of things with HAVE_CUDA

Comment thread src/buildblock/ProjDataInMemory.cxx Resolved
Comment thread src/buildblock/ProjDataInMemory.cxx Resolved
Comment thread src/buildblock/ProjDataInMemory.cxx Resolved
Comment thread src/buildblock/ProjDataInMemory.cxx Resolved
error("CudaGibbsPenalty: set_up has not been called");

d_scalar[0] = 0.0;
array_to_device(d_image_data, current_image_estimate);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you sure we can delete this? What happens if it's not on the device?

@vcosmi vcosmi Sep 30, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I checked this part, and I think the original reason for removing that transfer was related to the changes to DiscretisedDensity constructors and how they allocate memory. The idea was that the memory would now be accessible by both host and device. However, if a DiscretisedDensity object is constructed via a different route that doesn't use CuAlloc (or cudaMallocManaged) this is no longer valid.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly, array_to_device should just copy the pointer (not the data) or so when it's already on the device. not sure...

Comment thread src/include/stir/DiscretisedDensity.inl Resolved
@KrisThielemans

Copy link
Copy Markdown
Collaborator

BTW, please commit with [ci skip] in the "subject". We know this will fail, so avoid wasting resources.

@KrisThielemans KrisThielemans left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great stuff. some more "moving things" around comments such that we'll have a great starting base for other structures.

BTW, we have in various places std::allocate_shared<elemT[]>(CuAlloc<elemT>(), range_v.size_all()) which ATM we need to guard with STIR_WITH_CUDA. @casperdcl what about something like this in cuda_utilities.h

#ifdef STIR_WITH_CUDA
using device_allocator = CuAlloc;
#else
using device_allocator = std::allocator;
#endif

see also the discussion in #1687 (comment)

Comment thread src/buildblock/CMakeLists.txt Resolved
Comment thread src/buildblock/ProjDataInMemory.cxx Resolved
Comment thread src/buildblock/ProjDataInMemory.cxx Resolved
Comment thread src/buildblock/ProjDataInMemory.cxx Resolved
@KrisThielemans KrisThielemans self-assigned this Oct 1, 2026
Comment thread src/include/stir/DiscretisedDensity.inl Resolved
Comment thread src/include/stir/DiscretisedDensity.inl Resolved
Comment thread src/include/stir/DiscretisedDensity.inl Resolved
error("CudaGibbsPenalty: set_up has not been called");

d_scalar[0] = 0.0;
array_to_device(d_image_data, current_image_estimate);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly, array_to_device should just copy the pointer (not the data) or so when it's already on the device. not sure...

@KrisThielemans

Copy link
Copy Markdown
Collaborator

Also, looks like you don't run pre-commit, see https://github.com/UCL/STIR/blob/master/documentation/devel/git-hooks.md

You'll have to do pre-commit run -all to fix your files.

@KrisThielemans

Copy link
Copy Markdown
Collaborator

#1772 has been merged. Please merge master here. There will be a conflict in cuda_utilities.h

@KrisThielemans

Copy link
Copy Markdown
Collaborator

BTW, I think the merge commit can be pushed with [ci skip]. Some feedback will be useful.

@KrisThielemans

Copy link
Copy Markdown
Collaborator

feel like having a go at merging master. will help...

@vcosmi

vcosmi commented Oct 6, 2026

Copy link
Copy Markdown
Author

After merging, I had an issue with cuda_utilities: image and projection data were always being allocated on host memory rather than managed memory, even in CUDA builds. So, I guarded the definition with #ifndef STIR_WITH_CUDA, which restores the original managed memory behavior. I'm not sure if this change affects other components.

I also added two helper functions in cuda_utilities to handle instances where image data remains in host memory during Gibbs penalty calculations. This should (hopefully) address the cases we were discussing in one of the intial comments.

@KrisThielemans

Copy link
Copy Markdown
Collaborator

See #1687 (comment). Let's keep the discussion on cuvec .cxx/.cu in #1687

@KrisThielemans KrisThielemans left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking good except for some house-keeping. What's the current status?

Comment thread src/buildblock/ProjDataInMemory.cxx Outdated

if (initialise_with_0)
{
std::fill(this->buffer.begin_all(), this->buffer.end_all(), 0.F);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally replace later with some CUDA-aware function

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a function, fill_with_zero, in cuda_utilities that follows the same idea as the copy function. If the memory is on the GPU or is CUDA managed memory it uses cudaMemset, otherwise it falls back to the original CPU implementation via std::fill.

Comment thread src/include/stir/algebraic_kernels.h Outdated
@@ -0,0 +1,19 @@
#pragma once

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

amazingly this isn't portable so need to use our standard #ifdef ... #define ...

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@vcosmi

vcosmi commented Oct 22, 2026

Copy link
Copy Markdown
Author

A quick update. Since the hackathon, I haven't made much progress beside some cleanup here and there. My plan is to address the housekeeping comments and to take a look at the parts we said that should be more CUDA-aware. I'm hoping to be able to work on it next week.

@MatteoNeelColombo

Copy link
Copy Markdown
Contributor

Hi Valerio,
FYI, on WSL2 and native Windows managed memory has only limited support (concurrentManagedAccess=0). All managed memory is migrated to the GPU at every kernel launch, See the https://docs.nvidia.com/cuda/wsl-user-guide/index.html#known-limitations-for-linux-cuda-applications and the https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/unified-memory.html#unified-memory-on-windows-wsl-and-tegra. I see a big slowdown of the Cuda Gibbs priors on WSL since they moved to CuVec, and I'll open an issue on that soon. Not really sure how much do we care about performances on WSL @KrisThielemans, but i thought it was fair to point out.

@casperdcl

casperdcl commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

debugging a bit with @MatteoNeelColombo 1 (stir_timings --runs 20 --skip-BB 1 --skip-PMRT 1 --skip-PP 1 --image OSEM_image.hv --template-projdata prompts.hs2) running on my machine (CUDA 13.4 NVIDIA A100 GPU, AMD EPYC 7452 32-Core CPU):

wall time/[ms] #1629 (merged) master (7ea5c91) #1768 (this PR)
Cuda Gibbs RDP value 27.09 74.68 4.82
Cuda Gibbs RDP gradient 37.34 151.92 13.72
Cuda Gibbs RDP Hessian×input 70.06 220.43 6.96

Footnotes

  1. during our STIR UM hackathon 😜 ↩

  2. https://petric.tomography.stfc.ac.uk/2/data/files/NeuroLF_Esser_Dataset/ ↩

@MatteoNeelColombo

MatteoNeelColombo commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Here are the timings on WSL2 (RTX 4090, Ryzen 9 7900, CUDA 12.9), same command and NeuroLF Esser dataset. I counted the cudaMemcpy with nsys and the std::copy on master from the code.

per call #1629 (merged) master (243aa8e) #1768 (this PR)
Cuda Gibbs RDP value [ms] 1.75 26.11 30.04
Cuda Gibbs RDP gradient [ms] 3.35 6.80 5.77
Cuda Gibbs RDP Hessian×input [ms] 6.03 14.95 10.82
copies (value / gradient / Hessian×input) 2 / 2 / 4 cudaMemcpy 1 / 2 / 4 std::copy to managed 0 / 0 / 0

i would say that the timings in stir timings are not really designed for this PR, I think. #1629 copies the image to the GPU (and the result back) at every call, and master does the same with std::copy into managed memory. This PR instead passes the managed pointers straight to the kernels: 0 cudaMemcpy, and no allocations after set_up. As stir_timings never touches the image on the CPU between calls, it only moves to the GPU on the first call, so the numbers for this PR are basically kernel time. In a reconstruction the image changes on the CPU at every iteration.

So on WSL my best bet is that the GPU reads the managed memory over PCIe instead of moving it to the GPU, but I can't check that on WSL. On native Linux the pages stay on the GPU, which is why you see this PR much faster.

To properly understand what's going on under the hood i think we would need to profile them, e.g. with nsys profile --cuda-um-cpu-page-faults=true --cuda-um-gpu-page-faults=true. Unfortunately i cannot do that ATM since on WSL nsys only traces the CUDA API calls (no kernels, transfers or unified memory event). Could you try it on your machine @casperdcl?

@casperdcl

casperdcl commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Using https://gist.github.com/MatteoNeelColombo/f069e64fe8ac67b4ed4d9b2e67a1641d:

Cuda Gibbs RDP per call (no) touch/[ms] #1629 master #1768
value (6.97) 9.21 (79.91) 80.26 (10.47) 82.50
gradient (14.28) 17.26 (148.21) 149.18 (6.93) 155.85
Hessian×input (27.87) 26.83 (222.99) 227.03 (7.35) 229.16
UM page-faults #1629 master #1768 no touch #1768 touch
HtoD migration 0 5.98 GB 276 MB 9.08 GB
DtoH migration 0 5.96 GB 70 MB 8.92 GB
CPU page faults 0 18k 1k 28k
GPU page faults 0 418k 22.6k 498k
explicit HtoD copies 709 × ~10 MB ~203 × ~10 MB 0 0
explicit DtoH copies 505 × ~10 MB ~202 × ~15 MB 101 × 19.7 MB 101 × 19.7 MB

P.S.: might be worth testing in CI (#1764)

@casperdcl casperdcl mentioned this pull request Nov 13, 2026
@casperdcl

Copy link
Copy Markdown
Collaborator

btw I could rebase this PR (for fixes such as #1780, #1792) and push a commit to prefetch (makes the "touch" case ~15% faster).

@vcosmi

vcosmi commented Nov 13, 2026

Copy link
Copy Markdown
Author

@casperdcl Yes please, go ahead with rebase. Thanks!

I will try to go through the comments from you and @MatteoNeelColombo this weekend or beginning of next week.

@KrisThielemans

Copy link
Copy Markdown
Collaborator

if we all agree to rebase, fine, but I would minimse this as much as possible. For multi-contributor PRs, it's bound to give problems. i.e. let's rebase once, but then not anymore (unless we have to)

@MatteoNeelColombo

MatteoNeelColombo commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

@casperdcl About the prefetch: on WSL/Windows cudaMemPrefetchAsync to the GPU returns cudaErrorInvalidDevice (no concurrentManagedAccess), I just checked. The Gibbs kernels call checkCudaError() right after the launch, so that error would show up as a kernel failure. Could you only prefetch when cudaDevAttrConcurrentManagedAccess is set?

@MatteoNeelColombo

MatteoNeelColombo commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

following up on the WSL point, with a correction to my earlier comment: I wrote that on WSL/Windows managed memory is migrated to the GPU at every kernel launch (that's what the CUDA programming guide says), but my tests show it isn't. I ran some tests on WSL2 comparing plain GPU memory and managed memory.

What I see is that on WSL managed memory is never moved to the GPU: it stays in host RAM, cudaMemPrefetchAsync fails, and setting CUDA_MANAGED_FORCE_DEVICE_ALLOC=1 makes no difference. So the GPU keeps reading host RAM over PCIe at every access, and with managed memory everything runs orders of magnitude slower than with plain GPU memory, even when all the work is on the GPU.

So managed memory should work well on native Linux, but on Windows/WSL it makes the GPU unusable for reconstruction. In my opinion i see 2 possible solutions:

1.If the long-term plan is to move everything to managed memory, we might need a separate treatment for WSL/Windows; otherwise, on those systems it would be much faster not to use the GPU at all. It would also be interesting to know how many users run STIR on Windows (native or WSL), to see how much that separate treatment is worth.
2. Do we really need managed memory for what we want to achieve in the first place? even on native linux as @casperdcl showed it seems like the managed copies are slower compared to the explicit ones.

casperdcl added a commit that referenced this pull request Nov 15, 2026
- follow-up to #1772
- required by #1768
@casperdcl
casperdcl force-pushed the feature-CUDA-managed-pointers branch from 775795d to 0926665 Compare November 15, 2026 10:28
@codacy-production

codacy-production Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 88 complexity · 6 duplication

Metric Results
Complexity 88
Duplication 6

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@casperdcl

casperdcl commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Unsure I follow @MatteoNeelColombo's comments. Personally when running CuVec:tests/test_perf.py I don't see any problems on WSL2.

Anyway, just rebased over #1792 (won't rebase again unless really needed), added a explicit prefetch helper, and here are the timings:

Cuda Gibbs RDP per call (no) touch/[ms] before prefetch (c1f0964) after (0926665)
value (4.28) 90.74 (4.16) 80.48
gradient (4.73) 160.72 (4.24) 150.29
Hessian×input (5.42) 258.84 (7.72) 220.73

@casperdcl

casperdcl commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

After more digging:

  • device2host (Linux): prefetching makes managed at least as fast as explicit
  • device2host (WSL2) & host2device: managed is designed for sparse access (<~20% of array), otherwise explicit is faster

So if we're migrating to managed, we need to ensure:

  • interleaved CPU/GPU operations are sparse
  • dense operations are only on GPU (or at least not run every iteration/frequently on the CPU) - they have a 0.2GB/s transfer overhead (compared to explicit's 7GB/s)

casperdcl added a commit that referenced this pull request Nov 15, 2026
- follow-up to #1772
- required by #1768
…use CUDA unified memory

- Modified memory allocation in DiscretisedDensity and ProjDataInMemory to use cudaMallocManaged via the custom allocator CuAlloc accessed through CuVec library.

- Updated ProjDataInMemory to manage CUDA pointers and use them efficiently.

- Added copy constructor in DiscretisedDensity and forced managed memory allocation in VoxelsOnCartesianGrid.
…jection and image data relevant classes.

- Added ProjDataInMemoryCUDA (.cu/.h) implementing GPU-accelerated element-wise operations (+=, -=, *=, /=) and linear combinations (xapyb)

- Removed manual transfers between host and device in CudaGibbsPenalty and CudaRelativeDifferencePrior in favor of managed pointers

- Updated CMake build scripts to include new CUDA compilation targets

- Added timing benchmarks for copy_only and xapyb_operator in stir_timings
- Moved onGPU helper functions to cuda_utilities.h with stir::Array overload
- Renamed ProjDataInMemoryCUDA files to algebraic_kernels (.cu / .h)
- Included original code fallback in ProjDataInMemory allocation
- Removed temporary comments ("Modified version")
- CMakeLists.txt: Updated CUDA target dependencies.
- algebraic_kernels.cu: Updated CUDA kernel implementations for compatibility with managed pointers.
- ProjDataInMemory.cxx: Fixed memory allocations and pointer operations for CUDA managed buffers.
- VoxelsOnCartesianGrid.cxx: Reintroduced original CPU allocation and implemented STIR_WITH_CUDA for the GPU version.
- algebraic_kernels.h: Updated kernel declarations to be compatible with the new version of cuda_utilities.
- cuda_utilities.h: Updated CUDA memory allocation and utility wrappers.
- cuda_utilities.h: modified the include section so that CUVEC_DISABLE_CUDA is only defined when STIR has no CUDA. Added helper functions device_readable_ptr and device_writable_ptr. They pass the array's own pointer to the kernel if it is on the GPU, otherwise they fall back to array_to_device/array_to_host with a temporary CuVec.

- CudaGibbsPenalty, CudaRelativeDifferencePrior: used these helpers in the compute_* functions. Previously the kernels received raw host pointers for images not allocated with CuAlloc, causing 'illegal memory access'. This should fix this.
@casperdcl
casperdcl force-pushed the feature-CUDA-managed-pointers branch from 0926665 to df1cf5b Compare November 15, 2026 22:13
@MatteoNeelColombo

Copy link
Copy Markdown
Contributor

@casperdcl I tried df1cf5b on WSL2 and with the new concurrentManagedAccess check the Cuda Gibbs prior gradient and Hessian×input come out as all zeros. I think the rebase over #1792 dropped the lines in CudaGibbsPenalty.cuh that copy the result from tmp back to the image (if (!onGPU(x)) array_to_host(x, tmp_x).

@casperdcl
casperdcl force-pushed the feature-CUDA-managed-pointers branch from df1cf5b to 7a58506 Compare November 16, 2026 07:17
Assisted-by: Qwen3.8-27B <noreply@pi.dev>
@casperdcl
casperdcl force-pushed the feature-CUDA-managed-pointers branch from 7a58506 to 5dddfc1 Compare November 16, 2026 09:30
@MatteoNeelColombo

Copy link
Copy Markdown
Contributor

Same command as before, ms per call,(no touch / --cpu-touch 1):

  #1629 master 5dddfc1
value 1.75 / 2.05 25.8 / 35.5 44.8 / 55.2
gradient 3.43 / 4.29 6.29 / 7.33 20.5 / 33.0
Hessian×input 5.96 / 7.18 13.2 / 14.5 39.7 / 58.5

I think it's because on WSL2 the image is now in host memory, but the temporary copy made before each kernel is a CuVec, so it's managed memory again, and the kernel still reads it from host RAM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants