Skip to content

Keep CSR column ids int32 when the edge index is int32 - #788

Merged
dsaini2-sc merged 5 commits into
mainfrom
dsaini2/int32-csr-indices-main
Oct 7, 2026
Merged

dsaini2-sc merged 5 commits into
mainfrom
dsaini2/int32-csr-indices-main

Conversation

@dsaini2-sc

Copy link
Copy Markdown
Collaborator

Summary

Since #786 the partitioners hand DistDataset an int32 edge index whenever node ids fit, but the compact CSR build (#768) still widened the column ids to int64. This PR keeps them int32, which GiGL's patched GLT samplers (#761) read directly. Peak memory for the CSR conversion drops from ~2x one int64 edge array to ~1.5x, and the stored column ids are half the size. int64 inputs are unchanged.

Approach

  • build_csr_from_coo allocates indices with the width of col; indptr stays int64.

User-facing changes

  • Topology.indices on the compact path is int32 for int32 edge indices. Graphs with edge ids, weights or features still go through GLT's init_graph and stay int64.
  • This needs the GLT patches from Apply and verify local patches on the pinned GraphLearn-Torch build #761 in the image. The base images are rebuilt and the pins bumped in this PR by the build-base-docker-images workflow.

Verification

  • csr_test, compact_topology_test (adds int32 CSR/CSC sampling parity against GLT's own Topology), glt_int32_indices_test: 64 tests pass.
  • distributed_neighborloader_test (36, one pre-existing skip), distributed_dataset_test (15), distributed_partitioner_test (34): pass, with real spawned loaders.
  • Full unit suite (1274 tests, runtime shape checks on): passes except 3 config_populator tests that need CI bucket access the dev machine lacks.
  • ruff, ty, dprint: clean.

build_csr_from_coo now returns int32 indices for int32 columns, which the
partitioners produce when node ids fit (#786) and GiGL's patched GLT
samplers read directly (#761). Peak conversion memory drops from ~2x one
int64 edge array to ~1.5x. int64 inputs are unchanged.
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 18:48:26UTC : Starting to build base images for CUDA and CPU.
This may take a while, please be patient.
The images will be pushed to the GCP Artifact Registry.
Once done, the workflow will update the gigl/dep_vars.env file with the new image names.

Comment thread gigl/utils/csr.py
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 18:58:18UTC : Built and pushed new images:

  • CUDA base image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/public-gigl/gigl-cuda-base:6caecb4ed5a703d7a30b27a37f65ae10747e23af.114.1
  • CPU base image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/public-gigl/gigl-cpu-base:6caecb4ed5a703d7a30b27a37f65ae10747e23af.114.1
  • Dataflow base image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/public-gigl/gigl-dataflow-base:6caecb4ed5a703d7a30b27a37f65ae10747e23af.114.1
  • Builder image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/gigl-base-images/gigl-builder:6caecb4ed5a703d7a30b27a37f65ae10747e23af.114.1

Updated gigl/dep_vars.env with new image names.
Updated .github/cloud_builder/run_command_on_active_checkout.yaml with new builder image.

@github-actions
github-actions Bot requested a review from nshah-sc as a code owner October 5, 2026 18:58

@zfan3-sc zfan3-sc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approve to unblock. lgtm once Kyle's comment is addressed

@dsaini2-sc
dsaini2-sc added this pull request to the merge queue Oct 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 5, 2026
@dsaini2-sc
dsaini2-sc added this pull request to the merge queue Oct 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 5, 2026
@dsaini2-sc
dsaini2-sc added this pull request to the merge queue Oct 6, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 05:04:05UTC : Starting to build base images for CUDA and CPU.
This may take a while, please be patient.
The images will be pushed to the GCP Artifact Registry.
Once done, the workflow will update the gigl/dep_vars.env file with the new image names.

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 05:12:21UTC : Built and pushed new images:

  • CUDA base image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/public-gigl/gigl-cuda-base:86b4bc2e01ded1e5181c56fb7f85015833676308.115.1
  • CPU base image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/public-gigl/gigl-cpu-base:86b4bc2e01ded1e5181c56fb7f85015833676308.115.1
  • Dataflow base image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/public-gigl/gigl-dataflow-base:86b4bc2e01ded1e5181c56fb7f85015833676308.115.1
  • Builder image: us-central1-docker.pkg.dev/external-snap-ci-github-gigl/gigl-base-images/gigl-builder:86b4bc2e01ded1e5181c56fb7f85015833676308.115.1

Updated gigl/dep_vars.env with new image names.
Updated .github/cloud_builder/run_command_on_active_checkout.yaml with new builder image.

@dsaini2-sc
dsaini2-sc added this pull request to the merge queue Oct 6, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 6, 2026
@dsaini2-sc
dsaini2-sc added this pull request to the merge queue Oct 6, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 6, 2026
@dsaini2-sc
dsaini2-sc added this pull request to the merge queue Oct 6, 2026
Merged via the queue into main with commit cab0e4a Oct 7, 2026
7 checks passed
@dsaini2-sc
dsaini2-sc deleted the dsaini2/int32-csr-indices-main branch October 7, 2026 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants