Skip to content

fix out-of-bounds l2 cache write in cpuinfo_arm_linux_init - #418

Open
soma0212 wants to merge 1 commit into
pytorch:mainfrom
soma0212:arm-linux-l2-cache-count
Open

soma0212 wants to merge 1 commit into
pytorch:mainfrom
soma0212:arm-linux-l2-cache-count

Conversation

@soma0212

Copy link
Copy Markdown

The L2 cache count and population passes in cpuinfo_arm_linux_init disagree on when a core gets an L2 entry:

  • counting keys off the decoded L2 size, but population overrides it with the size from sysfs (cache/index2/size)
  • a core whose built-in decode reports no L2 (ARM11, or an unrecognized ARMv6 core) but whose sysfs exposes an L2 size is skipped while counting yet still gets a slot while populating
  • l2_index then advances past l2_count, writing past the end of the calloc'd l2 array, or through a NULL l2 when the count is zero (the L2 writes are unguarded, unlike the L3 writes)

Applying the same sysfs size override in the counting pass keeps l2_count aligned with the number of entries written.

@meta-cla

meta-cla Bot commented Jul 14, 2026

Copy link
Copy Markdown

Hi @soma0212!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

@meta-cla meta-cla Bot added the cla signed label Aug 22, 2026
@meta-cla

meta-cla Bot commented Aug 22, 2026

Copy link
Copy Markdown

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

@soma0212

Copy link
Copy Markdown
Author

any update?

@Qubitium

Qubitium commented Nov 6, 2026

Copy link
Copy Markdown

Independent reproduction on an NVIDIA GB10 (aarch64, Cortex-A725), which may help get this merged:

Topology: 20 CPUs online (/sys/devices/system/cpu/online = 1-19), but /proc/cpuinfo lists only 19 processors (0-18). Per-core L2, L3 split across two clusters (0-9 / 10-19).

Symptom: import torch aborts with malloc(): corrupted top size. valgrind on the standalone cpu-info tool at the commit PyTorch currently vendors (bc3c01e) reports 4 invalid writes:

Invalid write of size 8
   at cpuinfo_arm_linux_init (src/arm/linux/init.c:898)      <- l2[l2_index] = ...
 Address 0x... is 0 bytes after a block of size 576 alloc'd
   by calloc (init.c:831)                                    <- l2 = calloc(l2_count, ...)

l2_count == 18, fill pass writes 19 entries. Instrumenting both passes shows the mismatch is exactly the one this PR describes: the core that is online but missing from /proc/cpuinfo decodes temp_l2.size == 0 (so the counting pass, gated on if (temp_l2.size != 0), skips it) while sysfs reports a per-core L2, so the fill pass (if (temp_l3.size != 0 || l2_is_per_core)) records it. On this host the decoder also reports no L3 for any core, so every L2 entry comes from the sysfs override.

With the counting pass made sysfs-aware (and the L2 writes bounds-checked like L3), valgrind reports 0 errors and the reported topology matches sysfs. Patch: ModelCloud@4c0907a

@Qubitium

Qubitium commented Nov 6, 2026

Copy link
Copy Markdown

Adding the exact environment for the reproduction above, because the trigger is a
kernel/firmware hotplug bookkeeping inconsistency rather than a plain "a CPU was
hotplugged" case:

  • OS: Ubuntu 26.04.1 LTS (Resolute Raccoon), aarch64
  • Hardware: NVIDIA DGX Spark (product version A.7, board P4242), NVIDIA GB10
  • Kernel: 7.3.1-gb10 (NVIDIA GB10 kernel), #1gb10 SMP PREEMPT_DYNAMIC, built 2026-11-03
  • Firmware: AMI BIOS 5.36_0ACUM018 (2025-09-12), GPU VBIOS 9A.0B.2D.00.00, driver 615.71.10

The kernel's own view of the CPU set is inconsistent:

/sys/devices/system/cpu/possible : 0-19
/sys/devices/system/cpu/present  : 0-19
/sys/devices/system/cpu/online   : 1-19     (excludes cpu0)
/sys/devices/system/cpu/offline  : (empty)  (yet cpu0 is not in `online`)

cpu0/online  : 1
cpu19/online : 1

/proc/cpuinfo processors         : 0-18     (includes cpu0, excludes cpu19)
cpu19/cache/index2/size          : 2048K
cpu19/cache/index2/shared_cpu_list : 19     (per-core L2)

cpuinfo enumerates 19 processors from the sysfs set (cpu1..cpu19) but finds no
/proc/cpuinfo entry for cpu19, so the decoder leaves temp_l2.size == 0 while
sysfs still reports a per-core 2 MiB L2 for it. That is exactly the count/fill
disagreement this PR fixes.

The kernel-side inconsistency is what reaches the state; the unguarded l2[...]
write is what turns it into heap corruption.

@Qubitium

Qubitium commented Nov 6, 2026

Copy link
Copy Markdown

Opened #457 which carries this counting-pass fix (your commit is cherry-picked with authorship preserved) plus two additions:

  1. Bounds check on both L2 write sites in cpuinfo_arm_linux_init() (if (l2_index < l2_count)), mirroring the existing l3_index < l3_count guard. The counting fix removes the known divergence, but the write loop has no defense-in-depth if the two passes ever disagree again.
  2. A mock topology regression test (sysfs-l2-missing-arch) that reproduces the overflow. Without the counting fix it aborts in cpuinfo_initialize() with heap corruption (malloc(): unaligned tcache chunk detected); valgrind reports 6 invalid writes, the first 0 bytes past a 32-byte block. With the fix it passes and valgrind is clean.

Confirmed on real hardware too: NVIDIA GB10 (aarch64, Ubuntu 26.04, kernel 7.3.1-gb10), CPU 19 is online in sysfs but absent from /proc/cpuinfo. At bc3c01e the counting pass yields l2_count=18 while the fill pass writes 19 entries, and valgrind pins 4 invalid writes to cpuinfo_arm_linux_init(). The user-visible symptom is import torch aborting with malloc(): corrupted top size.

Note that current main masks this specific host because #438 propagates architecture_version from the cluster leader, which makes the decoder report a generic L2 for CPU 19. The pass asymmetry and the unguarded writes are still there, so the fix is still needed for topologies that do not hit that fallback (e.g. the 32-bit ARM path, arch_version < 7).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants