Skip to content

arm/linux: fix out-of-bounds L2 cache write, add bounds check and regression test - #457

Open
Qubitium wants to merge 3 commits into
pytorch:mainfrom
ModelCloud:fix/arm-linux-l2-count-overflow
Open

Qubitium wants to merge 3 commits into
pytorch:mainfrom
ModelCloud:fix/arm-linux-l2-count-overflow

Conversation

@Qubitium

@Qubitium Qubitium commented Nov 6, 2026 •

Copy link
Copy Markdown

Problem

cpuinfo_arm_linux_init() sizes the L2 cache array in one pass and fills it in a second pass, and the two passes do not agree on which cores get an entry.

The counting pass is gated on the decoded L2 size:

if (temp_l2.size != 0) {
    ...
    if (l2_is_per_core) l2_count += 1;
}

The population pass also accepts a sysfs L2, and does not bounds-check:

if (temp_l3.size != 0 || l2_is_per_core) {
    l2_index += 1;
    l2[l2_index] = (struct cpuinfo_cache){ ... };
}

A core with a per-core L2 in sysfs but no decoded L2 is skipped while counting and appended while populating. l2_index runs past l2_count and writes one struct cpuinfo_cache past the calloc(l2_count, sizeof(struct cpuinfo_cache)) block. When l2_count is 0 the write goes through a NULL l2.

The L3 writes below are guarded (if (l3_index < l3_count)). The L2 writes are not.

Reproduction: LXD thin container on GB10

Host: NVIDIA GB10, aarch64, Ubuntu 26.04.1 LTS, kernel 7.3.1-gb10.

The process ran in an LXD thin container whose cpuset was limited to 19 of the host's 20 CPUs. lxcfs is mounted over /proc/cpuinfo, /proc/stat and /sys/devices/system/cpu. It filters /proc/cpuinfo to the cpuset and renumbers the entries from 0, while /sys/devices/system/cpu still exposes all 20 host CPUs:

/sys/fs/cgroup/cpuset.cpus.effective : 1-19
/sys/devices/system/cpu/cpu[0-9]*    : 20 dirs (cpu0 ... cpu19)
/proc/cpuinfo processors             : 19 entries (0 ... 18)

cpu0/online  = 1
cpu19/online = 1
/proc/interrupts columns = 20

Nothing is offline. The kernel view is consistent; only the container's /proc/cpuinfo is not.

The renumbering is visible in the cluster layout. The host is four clusters of five:

host CPUs core L2 L3
0-4 Cortex-A725 (0xd87) 512K 8M
5-9 Cortex-X925 (0xd85) 2048K 8M
10-14 Cortex-A725 (0xd87) 512K 16M
15-19 Cortex-X925 (0xd85) 2048K 16M

The container's /proc/cpuinfo reports 0xd87 x4, 0xd85 x5, 0xd87 x5, 0xd85 x5 for processors 0-3, 4-8, 9-13, 14-18. The first block lost an entry, so processor N describes host CPU N+1.

Host CPU 19 therefore has no /proc/cpuinfo entry: the decoder reports no L2 for it and architecture_version stays 0. The counting pass yields l2_count = 18; the population pass writes 19 entries.

COUNT i=9 sysid=19 decoded_l2=0 decoded_l3=0 l2_count=18
FILL  i=9 sysid=19 l2=2097152 per_core=1 l2_index=8  l2_count=18

Valgrind on the process:

Invalid write of size 8   Address ... is 0 bytes after a block of size 576 (18 x 32) alloc'd
3 further invalid writes (sizes 8, 4, 4) after the same block
ERROR SUMMARY: 4 errors from 4 contexts

Visible symptom on this machine: import torch aborts with malloc(): corrupted top size. This happens before CUDA initializes, and reproduces with both CUDA and CPU-only wheels.

The trigger is any /proc/cpuinfo that is a strict subset of the CPUs in /sys/devices/system/cpu, which includes any container cpuset that is not all host CPUs.

Note on current main

#438 propagates architecture_version from the cluster leader, so on this host the decoder now returns a generic L2 for CPU 19 and both passes agree (19/19, valgrind clean). The pass asymmetry and the unguarded writes are unchanged, so the overflow remains reachable where a core gets a sysfs L2, no decoded L2, and no inherited architecture (e.g. arch_version < 7, or a cluster whose leader also lacks an identity).

Reproduction in the mock harness

New sysfs-l2-missing-arch topology in test/mock/arm-linux-topology.cc: every CPU advertises a per-core L2 in sysfs, and one CPU has no CPU architecture line and no cluster sibling to inherit one from.

Without the fix:

$ ./topology-test sysfs-l2-missing-arch
malloc(): unaligned tcache chunk detected
Aborted (core dumped)

$ valgrind ./topology-test sysfs-l2-missing-arch
Invalid write of size 8   Address 0x... is 0 bytes after a block of size 32 alloc'd
Invalid write of size 4   Address 0x... is 8 bytes after a block of size 32 alloc'd
Invalid write of size 4   Address 0x... is 24 bytes after a block of size 32 in arena "client"
Invalid write of size 8   Address 0x... is 16 bytes after a block of size 32 alloc'd
ERROR SUMMARY: 6 errors from 4 contexts

With the fix: 3/3 pass, valgrind 0 errors.

Fix

  1. fix out-of-bounds l2 cache write in cpuinfo_arm_linux_init — cherry-picked from fix out-of-bounds l2 cache write in cpuinfo_arm_linux_init #418, authorship preserved. The counting pass applies the same sysfs override the population pass already applies, so l2_count matches the number of entries written.
  2. arm/linux: bounds-check the L2 cache writes — mirror the L3 handling: skip out-of-range writes, and only set the processor's cache pointer for an entry that was written.
  3. test/mock: cover a sysfs L2 with no decoded L2 — the sysfs-l2-missing-arch topology above.

Difference from #418

#418 this PR
counting-pass fix yes (+19/-15) yes, same hunk, authorship preserved
bounds check on the L2 writes no yes, both write sites
regression test none mock topology that aborts without the fix
reproduction evidence prose GB10 hardware, instrumented count/fill, valgrind, mock test
files touched src/arm/linux/init.c src/arm/linux/init.c, test/mock/arm-linux-topology.cc, CMakeLists.txt
status open since 2026-07-14, blocked

PR #418 has the right root cause and a correct counting fix, but leaves the unguarded write and ships no test, so any future divergence between the passes reintroduces the same silent heap corruption. Commits 2 and 3 here also apply cleanly on top of it if it lands on its own.

How to run the test

The topology tests are gated on CMAKE_SYSTEM_NAME STREQUAL "Android", so on Linux/aarch64 build the harness directly:

cmake -S . -B build -DCPUINFO_BUILD_MOCK_TESTS=ON
cmake --build build -j --target cpuinfo_mock gtest
g++ -std=c++17 -DCPUINFO_MOCK=1 -Iinclude -Isrc -Itest/mock \
    -Ideps/googletest/googletest/include test/mock/arm-linux-topology.cc \
    build/libcpuinfo_mock.a build/lib/libgtest.a -lpthread -o topology-test
./topology-test sysfs-l2-missing-arch

Existing modes (pixel-8, pixel-9, pixel-10, missing-midr, frequency-groups, unknown-parts, mixed-caches) still pass.

@meta-cla

meta-cla Bot commented Nov 6, 2026

Copy link
Copy Markdown

Hi @Qubitium!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

@meta-cla

meta-cla Bot commented Nov 6, 2026

Copy link
Copy Markdown

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

@fbarchard fbarchard left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The root cause and fix look solid — aligning the Pass 1 L2 counting logic with Pass 2's sysfs override, adding bounds checks on both l2[l2_index] write sites (which #434 didn't cover), and adding a mock regression test while preserving #418's authorship all make sense.

A couple of things before merging:

  1. Rebase onto main: Commit 2ca65a8a (#434) added if (l2 != NULL && l2_index < l2_count) on the shared-L2 pointer assignment in src/arm/linux/init.c, which currently conflicts with this PR. Keeping the l2 != NULL && l2_index < l2_count guard consistently across both L2 write/assignment sites looks clean.
  2. Test ordering fragility in sysfs-l2-missing-arch: The new test pairs 0xD87 (Cortex-A725, group 0) with 0xFFF (unknown, group 1) at the same frequency and expects expected_linux_ids = {1, 0}. That order only happens because Cortex-A725 isn't in midr_score_core() in src/arm/midr.h yet and falls through to default: return 3, tying with 0xFFF and falling back to the cluster_a > cluster_b tie-break. Once midr_score_core() is updated to score Cortex-A725 (score 5), this test will flip to {0, 1} and fail. Using an already-scored core (e.g. 0xD4D Cortex-A715 with {0, 1}, or 0xD05 Cortex-A55 with {1, 0}) or a second unknown part (0xFFE) avoids that coupling.

Non-blocking follow-up note: when cpuinfo_arm_decode_cache() leaves temp_l2 zeroed and size comes solely from sysfs, temp_l2.line_size and temp_l2.partitions stay 0 so temp_l2.sets is never computed (and associativity stays 0 if ways_of_associativity isn't present). That still eliminates the OOB write here, though init-test's L2_CACHE.{valid_size, non_zero_line_size, reasonable_line_size} assertions would still fail on such a core unless sysfs coherency_line_size / fallback defaults are populated in a separate follow-up.

soma0212 and others added 3 commits November 22, 2026 03:39
The population pass appends an L2 entry whenever the core has an L3 or sysfs
reports a per-core L2, but the array is sized by a separate counting pass that
only consults sysfs when the decoder already reported a non-zero L2. If the
two passes disagree, the write runs past the end of the array.

Mirror the existing L3 handling: skip the write when the index is out of
range, and only set the processor's cache pointer for an entry that was
actually written.
Give both CPUs a per-core sysfs L2 and omit the architecture version for the unknown core. Verify that both L2 entries are present even though the cache decoder supplies no L2 for the unknown core.

Pair it with an explicitly scored Cortex-A55 to keep the expected processor order stable.
@Qubitium
Qubitium force-pushed the fix/arm-linux-l2-count-overflow branch from 9c25921 to e6b885c Compare November 22, 2026 03:42
@Qubitium

Qubitium commented Nov 22, 2026 •

Copy link
Copy Markdown
Author

@fbarchard Applied the null and bounds checks at both L2 write sites, and switched the regression fixture to Cortex-A55, with both native Linux tests and all 10 ARM topology cases passing, including under AddressSanitizer for the topology cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants