Skip to content

Fix value aliasing, map merging on scan and silent integer truncation - #10

Merged
rowanseymour merged 4 commits into
mainfrom
rowan/fix-shared-buffer-aliasing
Aug 31, 2026
Merged

rowanseymour merged 4 commits into
mainfrom
rowan/fix-shared-buffer-aliasing

Conversation

@rowanseymour

@rowanseymour rowanseymour commented Aug 31, 2026 •

Copy link
Copy Markdown
Member

Summary

Fixes three data-integrity bugs found while auditing the package. The most serious lets a single JSON value corrupt package-level state for the whole process; the other two silently carry data between records or wrap an out-of-range integer onto a different valid value.

Changes

json.go — values no longer share backing arrays

ScanJSON assigned the package-level NullJSON slice directly into the destination on a NULL, so every value scanned from NULL aliased the same 4-byte array. UnmarshalJSON then decoded through json.RawMessage, whose UnmarshalJSON does append((*m)[0:0], data...) — an in-place write when the payload fits the existing capacity. Scanning a NULL and later unmarshalling any payload of 4 bytes or fewer into that value therefore wrote through to NullJSON itself.

  • IsNull compares against a new unexported nullJSON constant rather than the mutable NullJSON var, so null detection no longer depends on package state a caller can change.
  • ScanJSON allocates a fresh slice at both NULL sites.
  • UnmarshalJSON decodes into a local json.RawMessage and assigns, instead of appending into whatever *j already points at.
  • The manual clone in ScanJSON is now bytes.Clone.

NullJSON stays an exported var — removing it would break the v3 API — but nothing internal relies on its contents any more, and it's documented as not to be mutated.

MarshalJSON still returns the value's own bytes rather than a copy, matching json.RawMessage.MarshalJSON; that's now stated in its doc comment. Copying there would put a full-payload allocation on the hottest path in the package to guard against a mutation encoding/json never performs, since it compacts into its own buffer.

map.go — scanning replaces instead of merging

ScanMap and UnmarshalMap decoded into the destination map, and encoding/json merges object keys into a non-nil map rather than replacing it. Reusing one map across scans — the ordinary pattern when iterating rows — retained keys from previously scanned records. Both now decode into a new map and assign it; ScanMap delegates to UnmarshalMap rather than duplicating the decode. A scan that fails partway now also leaves the previous value intact.

int.go — out-of-range integers error instead of wrapping

ScanInt and UnmarshalInt converted straight from int64 to T. Since Signed admits ~int8/~int16/~int32 and the README steers custom ID types this way, an out-of-range value silently wrapped onto a different valid ID. A new convertInt helper round-trips through int64 and returns an error if the value doesn't fit, leaving the destination unchanged.

Behavior examples

Input Before After
Scan NULL, then json.Unmarshal("12") into it NullJSON becomes "12ll" process-wide; unrelated values scanned from NULL change with it each value independent; NullJSON unchanged
JSON("null").Value() after the above []byte("null") — writes the literal string to the column nil — writes SQL NULL
Scan {"a":1} then {"b":2} into one map {"a":1,"b":2} {"b":2}
UnmarshalInt("4294967297") into an int32 type 1 error: 4294967297 is out of range for T
UnmarshalInt("4294967296") into an int32 type 0 — the value treated as null error

Test plan

  • go test -p=1 -coverprofile=coverage.text -covermode=atomic ./... (the CI command) passes against Postgres; coverage 93.9%
  • Full suite passes under -race
  • gofmt -l and go vet ./... clean
  • Each bug reproduced against the previous code first, then confirmed closed
  • Every new regression test re-run against the pre-fix source to confirm it actually fails there
  • New regression tests:
    • TestJSONNullIsNotShared — the scan-NULL-then-unmarshal sequence, asserting no write-through to a sibling value or to NullJSON, and that a real null still writes as SQL NULL
    • TestJSONIsNullDoesntDependOnNullJSON — reassigns NullJSON and checks IsNull is unaffected
    • TestJSONMarshalReturnsRawBytes, TestJSONScanDoesntAliasDriverBuffer — marshal output and the driver-buffer clone
    • TestMapScanReplacesExistingKeys / TestMapUnmarshalReplacesExistingKeys — one map reused across {...}, {}, null, nil, empty bytes, plus the failed-scan case
    • TestIntOutOfRange — overflow, the wrap-to-zero-sentinel case, both type limits, the full int64 range, and null; seeded with a non-zero destination so that leaving it alone is distinguishable from writing a wrapped value

Upgrade note

The integer change turns a silent wrap into an error, so callers currently relying on out-of-range values landing on some arbitrary number will now see a scan or unmarshal failure. That's the intended correction, but it is a behavior change worth a changelog entry when this is released.

Scanning a SQL NULL assigned the package-level NullJSON slice directly to
the destination, so every value scanned from NULL aliased one 4-byte array.
UnmarshalJSON then decoded via json.RawMessage, which appends into the
existing backing array, letting any payload short enough to fit write
through to NullJSON and to every other value sharing it.

Detect null against an untyped constant so IsNull no longer depends on
mutable package state, allocate a fresh slice on each NULL scan, decode
into a new value in UnmarshalJSON, and return a copy from MarshalJSON.
ScanMap and UnmarshalMap decoded into the destination map, and encoding/json
merges object keys into a non-nil map rather than replacing it. A map reused
across scans - the standard pattern when iterating rows - therefore retained
keys from previously scanned records.

Decode into a new map and assign it, which also leaves the previous value
intact when a scan fails partway.
ScanInt and UnmarshalInt converted straight from int64 to the target type,
so values outside the range of a narrower type silently wrapped - including
onto the zero value that this package treats as null.
Copying the value on every marshal put a full-payload allocation on the
hottest path in the package to guard against a caller mutating the result,
which encoding/json never does - it compacts into its own buffer. Match
json.RawMessage.MarshalJSON, which returns its slice as-is, and document
that callers must not mutate it.

Also seed the out-of-range integer test with a non-zero value, so that
leaving the destination alone is distinguishable from writing a wrapped
one - 4294967296 wraps to exactly 0 for an int32, which a zero-valued
destination hid.
@rowanseymour
rowanseymour merged commit a5bb454 into main Aug 31, 2026
8 checks passed
@rowanseymour
rowanseymour deleted the rowan/fix-shared-buffer-aliasing branch August 31, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant