Repository navigation
Fix value aliasing, map merging on scan and silent integer truncation - #10
Merged
Merged
Conversation
Scanning a SQL NULL assigned the package-level NullJSON slice directly to the destination, so every value scanned from NULL aliased one 4-byte array. UnmarshalJSON then decoded via json.RawMessage, which appends into the existing backing array, letting any payload short enough to fit write through to NullJSON and to every other value sharing it. Detect null against an untyped constant so IsNull no longer depends on mutable package state, allocate a fresh slice on each NULL scan, decode into a new value in UnmarshalJSON, and return a copy from MarshalJSON.
ScanMap and UnmarshalMap decoded into the destination map, and encoding/json merges object keys into a non-nil map rather than replacing it. A map reused across scans - the standard pattern when iterating rows - therefore retained keys from previously scanned records. Decode into a new map and assign it, which also leaves the previous value intact when a scan fails partway.
ScanInt and UnmarshalInt converted straight from int64 to the target type, so values outside the range of a narrower type silently wrapped - including onto the zero value that this package treats as null.
Copying the value on every marshal put a full-payload allocation on the hottest path in the package to guard against a caller mutating the result, which encoding/json never does - it compacts into its own buffer. Match json.RawMessage.MarshalJSON, which returns its slice as-is, and document that callers must not mutate it. Also seed the out-of-range integer test with a non-zero value, so that leaving the destination alone is distinguishable from writing a wrapped one - 4294967296 wraps to exactly 0 for an int32, which a zero-valued destination hid.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes three data-integrity bugs found while auditing the package. The most serious lets a single
JSONvalue corrupt package-level state for the whole process; the other two silently carry data between records or wrap an out-of-range integer onto a different valid value.Changes
json.go— values no longer share backing arraysScanJSONassigned the package-levelNullJSONslice directly into the destination on a NULL, so every value scanned from NULL aliased the same 4-byte array.UnmarshalJSONthen decoded throughjson.RawMessage, whoseUnmarshalJSONdoesappend((*m)[0:0], data...)— an in-place write when the payload fits the existing capacity. Scanning a NULL and later unmarshalling any payload of 4 bytes or fewer into that value therefore wrote through toNullJSONitself.IsNullcompares against a new unexportednullJSONconstant rather than the mutableNullJSONvar, so null detection no longer depends on package state a caller can change.ScanJSONallocates a fresh slice at both NULL sites.UnmarshalJSONdecodes into a localjson.RawMessageand assigns, instead of appending into whatever*jalready points at.ScanJSONis nowbytes.Clone.NullJSONstays an exported var — removing it would break the v3 API — but nothing internal relies on its contents any more, and it's documented as not to be mutated.MarshalJSONstill returns the value's own bytes rather than a copy, matchingjson.RawMessage.MarshalJSON; that's now stated in its doc comment. Copying there would put a full-payload allocation on the hottest path in the package to guard against a mutationencoding/jsonnever performs, since it compacts into its own buffer.map.go— scanning replaces instead of mergingScanMapandUnmarshalMapdecoded into the destination map, andencoding/jsonmerges object keys into a non-nil map rather than replacing it. Reusing one map across scans — the ordinary pattern when iterating rows — retained keys from previously scanned records. Both now decode into a new map and assign it;ScanMapdelegates toUnmarshalMaprather than duplicating the decode. A scan that fails partway now also leaves the previous value intact.int.go— out-of-range integers error instead of wrappingScanIntandUnmarshalIntconverted straight fromint64toT. SinceSignedadmits~int8/~int16/~int32and the README steers custom ID types this way, an out-of-range value silently wrapped onto a different valid ID. A newconvertInthelper round-trips throughint64and returns an error if the value doesn't fit, leaving the destination unchanged.Behavior examples
json.Unmarshal("12")into itNullJSONbecomes"12ll"process-wide; unrelated values scanned from NULL change with itNullJSONunchangedJSON("null").Value()after the above[]byte("null")— writes the literal string to the columnnil— writes SQL NULL{"a":1}then{"b":2}into one map{"a":1,"b":2}{"b":2}UnmarshalInt("4294967297")into anint32type14294967297 is out of range for TUnmarshalInt("4294967296")into anint32type0— the value treated as nullTest plan
go test -p=1 -coverprofile=coverage.text -covermode=atomic ./...(the CI command) passes against Postgres; coverage 93.9%-racegofmt -landgo vet ./...cleanTestJSONNullIsNotShared— the scan-NULL-then-unmarshal sequence, asserting no write-through to a sibling value or toNullJSON, and that a real null still writes as SQL NULLTestJSONIsNullDoesntDependOnNullJSON— reassignsNullJSONand checksIsNullis unaffectedTestJSONMarshalReturnsRawBytes,TestJSONScanDoesntAliasDriverBuffer— marshal output and the driver-buffer cloneTestMapScanReplacesExistingKeys/TestMapUnmarshalReplacesExistingKeys— one map reused across{...},{},null,nil, empty bytes, plus the failed-scan caseTestIntOutOfRange— overflow, the wrap-to-zero-sentinel case, both type limits, the fullint64range, and null; seeded with a non-zero destination so that leaving it alone is distinguishable from writing a wrapped valueUpgrade note
The integer change turns a silent wrap into an error, so callers currently relying on out-of-range values landing on some arbitrary number will now see a scan or unmarshal failure. That's the intended correction, but it is a behavior change worth a changelog entry when this is released.