Conversation
cfjson generates JSON encoders and decoders, cfprotobuf generates Protobuf marshalers and unmarshalers. Both are packages of this module with a small runtime which depends on the standard library only, and the module no longer depends on easyjson, segmentio/encoding and vtprotobuf. JSON output is byte for byte what easyjson produced. Decoding follows RFC 8259 strictly: invalid UTF-8, data after a value, repeated keys and numbers which do not fit are errors, keys are matched exactly, nesting is limited to 1024 levels. Protobuf output is byte for byte what vtprotobuf produced, decoding agrees with google.golang.org/protobuf on what is acceptable and on the decoded message, strings must be valid UTF-8, fields a message does not have are dropped. internal/cfjsoncmp and internal/cfprotobufcmp are modules of their own which keep the old implementations to test compatibility against and to benchmark with, see cfjson/BENCHMARKS.md and cfprotobuf/BENCHMARKS.md. Also: Go 1.26, the stream decoder reads large Protobuf messages as their bytes arrive, security linters for the whole module, govulncheck, CodeQL, tests on 386 and s390x, fuzz targets for the new code with the corpus kept between nightly runs.
JSON decoders first compare the input with the key the next field would have, as encoders write it, before reading a key and looking it up. Maps are filled with one hash operation per entry, long strings are checked four words at a time when encoding, and an omitempty bool is written together with its key. The Protobuf methods of messages are named after cfprotobuf: MarshalCF, MarshalToCF, MarshalToSizedBufferCF, SizeCF and UnmarshalCF. The -suffix option of the generator gives other names, VT for those of vtprotobuf.
JSON reply and push encoders check every raw payload of a message to be a JSON value before encoding it, with a method cfjson generates for that (-valid-raw), where they validated the encoded message before. It is less to look at: the rest of a message is written by generated code. And it is what has to hold, a payload being one value; a payload with a raw newline inside of a string is refused now, it was written without the newline before. cfjson generator: an alias is the type it names, methods included; JSON methods of generated files count unless easyjson generated them; structs with methods in another generated file of the package are not generated again, and one of the two methods without the other is an error. The stream decoder does not overflow on 32-bit platforms when it reads a message over 1 GiB. The integer fuzz test accepts whitespace after a number, as encoding/json does. Benchmark documents are regenerated from a run which has the faster key matching in it, and not yet the payload check of this commit.
Tests are in the file of the source file they test, share their helpers (helpers_test.go) and one registry of message types, run behaviour common to JSON and Protobuf as one table for both, and use require throughout. large_message_test.go and encode_json_test.go are merged into the tests of decoders and encoders, cfjson_test.go is generated_test.go. Fuzz targets are all in fuzz_test.go. FuzzJSONCommandDecode and FuzzProtobufCommandDecode replace FuzzJSONDecodeSingle, FuzzJSONDecodeMultiple and FuzzProtobufDecode, and check that what is decoded encodes again. FuzzJSONPayloads checks that no payload can alter the reply it is in. Benchmarks are rewritten on messages shaped like real ones, as Benchmark<Operation>/type=<json|protobuf>/msg=<message>. They use the exported API only, so make bench-compare REF=<ref> runs them against any earlier version. The embedded-field fixtures of the cfjson tests no longer repeat struct tags, which go vet reports.
The cfjson generator gives the JSON methods declared on an alias to the type the alias names, which is the type Go declares them on, and unwraps a parenthesised receiver. An alias may be given in -types. Raw.MarshalJSON writes null for a value with nothing in it, as cfjson.AppendRaw does. The JSON stream decoder does not count a CR before the newline against the size limit of a command. make bench-compare removes its worktree when interrupted. The Dependabot ignore of easyjson is gone with easyjson. Parallel benchmarks keep their results per goroutine instead of writing to one variable, which measured the contention on it.
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
Vulnerabilities are published without a change in this repository, in the standard library or in a dependency. The Security job runs only on pushes and pull requests, so the same check now also runs every day.
A MarshalJSON which fails, or returns something which is not valid JSON, was written as null. It is reported now, as encoding/json reports it: cfjson.AppendMarshaler panics with a *cfjson.MarshalerError, and the new cfjson.Marshal returns it as an error. Generated encoders have no error to return; a type without JSON methods of its own cannot fail to encode. encoding/json does not call a MarshalJSON with a pointer receiver for a value of a map, which it cannot take the address of, while generated code did. Such maps are an error at generation time now. The cfjson README lists the remaining differences to encoding/json, which are deliberate.
cfjson always copies now: no decoded string points into the input, and the package uses no unsafe. JSONCommandDecoder was the only user of the ZeroCopy flag, and the one decoder whose strings outlived a reused read buffer only by convention; it copies like the stream decoder does. That costs the frame decoder a string allocation per field on small frames. Transports of centrifuge and Centrifugo use the stream decoder, which copied already.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This replaces the three external code generators the package depended on — easyjson (JSON encoding), segmentio/encoding (JSON decoding) and vtprotobuf (Protobuf) — with two generators of our own,
cfjsonandcfprotobuf, both part of this module. The root module now depends ongoogle.golang.org/protobufonly (plus testify in tests).What is new
cfjson (
cfjson/) — a generator of JSON encoders and decoders for Go structs and the runtime the generated code uses (standard library only).[]byte:AppendJSON(b []byte) []byte. Output is byte for byte what easyjson produced for the protocol messages (checked against a golden file and against the old code on random messages).DecodeJSON(b []byte, i int, f cfjson.Flags) int, with an optional zero-copy mode for strings.-fold-keysfor case-insensitive key matching, generating into a separate file without touching the file with the types.cfprotobuf (
cfprotobuf/) — a generator of Protobuf marshalers and unmarshalers for the structsprotoc-gen-goproduces, and its runtime (standard library only).google.golang.org/protobufon what is accepted and on the decoded message (tested against it directly, including by fuzzing).google.golang.org/protobufmessages.Both packages are documented as being for the Centrifugal ecosystem only: they are not general-purpose libraries and will change as our projects need.
Performance
Old implementation vs new, Linux amd64 (AMD EPYC Genoa), 10 samples each, from
cfjson/BENCHMARKS.mdandcfprotobuf/BENCHMARKS.md. Same messages, same structs, both implementations side by side (internal/cfjsoncmp,internal/cfprotobufcmp):Allocations per JSON encode go from 2–110 (depending on message) to 1.
On the paths centrifuge uses (stream decoder for a frame,
GetReplyEncoder(...).Encode,GetPushEncoder(...).Encode), measured against v0.22.1 with the newmake bench-compare(Apple M4, quick run — to be repeated on Linux):Protobuf on the same paths: −4% geomean (decode +1–3%, which is UTF-8 validation of strings vtprotobuf did not do; encode −3% to −11%).
What makes the JSON side faster: append-style encoders without a writer object; index-based decoders that keep their state in registers; a whole-input check which lets strings end at the next quote when a message is plain ASCII; word-at-a-time scanning; matching the key of the next field in declaration order before falling back to a key lookup; encoders of replies and pushes checking only the payloads of a message instead of re-scanning the whole encoded message.
Behaviour changes
API
MarshalCF,MarshalToCF,MarshalToSizedBufferCF,SizeCF,UnmarshalCF(were…VT). Same signatures; it is a rename.MarshalEasyJSON/UnmarshalEasyJSONare gone; messages haveAppendJSON/DecodeJSONinstead.JSON decoding
FilterNode) nested deeper than 1024 levels are an error.The Go and JS SDK test suites pass against a Centrifugo built with this branch, with no decode errors in the server logs.
JSON encoding
Rawfield) is one JSON value. A payload with a raw newline inside of a string is now refused; it used to be sent with the newline removed.Protobuf decoding
google.golang.org/protobuf.Other
Raw.MarshalJSONwritesnullfor an empty value.Testing
internal/cfjsoncmp(easyjson, segmentio),internal/cfprotobufcmp(vtprotobuf). Run withmake json-compat/make protobuf-compat.encoding/jsonandgoogle.golang.org/protobuf.encoding/jsontests run through the decoders withencoding/jsonas the oracle.make bench-compare REF=<ref>compares with any earlier version.CI and tooling
generate.shuses the new generators; tests fail if generated files are out of date.Migration in other projects
VT→CFrename, segmentio can then be dropped.