cbor: Scapy-native fields, codec hardening, and cbor2 CI by polybassa · Pull Request #5125 · secdev/scapy

polybassa · 2026-09-01T11:41:21Z

Summary

Rewrite CBOR packet/field APIs toward Scapy ASN.1-style conventions
Harden maps, floats, optionals, and raw-cache fidelity
Add pinned cbor2 differential coverage behind isolated tox/CI

Stack

PR 2 of 3. Depends on #5124 (CRC presets). Reviewable CBOR-only commits sit on top of that foundation; next is BPv7 contrib.

Test plan

./test/run_tests -t test/scapy/layers/cbor.uts -N
tox -e cbor2 (Python 3.12)
flake8 / mypy on touched paths

codecov · 2026-09-01T11:46:20Z

Codecov Report

❌ Patch coverage is 86.64303% with 226 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.78%. Comparing base (df72ff3) to head (9761613).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
scapy/cbor/cborfields.py	84.19%	147 Missing ⚠️
scapy/cbor/cborcodec.py	90.92%	41 Missing ⚠️
scapy/cbor/cbor.py	86.42%	38 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5125      +/-   ##
==========================================
+ Coverage   80.07%   80.78%   +0.70%     
==========================================
  Files         375      375              
  Lines       98740    99974    +1234     
==========================================
+ Hits        79065    80760    +1695     
+ Misses      19675    19214     -461

Files with missing lines	Coverage Δ
scapy/cbor/__init__.py	`100.00% <ø> (ø)`
scapy/cborpacket.py	`97.14% <100.00%> (+3.39%)`	⬆️
scapy/packet.py	`84.95% <100.00%> (+0.11%)`	⬆️
scapy/utils.py	`73.76% <100.00%> (+0.19%)`	⬆️
scapy/cbor/cbor.py	`86.17% <86.42%> (+2.32%)`	⬆️
scapy/cbor/cborcodec.py	`90.60% <90.92%> (+2.86%)`	⬆️
scapy/cbor/cborfields.py	`84.29% <84.19%> (+6.35%)`	⬆️

... and 26 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

🔵 Needs a closer look

It makes wide-reaching changes across CBOR core encoding/decoding and packet cache semantics, which warrants careful human validation beyond static review.

Pull request overview

This PR refactors Scapy’s CBOR support toward Scapy-native field conventions (Python-native values, ASN.1-like hooks), strengthens codec correctness (maps, floats, optionals, nesting/indefinite handling, deterministic encoding), and adds an isolated cbor2-based interoperability/differential test lane in tox and CI.

Changes:

Rework CBOR field/container APIs to use Scapy-style any2i/i2m/m2i plus build_result/dissect_result, with improved raw-cache fidelity and parent/child ownership handling.
Harden CBOR codec behavior (indefinite containers, duplicate keys, preferred float widths, deterministic map ordering, max nesting) and introduce CBOR map pair preservation (CBORMapData).
Add pinned cbor2==6.1.4 differential UTScapy campaign plus a dedicated tox env and GitHub Actions job.

File summaries

File	Description
`tox.ini`	Adds a dedicated `cbor2` tox env for pinned differential tests; removes `cbor2` from main deps.
`test/scapy/layers/requirements-cbor2.txt`	Documents/pins optional cbor2 dependency for interop testing.
`test/scapy/layers/generate_cbor2_corpus.py`	Adds helper script to generate reproducible CBOR corpora using cbor2.
`test/scapy/layers/cbor_cbor2_interop.uts`	Adds extensive cbor2-vs-Scapy differential/interoperability coverage.
`test/fields.uts`	Adds a regression test around `ConditionalField.__getattr__`.
`test/configs/windows2.utsc`	Excludes the cbor2 interop campaign from default Windows2 runs and marks keyword.
`test/configs/windows.utsc`	Excludes the cbor2 interop campaign from default Windows runs and marks keyword.
`test/configs/solaris.utsc`	Excludes the cbor2 interop campaign from default Solaris runs and marks keyword.
`test/configs/linux.utsc`	Excludes the cbor2 interop campaign from default Linux runs and marks keyword.
`test/configs/bsd.utsc`	Excludes the cbor2 interop campaign from default BSD runs and marks keyword.
`scapy/cborpacket.py`	Adds CBOR-specific packet cache integration, default-copy deepening, and parent reattachment on copy.
`scapy/cbor/cborfields.py`	Major CBOR field API rewrite (native values, optional/conditional semantics, sequences, deterministic map schema behavior).
`scapy/cbor/cborcodec.py`	Hardens low-level codec: nesting limit, indefinite support, strictness checks, float width preference, deterministic encoding helpers.
`scapy/cbor/cbor.py`	Introduces `CBORMapData`, richer equality/hash semantics, float encoding retention, and native wrapper classes.
`scapy/cbor/__init__.py`	Exposes new CBOR types/fields/results in the public API surface.
`README.md`	Fixes packaging status badge markdown formatting.
`.github/workflows/unittests.yml`	Adds a CI job running `tox -e cbor2` on Python 3.12.

Review details

Files reviewed: 17/18 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Capture map-key equivalence, deterministic floats, hash/tag field semantics, and multi-map unknown ownership before changing the implementation. Co-authored-by: Cursor <cursoragent@cursor.com>

Capture map-key equivalence, deterministic floats, hash/tag field semantics, and multi-map unknown ownership before changing the implementation. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Composer)

polybassa · 2026-09-11T06:55:23Z

@gpotter2 I've done with a lot of refactorings and a manual review; Could you please have a look?

Capture map-key equivalence, deterministic floats, hash/tag field semantics, and multi-map unknown ownership before changing the implementation. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Composer)

polybassa · 2026-09-21T07:56:33Z

@gpotter2 Could you please have a look?

Capture map-key equivalence, deterministic floats, hash/tag field semantics, and multi-map unknown ownership before changing the implementation. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Composer)

gpotter2 · 2026-10-30T08:52:34Z

I was looking at the CBOR doc yesterday, I don't find it particularly useful teaching how to actually make CBOR packets. Could you drop anything's that been written unsupervised by AI? Thx

Do you want me to take a look at this PR again?

polybassa · 2026-10-30T13:04:59Z

I'll rework the docs.

Could you please have a look on the code.

polybassa · 2026-11-09T06:26:09Z

@gpotter2 I've rewritten the docs file and reviewed it.

Count preserved unknown MAP members in is_empty, encode nested ARRAY_OF fields, and reject remainder/kwargs misplacements. AI-Assisted: yes (Cursor Grok 4.6) Co-authored-by: Cursor <cursoragent@cursor.com>

Cover CBORF_ITEMS TypeError for unexpected keywords alongside the existing ARRAY encode_indefinite case. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

Stop conditional suffix reservation from using undecoded discriminators, initialize zero-budget remainders and nested optionals, fingerprint float bit patterns, and reject nested packet-valued homogeneous collections. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

Inline unframed ITEMS into array schemas for budgeting, reserve conditionals only after required predecessors are decoded, and trim preferred-float helpers. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

Replace the positional between-required heuristic with reservation based on already-decoded field names so unrelated required fields do not suppress a known active conditional. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

Put external_cbor2 on the interop campaign, assert preferred float wire bytes, fix indefinite budget and ITEMS payload checks, and drop the non-determinism scanner helper. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

Proxy getfieldval and subclass methods on the reservation view, propagate real predicate errors, cap indefinite min_items work instead of requiring quadratic growth, and trim interop helpers. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

Drop the fake reservation view. Predicates run on the packet; unread fields do not reserve from defaults. Map dissection uses field wrappers, and UTS covers greedy optional-before-flag. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

Drop unread-field interception. Suffix reservation uses min_items on the real packet. Default-on trailing conditionals over-reserve until the discriminator is dissected. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

Only optionals inspect the suffix. Conditionals reserve when named discriminators are already in pkt.fields, so default-on flags round-trip instead of rejecting their own encodings. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

Unread default-on conditionals no longer starve a preceding optional during suffix budgeting. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

A skipped conditional is decided but never stored in pkt.fields, so later depends_on treated it as unread. AI-Assisted: yes (Cursor Grok 4.6) Co-authored-by: Cursor <cursoragent@cursor.com>

depends_on does not delay ConditionalField evaluation. Unknown names include the conditional field in the error. AI-Assisted: yes (Cursor Grok 4.6) Co-authored-by: Cursor <cursoragent@cursor.com>

AI-Assisted: yes (Cursor Grok 4.6) Co-authored-by: Cursor <cursoragent@cursor.com>

AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

Same-semantic child assignment clears the child cache; the parent must see that and rebuild overlong encodings. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

Move the seeded corpus into the UTS campaign and remove the standalone generator and duplicate requirements file. AI-Assisted: yes (OpenCode, GPT-6.1)

Document native fields, optional values, and map handling. AI-Assisted: yes (OpenAI GPT-6.1-sol)

Keep null and undefined as explicit CBOR objects in packet fields. AI-Assisted: yes (OpenAI Codex)

Remove recursive cache fingerprints and normalize packet encodings on rebuild. AI-Assisted: yes (OpenAI Codex)

Keep internal item accounting while honoring build, dissect, and m2i overrides on compound fields. AI-Assisted: yes (OpenAI Codex)

Rename the collection base and keep array reservation logic beside child dissection. AI-Assisted: yes (OpenAI Codex)

Keep payload-sensitive width selection in enc_float and simplify deterministic value dispatch. AI-Assisted: yes (OpenAI Codex)

Remove the header sentinel and counted-hook introduction, and keep buffer scans and imports consistent. AI-Assisted: yes (OpenAI Codex)

Describe None as absence, explicit null and undefined objects, and normalization of decoded packets. AI-Assisted: yes (OpenAI Codex)

Copilot

🟡 Changes recommended

Active issues remain in the documentation policy compliance, canonical CBOR map ordering, and tox environment/test isolation.

3 open issues

2 resolved since last review

The commands entry for the cbor2 tox env uses a trailing \ as if tox supported shell-style…
This test asserts implementation details by grepping inspect.getsource(...) for a substring,…

_{Review effort: Lite}

+                value
+            )
+            encoded_pairs.append((key_bytes, value_bytes))
+        encoded_pairs.sort(key=lambda item: item[0])


-  bsd-non_root: {envpython} {env:DISABLE_COVERAGE:-m coverage run} -m scapy.tools.UTscapy -c test/configs/bsd.utsc -K tshark -N {posargs}
-  bsd-root: sudo -E {envpython} {env:DISABLE_COVERAGE:-m coverage run} -m scapy.tools.UTscapy -c test/configs/bsd.utsc -K tshark {posargs}
-  windows: {envpython} {env:DISABLE_COVERAGE:-m coverage run} -m scapy.tools.UTscapy -c test/configs/windows.utsc {posargs}
+  linux-non_root: {envpython} {env:DISABLE_COVERAGE:-m coverage run} -m scapy.tools.UTscapy -c ./test/configs/linux.utsc -N {env:CBOR2_TEST_FLAGS:} {posargs}


+CBOR (Concise Binary Object Representation) is a binary data format defined
+in `RFC 8949 <https://www.rfc-editor.org/rfc/rfc8949.html>`_. Scapy provides
+CBOR objects and codecs for individual values, and ``CBOR_Packet`` with
+``CBORF_*`` fields for describing packet formats. No external CBOR library
+is required.


This was referenced Sep 1, 2026

New BPv7 with working UTS and patched cbor modules to support it #5075

Closed

bpv7: Bundle Protocol v7 contrib with RFC validation #5126

Open

polybassa force-pushed the split/cbor-scapy-native branch from 5f9b34d to 414ac55 Compare September 1, 2026 11:57

polybassa requested review from gpotter2 and a lite review from Copilot September 1, 2026 12:04

Copilot started reviewing on behalf of polybassa September 1, 2026 12:04

View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Comment thread tox.ini Resolved

Comment thread test/fields.uts Resolved

polybassa force-pushed the split/cbor-scapy-native branch from 414ac55 to 4a552ff Compare September 9, 2026 05:01

polybassa force-pushed the split/cbor-scapy-native branch from 4a552ff to f9e6104 Compare September 9, 2026 06:32

polybassa force-pushed the split/cbor-scapy-native branch from b433d7e to 06a065b Compare September 10, 2026 06:30

polybassa force-pushed the split/cbor-scapy-native branch 2 times, most recently from b7fc549 to 1f059ae Compare September 16, 2026 15:34

polybassa force-pushed the split/cbor-scapy-native branch from 1f059ae to fce7bb4 Compare September 16, 2026 15:37

polybassa force-pushed the split/cbor-scapy-native branch from fce7bb4 to f16842e Compare September 29, 2026 18:54

polybassa commented Sep 30, 2026

View reviewed changes

Comment thread test/scapy/layers/requirements-cbor2.txt Resolved

polybassa commented Sep 30, 2026

View reviewed changes

Comment thread test/scapy/layers/generate_cbor2_corpus.py Resolved

gpotter2 reviewed Sep 30, 2026

View reviewed changes

Comment thread tox.ini Resolved

Comment thread tox.ini Resolved

Comment thread test/scapy/layers/requirements-cbor2.txt Resolved

Comment thread test/scapy/layers/generate_cbor2_corpus.py Resolved

gpotter2 reviewed Nov 11, 2026

View reviewed changes

Comment thread scapy/cbor/cborcodec.py Resolved

Nils Weiss and others added 25 commits November 12, 2026 22:03

cbor: keep unknown maps through optional tags

9f3c0ed

Count preserved unknown MAP members in is_empty, encode nested ARRAY_OF fields, and reject remainder/kwargs misplacements. AI-Assisted: yes (Cursor Grok 4.6) Co-authored-by: Cursor <cursoragent@cursor.com>

cbor: reject unused ITEMS kwargs in UTS

d70362d

Cover CBORF_ITEMS TypeError for unexpected keywords alongside the existing ARRAY encode_indefinite case. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

cbor: default depends_on does not reserve array suffix items

1c7a70f

Unread default-on conditionals no longer starve a preceding optional during suffix budgeting. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

cbor: reserve array suffix from resolved fields, not pkt.fields

71caaf0

A skipped conditional is decided but never stored in pkt.fields, so later depends_on treated it as unread. AI-Assisted: yes (Cursor Grok 4.6) Co-authored-by: Cursor <cursoragent@cursor.com>

cbor: document depends_on as optional-tail reservation only

a0e8846

depends_on does not delay ConditionalField evaluation. Unknown names include the conditional field in the error. AI-Assisted: yes (Cursor Grok 4.6) Co-authored-by: Cursor <cursoragent@cursor.com>

cbor: drop unused ITEMS dissect helper and type-check depends_on

8ebd71d

AI-Assisted: yes (Cursor Grok 4.6) Co-authored-by: Cursor <cursoragent@cursor.com>

cbor: type depends_on as Sequence[str] to match runtime

d8e42ec

AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

cbor: include nested child raw_packet_cache in parent fingerprint

be4a3ca

Same-semantic child assignment clears the child cache; the parent must see that and rebuild overlong encodings. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>

cbor: integrate cbor2 checks into the regular test suite

17b1928

Move the seeded corpus into the UTS campaign and remove the standalone generator and duplicate requirements file. AI-Assisted: yes (OpenCode, GPT-6.1)

cbor: update packet and codec documentation

861299b

Document native fields, optional values, and map handling. AI-Assisted: yes (OpenAI GPT-6.1-sol)

cbor: use None for absent packet fields

22e9e65

Keep null and undefined as explicit CBOR objects in packet fields. AI-Assisted: yes (OpenAI Codex)

cbor: rebuild packet values without raw-cache tracking

b55a46b

Remove recursive cache fingerprints and normalize packet encodings on rebuild. AI-Assisted: yes (OpenAI Codex)

cbor: route counted operations through public field hooks

188ad28

Keep internal item accounting while honoring build, dissect, and m2i overrides on compound fields. AI-Assisted: yes (OpenAI Codex)

cbor: clarify collection names and optional-tail reservation

8146435

Rename the collection base and keep array reservation logic beside child dissection. AI-Assisted: yes (OpenAI Codex)

cbor: move deterministic NaN encoding into the float codec

7ec4a18

Keep payload-sensitive width selection in enc_float and simplify deterministic value dispatch. AI-Assisted: yes (OpenAI Codex)

cbor: use None for indefinite lengths and trim module scaffolding

be821aa

Remove the header sentinel and counted-hook introduction, and keep buffer scans and imports consistent. AI-Assisted: yes (OpenAI Codex)

cbor: update guide for optional values and packet rebuilds

9761613

Describe None as absence, explicit null and undefined objects, and normalization of decoded packets. AI-Assisted: yes (OpenAI Codex)

polybassa force-pushed the split/cbor-scapy-native branch from 97d386c to 9761613 Compare November 12, 2026 21:05

polybassa requested review from gpotter2 and a lite review from Copilot November 12, 2026 21:05

Copilot started reviewing on behalf of polybassa November 12, 2026 21:06

View session

Copilot AI reviewed Nov 12, 2026

View reviewed changes

Navigation Menu

Repository navigation

cbor: Scapy-native fields, codec hardening, and cbor2 CI - #5125

cbor: Scapy-native fields, codec hardening, and cbor2 CI#5125
polybassa wants to merge 90 commits into
secdev:masterfrom
polybassa:split/cbor-scapy-native

polybassa commented Sep 1, 2026

Uh oh!

codecov Bot commented Sep 1, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

polybassa commented Sep 11, 2026

Uh oh!

polybassa commented Sep 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gpotter2 commented Oct 30, 2026

Uh oh!

polybassa commented Oct 30, 2026

Uh oh!

polybassa commented Nov 9, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

polybassa commented Sep 1, 2026

Summary

Stack

Test plan

Uh oh!

codecov Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

polybassa commented Sep 11, 2026

Uh oh!

polybassa commented Sep 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gpotter2 commented Oct 30, 2026

Uh oh!

polybassa commented Oct 30, 2026

Uh oh!

polybassa commented Nov 9, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Sep 1, 2026 •

edited

Loading