Skip to content

Make public header components acyclic, and forbid new cycles - #8478

Open
Amaury Chamayou (achamayou) wants to merge 6 commits into
mainfrom
achamayou-musical-giggle
Open

Amaury Chamayou (achamayou) wants to merge 6 commits into
mainfrom
achamayou-musical-giggle

Conversation

@achamayou

@achamayou Amaury Chamayou (achamayou) commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Motivation

Follow-up to #8476, for #3517.

#8476 made the src/ component graph acyclic, but check-source-dependencies.py only scans src/. Under include/ccf/, three groups of public header components still include each other in cycles. For example, ccf/ds/json.h includes ccf/crypto/base64.h, and ccf/crypto/sha256_hash.h includes ccf/service/map.h for a single serialiser, so the lowest-level headers pull in the whole typed KV API. Nothing stops new cycles either.

ccf/research/create_tx_claims_digest.h also included the private <kv/kv_types.h>, which is not installed, so applications could not use it. includes-checks.sh only checks quoted includes, so it missed this.

This targets 8.x, a major release, so the fix does not preserve source compatibility: headers move to the component that owns them, without forwarding headers or aliases.

The resulting public header layering, and the includes this PR removes

Each node is a public header component, a directory under include/ccf/. Top-level headers are left out: an arrow means that a component's headers include another's, directly or through top-level headers. As in #8476, an arrow is omitted when a longer path implies it.

The dashed red arrows are the includes this PR removes. Three pointed up the layering and closed the cycles between crypto, ds, kv, pal and service, and one reached into a private header.

flowchart TD
  research --> node
  research --> endpoints
  js --> endpoints
  js --> indexing
  node --> service
  endpoints --> service
  indexing --> kv
  service --> pal
  pal --> kv
  kv --> crypto
  crypto --> ds
  ds --> threading
  ds -. "json.h: base64" .-> crypto
  crypto -. "sha256_hash.h: BlitSerialiser" .-> service
  pal -. "uvm_endorsements.h: DID, Feed" .-> service
  research -. "create_tx_claims_digest.h" .-> private_kv
  endpoints["endpoints<br/>now with endpoint.h, endpoint_context.h,<br/>common_auth_policies.h"]
  kv["kv<br/>now with tx.h"]
  private_kv["src/kv (private)"]
  classDef private stroke-dasharray:3 3
  class private_kv private
  linkStyle 12,13,14,15 stroke:#d1242f,stroke-width:2px,stroke-dasharray:5 5
Loading

Implementation summary

Public headers are checked as their own layer, not as part of the src/ component of the same name. A component's interface can legitimately be used by components that its implementation depends on: ccf/node_context.h exposes the indexing interface, while src/indexing depends on node. The checker now also requires that:

  • public headers only include public headers, quoted or angled, and
  • public header components form no cycle. Each directory under include/ccf/ is a component, and each top-level header is its own. There are no exceptions or ownership overrides.

Since public headers cannot include private ones and the src/ policy is acyclic, no cycle can span the two layers, so the whole include graph between components is acyclic.

Headers that belonged to a component but lived at the top level move into it:

Before After
ccf/tx.h ccf/kv/tx.h
ccf/endpoint.h ccf/endpoints/endpoint.h
ccf/endpoint_context.h ccf/endpoints/endpoint_context.h
ccf/common_auth_policies.h ccf/endpoints/common_auth_policies.h
ccf/crypto/base64.h ccf/ds/base64.h

The remaining cycles are broken as follows:

  • base64 (ds -> crypto): the functions are now ccf::ds::b64_from_raw, b64url_from_raw, raw_from_b64 and raw_from_b64url, declared in ccf/ds/base64.h. json.h must keep encoding std::vector<uint8_t> as base64, so ds has to own the declaration. The implementation stays in src/crypto/base64.cpp, as it uses the OpenSSL wrappers, which src/ds may not depend on.
  • BlitSerialiser<Sha256Hash> (crypto -> service): the specialisation moves next to its primary template in ccf/kv/serialisers/blit_serialiser.h, so ccf/crypto/sha256_hash.h no longer includes ccf/service/map.h.
  • UVM endorsements (pal -> service): ccf/pal/uvm_endorsements.h only used the ccf::DID and ccf::Feed aliases of std::string from ccf/service/tables/uvm_endorsements.h. It now uses std::string and no longer includes the table header, which is unchanged.
  • Indexing strategies (indexing -> node_context.h -> indexing): ccf/indexing/strategies/seqnos_by_key_bucketed.h forward-declares ccf::AbstractNodeContext, which it only takes by reference, instead of including ccf/node_context.h.
  • create_tx_claims_digest.h: includes ccf/claims_digest.h and ccf/kv/tx.h instead of the private header.

ccf/kv/tx.h includes what it uses, and no longer provides the typed KV maps. It gains ccf/kv/abstract_handle.h, which it always needed: it destroys a std::unique_ptr<AbstractHandle> of a forward-declared type, and only compiled because the full type arrived transitively.

Two changes under src/ follow from the moves:

  • NetworkTables::js_endpoints is removed. It was unused, and was the only reason src/service included endpoint.h, which is now in endpoints, a component that depends on service.
  • rust may depend on endpoints in source-dependencies.json, as src/rust/app_bridge.cpp includes the auth policies header.

The two scans share their include parsing, and report source and public violations together, with one include per edge of a cycle. On main, the checker reports:

Public headers include private headers:
  include/ccf/research/create_tx_claims_digest.h:5: #include <kv/kv_types.h>
Public header components form a cycle: ccf/crypto -> ccf/service -> ccf/entity_id.h -> ccf/kv -> ccf/byte_vector.h -> ccf/ds -> ccf/crypto
  include/ccf/crypto/sha256_hash.h:6: #include "ccf/service/map.h"
  include/ccf/service/tables/members.h:7: #include "ccf/entity_id.h"
  include/ccf/entity_id.h:6: #include "ccf/kv/serialisers/blit_serialiser.h"
  include/ccf/kv/serialisers/serialised_entry.h:4: #include "ccf/byte_vector.h"
  include/ccf/byte_vector.h:5: #include "ccf/ds/siphash.h"
  include/ccf/ds/json.h:4: #include "ccf/crypto/base64.h"

Validated locally:

  • A Debug build of all 347 targets passes (clang 21), including the samples and unit test binaries.
  • The 66 unit tests pass, as do the e2e_logging, e2e_basic_rust and programmability_and_jwt e2e tests. The rest of the e2e suite is left to CI.
  • includes-checks.sh passes without overrides, and reported the service -> endpoints and rust -> endpoints dependencies described above before they were resolved.
  • The Sphinx build passes with --fail-on-warning.
  • The clang-format 18.1.8, black, prettier, doc newline, release notes, ASCII, copyright, TODO and built-in maps checks pass.

Safety and compatibility

Runtime behaviour does not change. Every moved declaration and definition is otherwise unchanged, so there is no wire, ledger, KV or JSON format change, and no effect on mixed-version operation or recovery. Sha256Hash keys are still serialised as hex, and byte vectors as base64 in JSON. The removed NetworkTables::js_endpoints member was never read; the public:ccf.gov.endpoints table is unaffected.

This is a source-breaking change for C++ applications, intended for 8.0 only and not for backport to 7.x. Applications must:

  • update the includes in the table above,
  • rename ccf::crypto:: base64 calls to ccf::ds::, and
  • include ccf/service/map.h, ccf/node_context.h or ccf/service/tables/uvm_endorsements.h directly where they obtained them transitively through ccf/kv/tx.h, ccf/crypto/sha256_hash.h, seqnos_by_key_bucketed.h or ccf/pal/uvm_endorsements.h.

The 7.x to 8.x migration guide lists these replacements, and the change description links to it. ccf/app_interface.h still provides the endpoint registry, authentication policies and transaction types; the samples needed only the include renames.

Losing transitive includes cannot silently change behaviour here. The headers that some includers lose contain only two specialisations, fmt::formatter<ccf::ByteVector> and BlitSerialiser<Sha256Hash>, and each is in the header that declares its type or its primary template. Code that can name the type still sees the specialisation; anything else fails to compile.

The source dependency checker only scanned src/, so public headers under
include/ccf could still include each other in cycles between components,
and could include private headers with angle brackets.

Break the three cycles by moving declarations down, without renaming
anything: the base64 declarations to ccf/ds/base64.h, the
BlitSerialiser<Sha256Hash> specialisation next to its primary template,
and the DID and Feed aliases to ccf/pal/uvm_endorsements.h. Assign
endpoint.h, endpoint_context.h and common_auth_policies.h to endpoints,
tx.h to kv, and ccf/indexing/strategies to its own component.

Keep ccf/tx.h providing the typed KV maps, and stop
ccf/research/create_tx_claims_digest.h from including the private
kv/kv_types.h.

The checker now treats public headers as their own layer: they must only
include public headers, and their components must not form a cycle.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings September 30, 2026 21:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Four new changelog entries lack the required #8478 pull-request reference.

Review effort: Balanced
Findings: 2 Low severity

Open (2)
What changed in this PR

Extends dependency enforcement to public headers and removes existing component cycles without changing runtime behavior.

Changes:

  • Adds public-header dependency and private-include validation.
  • Relocates declarations and adds compatibility includes.
  • Updates release notes and checker documentation.

Custom instructions used: .github/copilot-instructions.md, .github/instructions/reviewing.instructions.md, .github/instructions/changelog.instructions.md, and the formatting/testing skills.

File Description
.github/​skills/​formatting-and-linting/​SKILL.md Updates include-check coverage.
CHANGELOG.md Documents header and dependency changes.
include/​ccf/​crypto/​base64.h Forwards to the new DS declaration header.
include/​ccf/​crypto/​sha256_hash.h Removes the KV dependency.
include/​ccf/​ds/​base64.h Adds relocated base64 declarations.
include/​ccf/​ds/​json.h Uses the lower-layer base64 header.
include/​ccf/​kv/​serialisers/​blit_serialiser.h Relocates the SHA-256 specialization.
include/​ccf/​pal/​uvm_endorsements.h Owns DID and Feed aliases.
include/​ccf/​research/​create_tx_claims_digest.h Replaces a private include.
include/​ccf/​service/​tables/​uvm_endorsements.h Imports aliases from PAL.
include/​ccf/​tx.h Makes required and compatibility includes explicit.
scripts/​check-source-dependencies.py Checks public-header layering and cycles.
scripts/​includes-checks.sh Describes expanded dependency checks.
scripts/​source-dependencies.json Adds public component ownership overrides.
src/​node/​internal_tables_access.h Adds its direct table dependency.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread CHANGELOG.md Resolved
Comment thread CHANGELOG.md Resolved
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ccf/tx.h used to provide ccf/kv/unit_value.h transitively, through
ccf/service/map.h. Include it with the other typed KV containers, so that
ccf/tx.h and its includers only lose the ccf::Service* aliases, and list
ccf::ServiceUnit among them in the changelog.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@achamayou
Amaury Chamayou (achamayou) marked this pull request as ready for review November 19, 2026 14:22
Copilot AI balanced review requested due to automatic review settings November 19, 2026 14:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are blocking the public-header dependency cleanup.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

8.x is a major release, so public headers no longer need to stay source
compatible. Move tx.h to ccf/kv/, and endpoint.h, endpoint_context.h and
common_auth_policies.h to ccf/endpoints/, which removes the ownership
overrides from the public header dependency check. Declare the base64
functions in ccf::ds, remove ccf/crypto/base64.h, and stop ccf/kv/tx.h
from including the typed KV maps for its includers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5506285a45b24ad086e34660c090a7ab
Copilot AI balanced review requested due to automatic review settings November 19, 2026 18:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The public-header scan silently ignores unresolved ccf/_private includes, allowing installed private headers to bypass the public/private dependency check.

1 open issue

Review effort: Balanced

include_path, path, root, quoted, generated_headers
)
if included_file is None:
if is_internal(include_path):

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants