Repository navigation
Fetch pages and assets through a source module - #1380
Open
jwrosewell wants to merge 40 commits into
Open
jwrosewell wants to merge 40 commits into
jwrosewell wants to merge 40 commits into
Conversation
…from Where content comes from was written twice, in [publisher] and in every asset route, and fetched by code that knew one kind of source. A source module now answers it. [source] selects the publisher's, and each [[proxy.asset_routes]] entry selects its own in a source table of the same shape. Core supplies origin, an HTTP origin, and a module a build links offers another through with_source_module. For each request a source answers a SourceLocation, being the address and folder, the Host to send, the signing and the index file. The publisher path and the asset routes fetch, cache, rewrite the origin host and follow the redirect rule from that answer, so a source that answers a different origin for each request is kept apart everywhere. The origin fields move into [source.origin] and [proxy.asset_routes.source.origin]. A document writing one where it was is refused naming the new place, and a document with no [source] is refused. A path in an asset route's url names a folder the target path is added under, a target path with a dot segment under a folder is answered 400, and a route applies directory_index as the publisher does. Stored page and asset keys are unchanged for an existing deployment. The configuration page masks the origin's url and host_header by default, and the signing secrets are declared at their new paths. Design: docs/superpowers/specs/2026-10-14-source-modules-design.md
Request addressing takes a prefix off the path and keeps it as a request extension, so a source can say where content comes from. A module was handed the continuing path only, so a module that names things after the tenant a path addressed, such as a prefix module naming cookies after the publisher, could not tell which one it was. ModuleRequest::of now reads the removed prefix and removed_prefix() gives it, with_removed_prefix sets it, and ResolvedRequest keeps it, so a held request still has it. A request addressing nothing carries none.
A route reads its request through its module call, and the call carried the request as resolved, the settings and the request state, but no evidence. A route that asks for the request as evidence, the client address and headers a module reads, was refused with "there is no request evidence where this module runs", so every such route answered 500 in production while its own tests, which build the context by hand, passed. handle_proxy now takes the evidence before the route takes the request, in the form the permission path reads it: the normalized client address, the headers after the caller's EC header is removed, and the address asked for. The test goes through handle_proxy, so it fails if the evidence is dropped again. (cherry picked from commit efad1ece83c3ec214e7d60c3e0754fb1053c5b41)
A document without [source] is refused since the source series, and the test built its registry from default settings.
The source series moved the origin's address out of [publisher], and the overlay ignores a variable for a key the document lacks, so the scripts that still set TRUSTED_SERVER__PUBLISHER__ORIGIN_URL left the template cache harness, the config by host harness and the integration runs on the placeholder origin, which the settings refuse.
A source may fetch through another host and send the publisher's own name as the Host, as for an origin behind a proxy at a different address. The page then names that Host's name and never the address's host. Core looked for the address's host, which the page does not carry, so it rewrote none of the page's own addresses, and every middleware was told the proxy's host as the page's origin. Core now hands the page pipeline and every middleware the Host the source sends as the page's origin host, for a page fetched for the request and for one served from the store alike. Where a source sends no other Host the two names are the same, so nothing else changes. The fetch itself still goes to the source's address. Checked: core's library tests pass natively, including a new test that fetches a page through another address and finds its own links pointed at the request's host, and cargo test-axum, test-cloudflare, test-spin and test-fastly pass. Clippy on core with -D warnings is clean.
…nges, and its limits Review round 1 found that the keys of stored pages and of the publisher's assets change with this series, that a page and every middleware are now told the Host the source sends as the page's origin host, and that the behavior the series adds to asset routes and to modules was missing from the design. The design now says all three, adds configuration, adapters, consent and privacy, failure handling and rollout, lists the paths with no test, records the spelling of core's source as a known gap and records the review round. The changelog entry no longer says stored keys are unchanged and now states the host change.
Round 2 approved the design with no blocking issues. It could not confirm that the per-request source test keeps no stored copies, and that holds. The test runs in the default assembly mode, which stores no page, with test services that select no cache, which store no asset, and their template cache is the unavailable one. The design names no design documents pull request, so no reference changed. The review log and the revision record carry the round.
The origin redirect guide sentences on redirect loops and a removed path prefix move to the Redirects paragraph under [source.origin], where this series holds the origin settings.
jwrosewell
added this pull request to stack #1401
October 15, 2026 17:56
jwrosewell
removed this pull request from stack #1401
October 15, 2026 20:51
This branch moves the origin's address from [publisher] origin_url to [source.origin] url, and the local test scripts set it there. The config push tests from the config by host change still set TRUSTED_SERVER__PUBLISHER__ORIGIN_URL, so every push they ran was refused for the example's insecure default origin. They now set TRUSTED_SERVER__SOURCE__ORIGIN__URL, as config_store_defaults.rs does.
The test of the InMobi module's CMP id range gave its settings a real origin by writing the publisher table's origin_url, which this branch moves to [source.origin] and no longer exposes, so the module's tests did not compile from here up. The test now sets the origin through Settings::set_origin_source, as the other tests here do.
The publisher's origin has moved from `[publisher] origin_url` to `[source.origin] url`, and settings refuse it in the old place. The Axum and Fastly tests of `/robots.txt` through the publisher path and the Cloudflare test of the attestation pages still wrote it in `[publisher]`, so each now selects the origin module in `[source]` and names the address there, as the other tests do.
This was referenced Oct 17, 2026
jwrosewell
added this pull request to stack #1408
October 17, 2026 09:05
handle_asset_proxy_request takes the route's source since sources became modules, and the test added with the failed load fix called it without one, so the core library tests did not compile. The test now passes origin_source_of(&route), as the other asset route tests do.
jwrosewell
added this pull request to stack #1418
October 19, 2026 12:18
The template cache key takes main's request_path_and_query, the reader's own path and query, and keeps this row's origin identity from the source module's location.
# Conflicts: # CHANGELOG.md # docs/guide/configuration.md
The header rewrite merged in main (#1238) names publisher.origin_url and publisher.origin_host_header_override in two warnings, a doc comment and the configuration guide, and its handler tests set those fields. This change moves both settings to source.origin.url and source.origin.host_header, so the warnings, the comment and the guide name the new settings and the tests set them through edit_origin, as the change's own tests do. Without the test change core's tests do not compile here. cargo test -p trusted-server-core --lib -- redirect response_header_rewrite passes 58 tests.
jwrosewell
removed this pull request from stack #1418
October 19, 2026 16:30
jwrosewell
added this pull request to stack #1421
October 19, 2026 17:38
jwrosewell
changed the title
jwrosewell
removed this pull request from stack #1421
October 19, 2026 18:29
jwrosewell
added this pull request to stack #1422
October 19, 2026 19:20
jwrosewell
marked this pull request as ready for review
October 19, 2026 20:23
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
crates/trusted-server-core/srcdocs/guidedocs/superpowerscrates/trusted-server-adapter-fastly/srccrates/trusted-server-adapter-cloudflare/srctrusted-server.example.tomlcrates/trusted-server-adapter-axum/testscrates/trusted-server-adapter-cloudflare/testscrates/trusted-server-adapter-spin/srccrates/trusted-server-adapter-axum/srccrates/trusted-server-adapter-spin/testscrates/trusted-server-integration-tests/testscrates/tag/google-tag-managercrates/testing/seam-probeCHANGELOG.mdcrates/trusted-server-integration-tests/srccrates/trusted-server-cli/srccrates/auction/prebid-servercrates/auction/prebidcrates/trusted-server-integration-tests/fixtures.github/actionscrates/cmp/inmobicrates/testing/testlightcrates/trusted-server-cli/testsscripts/template-cache-local-test.sh.github/workflowsscripts/attestation-local-test.shscripts/config-by-host-local-test.shscripts/integration-tests-browser.shscripts/integration-tests.shscripts/smoke-common.shCloses
Closes #1332
Spec
Spec: source-modules
Test plan
CI on this pull request at ade271c passed all 29 checks.
cargo test-fastly && cargo test-axumcargo clippy-fastly && cargo clippy-axumcargo fmt --all -- --checkcd crates/trusted-server-js/lib && npx vitest runcd crates/trusted-server-js/lib && npm run formatcd docs && npm run formatcargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1fastly compute serveChecklist
unwrap()in production code (clippy deniesunwrap_usedin CI)logmacros as AGENTS.md asks (notprintln!)On top of the middleware settings table (#1379). Issue: #1332.
What this adds
What changes.
A source module says where the publisher path and the asset routes fetch from.
[source]selects the publisher's source by name, and each[[proxy.asset_routes]]entry selects its own in a source table of the same shape. Core suppliesorigin, an HTTP origin, and a module a build links offers another throughwith_source_module.How it works.
SourceLocationfor each request, being the address and folder, theHostto send, the signing and the index file. The publisher path and the asset routes fetch, cache, rewrite the origin host and map its addresses in response headers from that answer, so a source that answers a different origin for each request is kept apart everywhere.[publisher]keeps who the publisher is. The origin fields leave it for[source.origin], wherehost_header,directory_indexandauthare optional. A document writing one where it was is refused naming the new place, and a document with no[source]is refused with the shape to write.[proxy.asset_routes.source]and the source's own table replace a route's own origin address and signing. A path in a route's url names a folder the target path is added under, a target path with a dot segment under a folder is answered 400, and a route appliesdirectory_indexas the publisher does.ModuleRequest::ofnow reads the removed prefix,removed_prefix()gives it, and a held request keeps it.Why
Where content comes from was written twice, in
[publisher]and in every asset route, and fetched by code that knew one kind of source. A deployment that wants a page from assets built into its binary, from a store its platform offers, or from a service of its own had no place for that code but core. One interface core declares, with the origin fetch as the source core supplies, gives each of those a crate of its own and keeps one table for who the publisher is and one for where the content comes from.What breaks
A
[publisher]table carryingorigin_url,origin_host_header_override,origin_auth,directory_indexor the origin query rule is refused at startup, with the[source]shape to write instead, and an asset route carrying its own origin address or signing is refused the same way. Every deployment's settings document changes shape, so the build and the documents go out together. The integration test environment and the attestation script setTRUSTED_SERVER__SOURCE__ORIGIN__URLin place of the previous variable, because the overlay ignores a variable for a key the document lacks and the old name would silently leave a build on the placeholder origin. Every stored page and every stored asset of the publisher is fetched from the origin again on its first request after the upgrade, because their keys hold a code worked out from the settings, which now write out a[source]section where[publisher]held the origin settings. An[inspect]showorhidepattern that names a moved path, such aspublisher.origin_url, matches no value in the new settings and is refused until it names the new path.The commits
[source.origin], throughTRUSTED_SERVER__SOURCE__ORIGIN__URL, because the settings overlay ignores a variable for a key the document lacks and the scripts' old variable left the test harnesses on the placeholder origin, which the settings refuse.Host, core now hands the page pipeline and every middleware that name as the page's origin host, so the page's own links are rewritten under the request's host. Where a source sends no otherHostthe two names are the same and nothing changes.handle_proxytakes the client address, the headers after the caller's Edge Cookie header is removed and the address asked for, so a route that asks for the request as evidence is no longer refused and answered 500.The other commits are two test fixes (d6d0da0 and 878941c), the design page and its review (6eabcc6 and 0f51b33), and merges of the series below. 315cdf0 gives the header rewrite merged in main (#1238) the source settings' names, in its warnings, its handler tests and the guide.
The design page is
docs/superpowers/specs/2026-10-14-source-modules-design.md, and the same page goes to the design documents pull request (#1342) so every design page stays in one place.What is not in this pull request
A source other than the HTTP origin. The interface and the selection are here, a source that answers another address is a module of its own, and a source that holds its content itself, with no network origin, is a later extension of the interface.