Fetch pages and assets through a source module

.github/actions/setup-integration-test-env/action.yml

@@ -74,7 +74,7 @@ runs:
       if: ${{ inputs.build-wasm == 'true' }}
       shell: bash
       env:
-        TRUSTED_SERVER__PUBLISHER__ORIGIN_URL: http://127.0.0.1:${{ inputs.origin-port }}
+        TRUSTED_SERVER__SOURCE__ORIGIN__URL: http://127.0.0.1:${{ inputs.origin-port }}
         TRUSTED_SERVER__PUBLISHER__PROXY_SECRET: integration-test-proxy-secret
         TRUSTED_SERVER__EC__PASSPHRASE: integration-test-ec-secret-padded-32
         TRUSTED_SERVER__EC__PARTNERS: >-
@@ -86,7 +86,7 @@ runs:
       if: ${{ inputs.build-axum == 'true' }}
       shell: bash
       env:
-        TRUSTED_SERVER__PUBLISHER__ORIGIN_URL: http://127.0.0.1:${{ inputs.origin-port }}
+        TRUSTED_SERVER__SOURCE__ORIGIN__URL: http://127.0.0.1:${{ inputs.origin-port }}
         TRUSTED_SERVER__PUBLISHER__PROXY_SECRET: integration-test-proxy-secret
         TRUSTED_SERVER__EC__PASSPHRASE: integration-test-ec-secret-padded-32
         TRUSTED_SERVER__PROXY__CERTIFICATE_CHECK: "false"
@@ -117,7 +117,7 @@ runs:
       if: ${{ inputs.build-cloudflare == 'true' }}
       shell: bash
       env:
-        TRUSTED_SERVER__PUBLISHER__ORIGIN_URL: http://127.0.0.1:${{ inputs.origin-port }}
+        TRUSTED_SERVER__SOURCE__ORIGIN__URL: http://127.0.0.1:${{ inputs.origin-port }}
         TRUSTED_SERVER__PUBLISHER__PROXY_SECRET: integration-test-proxy-secret
         TRUSTED_SERVER__EC__PASSPHRASE: integration-test-ec-secret-padded-32
         TRUSTED_SERVER__PROXY__CERTIFICATE_CHECK: "false"

.github/workflows/test.yml

@@ -190,7 +190,7 @@ jobs:
         # rejects (falling back to the generic 503 router). This proves the
         # documented Spin artifact can boot, not just compile.
         env:
-          TRUSTED_SERVER__PUBLISHER__ORIGIN_URL: http://127.0.0.1:8080
+          TRUSTED_SERVER__SOURCE__ORIGIN__URL: http://127.0.0.1:8080
           TRUSTED_SERVER__PUBLISHER__PROXY_SECRET: integration-test-proxy-secret
           TRUSTED_SERVER__EC__PASSPHRASE: integration-test-ec-secret-padded-32
           TRUSTED_SERVER__PROXY__CERTIFICATE_CHECK: "false"

CHANGELOG.md

@@ -9,6 +9,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- **Breaking:** Where content comes from is a source module. `[source]` selects the publisher's, and each `[[proxy.asset_routes]]` entry selects its own in a `source` table of the same shape, with `module` naming the source and its settings in the table beneath. Core supplies `origin`, an HTTP origin, and a module a build links can offer another through `IntegrationRegistrationBuilder::with_source_module`. A source answers for each request where that request's content comes from, and core fetches, caches, rewrites the origin host and follows the redirect rule from that answer. `[source]` is required. The origin settings move into the `origin` table, and a document that still writes one where it was is refused at startup naming the new place. An environment variable that set the address is now `TRUSTED_SERVER__SOURCE__ORIGIN__URL`. The old name is refused where the build reads variables, and `ts config push` ignores it, as it ignores any variable for a key the document does not hold. A path in an asset route's `url` names a folder the target path is added under, where a path was refused, a target path with a `.` or `..` segment under a folder is answered `400`, and a route applies `directory_index` as the publisher does. Stored pages and the publisher's stored assets are fetched again after the upgrade, because their keys hold a code worked out from the settings, but an asset route's stored copies keep their keys. Where `host_header` is set, the page pipeline and every middleware are now told that `Host` as the page's origin host, where they were told the address's host, so the page's own addresses under that name are rewritten to the request's host. **For crates outside core**, `handle_asset_proxy_request` takes the route's source, which `IntegrationRegistry::route_source` gives, and `Publisher::origin_host`, `origin_host_header` and `origin_query_is_stripped` are replaced by `Settings::origin_source` and the `SourceLocation` a source answers.
+
+  | Previous                                  | Now                                       |
+  | ----------------------------------------- | ----------------------------------------- |
+  | `[publisher] origin_url`                  | `[source.origin] url`                     |
+  | `[publisher] origin_host_header_override` | `[source.origin] host_header`             |
+  | `[publisher] origin_auth`                 | `[source.origin] auth`                    |
+  | `[publisher] directory_index`             | `[source.origin] directory_index`         |
+  | `[[proxy.asset_routes]] origin_url`       | `[proxy.asset_routes.source.origin] url`  |
+  | `[[proxy.asset_routes]] auth`             | `[proxy.asset_routes.source.origin] auth` |
+
 - The mapping of origin URLs in response headers (#1128) follows a path in `origin_url`. A URL on the origin is mapped to the serving host only when it is in that folder, and it loses the folder, so the reader of a site served from a folder of object storage is sent to the right address. A URL on the origin's host outside the folder is unchanged, because no address on the serving host leads there, and a CSP source with no path still gains the serving host beside it. A reader of a publisher whose request addressing hook removes a path prefix is sent to an address without the prefix, which a Fastly service whose `__KEY` selector carries `{host}` answers `421` if every selector value misses.
 - **Breaking:** a path in `publisher.origin_url` is kept, and the request path is added after it with one `/`. The path was discarded before, so a deployment whose `origin_url` has one now sends each request to the folder the path names, and answers `400` to a request path with a `.` or `..` segment, with or without `origin_auth`. A deployment whose `origin_url` has no path sends its requests as before. Before upgrading, check that the origin serves the site at that path, or remove the path.
 - **Breaking:** the check that closes `/_ts/admin` and `/admin/keys` on every adapter now also answers `404` to a request path still changing after four rounds of percent-decoding and to a path that reaches either once an escape that is not valid UTF-8 is replaced. The publisher received both before, and no setting turns the check off.

crates/auction/prebid-server/src/lib.rs

@@ -3046,10 +3046,15 @@ mod tests {
 [auction]
 modules = ["prebid"]
 
+[source]
+module = "origin"
+
+[source.origin]
+url = "https://origin.test-publisher.com"
+
 [publisher]
 domain = "test-publisher.com"
 cookie_domain = ".test-publisher.com"
-origin_url = "https://origin.test-publisher.com"
 proxy_secret = "test-secret"
 
 [ec]

crates/auction/prebid/src/lib.rs

@@ -1743,10 +1743,15 @@ mod tests {
 [auction]
 modules = ["prebid"]
 
+[source]
+module = "origin"
+
+[source.origin]
+url = "https://origin.test-publisher.com"
+
 [publisher]
 domain = "test-publisher.com"
 cookie_domain = ".test-publisher.com"
-origin_url = "https://origin.test-publisher.com"
 proxy_secret = "test-secret"
 
 [ec]

crates/cmp/inmobi/src/lib.rs

@@ -1150,7 +1150,9 @@ mod tests {
                 "cmp_id = {value}\n{}",
                 html_entry(None, &[MODULE])
             ));
-            settings.publisher.origin_url = "https://origin.publisher.example".to_owned();
+            settings.set_origin_source(&trusted_server_core::source::OriginSettings::new(
+                "https://origin.publisher.example",
+            ));
             settings
         };
         let load = |settings: &Settings| {

crates/tag/google-tag-manager/src/lib.rs

@@ -3122,10 +3122,15 @@ j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
     #[test]
     fn test_config_parsing() {
         let toml_str = r#"
+[source]
+module = "origin"
+
+[source.origin]
+url = "https://origin.test-publisher.com"
+
 [publisher]
 domain = "test-publisher.com"
 cookie_domain = ".test-publisher.com"
-origin_url = "https://origin.test-publisher.com"
 proxy_secret = "test-secret"
 
 [ec]
@@ -3157,10 +3162,15 @@ assume_single_jurisdiction = true
     #[test]
     fn a_block_for_an_unnamed_integration_is_refused() {
         let toml_str = r#"
+[source]
+module = "origin"
+
+[source.origin]
+url = "https://origin.test-publisher.com"
+
 [publisher]
 domain = "test-publisher.com"
 cookie_domain = ".test-publisher.com"
-origin_url = "https://origin.test-publisher.com"
 proxy_secret = "test-secret"
 
 [ec]

crates/testing/seam-probe/src/lib.rs

@@ -1131,10 +1131,15 @@ mod tests {
     fn settings_with_probe(body: &str) -> Settings {
         Settings::from_toml(&format!(
             r#"
+                [source]
+                module = "origin"
+
+                [source.origin]
+                url = "https://origin.test-publisher.example.com"
+
                 [publisher]
                 domain = "test-publisher.example.com"
                 cookie_domain = ".test-publisher.example.com"
-                origin_url = "https://origin.test-publisher.example.com"
                 proxy_secret = "seam-probe-test-proxy-secret"
 
                 [ec]
@@ -1374,10 +1379,15 @@ mod tests {
     fn validate_reports_not_selected_without_a_module_entry() {
         let settings = Settings::from_toml(
             r#"
+                [source]
+                module = "origin"
+
+                [source.origin]
+                url = "https://origin.test-publisher.example.com"
+
                 [publisher]
                 domain = "test-publisher.example.com"
                 cookie_domain = ".test-publisher.example.com"
-                origin_url = "https://origin.test-publisher.example.com"
                 proxy_secret = "seam-probe-test-proxy-secret"
 
                 [ec]

crates/testing/testlight/src/lib.rs

@@ -635,6 +635,9 @@ mod tests {
         </head><body></body></html>"#;
 
         let mut settings = Settings::default();
+        settings.set_origin_source(&trusted_server_core::source::OriginSettings::new(
+            "https://origin.example.com",
+        ));
         let shim_src = "https://edge.example.com/static/testlight.js".to_owned();
         settings
             .insert_module_config(

crates/trusted-server-adapter-axum/src/app.rs

@@ -945,10 +945,15 @@ mod tests {
     /// inject, with the `[ec.acme]` block that module's settings live in.
     /// `acme` is a fictional vendor key.
     const UNINJECTED_MODULE_TOML: &str = r#"
+        [source]
+        module = "origin"
+
+        [source.origin]
+        url = "https://origin.test-publisher.example.com"
+
         [publisher]
         domain = "test-publisher.example.com"
         cookie_domain = ".test-publisher.example.com"
-        origin_url = "https://origin.test-publisher.example.com"
         proxy_secret = "unit-test-proxy-secret"
 
         [ec]
@@ -970,10 +975,15 @@ mod tests {
     fn a_requests_services_carry_the_publishers_prefixes() {
         let settings = Settings::from_toml(
             r#"
+                [source]
+                module = "origin"
+
+                [source.origin]
+                url = "https://origin.publisher.example"
+
                 [publisher]
                 domain = "www.publisher.example"
                 cookie_domain = ".publisher.example"
-                origin_url = "https://origin.publisher.example"
                 proxy_secret = "fictional-proxy-secret-32-bytes-ok"
 
                 [geo]

crates/trusted-server-adapter-axum/src/middleware.rs

@@ -142,10 +142,15 @@ mod tests {
     fn settings_with_response_headers(headers: Vec<(&str, &str)>) -> Settings {
         let mut s = Settings::from_toml(
             r#"
+                [source]
+                module = "origin"
+
+                [source.origin]
+                url = "https://origin.test-publisher.example.com"
+
                 [publisher]
                 domain = "test-publisher.example.com"
                 cookie_domain = ".test-publisher.example.com"
-                origin_url = "https://origin.test-publisher.example.com"
                 proxy_secret = "unit-test-proxy-secret"
 
                 [ec]

crates/trusted-server-adapter-axum/tests/routes.rs

@@ -30,10 +30,15 @@ fn test_settings() -> trusted_server_core::settings::Settings {
 fn test_settings_with(extra: &str) -> trusted_server_core::settings::Settings {
     trusted_server_core::settings::Settings::from_toml(&format!(
         r#"
+            [source]
+            module = "origin"
+
+            [source.origin]
+            url = "https://origin.test-publisher.example.com"
+
             [publisher]
             domain = "test-publisher.example.com"
             cookie_domain = ".test-publisher.example.com"
-            origin_url = "https://origin.test-publisher.example.com"
             proxy_secret = "integration-test-proxy-secret"
 
             [ec]
@@ -674,10 +679,15 @@ async fn first_party_proxy_rebuild_is_routed() {
 /// inject, with the `[ec.acme]` block that module's settings live in.
 /// `acme` is a fictional vendor key.
 const UNINJECTED_MODULE_TOML: &str = r#"
+    [source]
+    module = "origin"
+
+    [source.origin]
+    url = "https://origin.test-publisher.example.com"
+
     [publisher]
     domain = "test-publisher.example.com"
     cookie_domain = ".test-publisher.example.com"
-    origin_url = "https://origin.test-publisher.example.com"
     proxy_secret = "integration-test-proxy-secret"
 
     [ec]
@@ -980,10 +990,15 @@ async fn the_sellers_module_makes_an_ads_txt_the_origin_does_not_have() {
 fn settings_refusing_every_crawler() -> trusted_server_core::settings::Settings {
     trusted_server_core::settings::Settings::from_toml(
         r#"
+            [source]
+            module = "origin"
+
+            [source.origin]
+            url = "https://origin.test-publisher.example.com"
+
             [publisher]
             domain = "test-publisher.example.com"
             cookie_domain = ".test-publisher.example.com"
-            origin_url = "https://origin.test-publisher.example.com"
             proxy_secret = "integration-test-proxy-secret"
 
             [ec]

crates/trusted-server-adapter-axum/tests/seam_probe.rs

@@ -58,10 +58,15 @@ assume_single_jurisdiction = true
     };
     Settings::from_toml(&format!(
         r#"
+            [source]
+            module = "origin"
+
+            [source.origin]
+            url = "https://origin.test-publisher.example.com"
+
             [publisher]
             domain = "test-publisher.example.com"
             cookie_domain = ".test-publisher.example.com"
-            origin_url = "https://origin.test-publisher.example.com"
             proxy_secret = "seam-probe-test-proxy-secret"
 
             [ec]
@@ -931,10 +936,15 @@ fn deploy_validation_accepts_a_modules_ad_server_only_with_its_builder() {
 fn settings_selecting_module(extra: &str) -> Settings {
     Settings::from_toml(&format!(
         r#"
+            [source]
+            module = "origin"
+
+            [source.origin]
+            url = "https://origin.test-publisher.example.com"
+
             [publisher]
             domain = "test-publisher.example.com"
             cookie_domain = ".test-publisher.example.com"
-            origin_url = "https://origin.test-publisher.example.com"
             proxy_secret = "seam-probe-test-proxy-secret"
 
             [geo]

crates/trusted-server-adapter-cloudflare/src/app.rs

@@ -1001,10 +1001,15 @@ mod tests {
     /// inject, with the `[ec.acme]` block that module's settings live in.
     /// `acme` is a fictional vendor key.
     const UNINJECTED_MODULE_TOML: &str = r#"
+        [source]
+        module = "origin"
+
+        [source.origin]
+        url = "https://origin.test-publisher.example.com"
+
         [publisher]
         domain = "test-publisher.example.com"
         cookie_domain = ".test-publisher.example.com"
-        origin_url = "https://origin.test-publisher.example.com"
         proxy_secret = "unit-test-proxy-secret"
 
         [ec]
@@ -1057,10 +1062,15 @@ mod tests {
     fn aps_profile_settings() -> Settings {
         let mut settings = Settings::from_toml(
             r#"
+                [source]
+                module = "origin"
+
+                [source.origin]
+                url = "https://origin.publisher.example"
+
                 [publisher]
                 domain = "publisher.example"
                 cookie_domain = ".publisher.example"
-                origin_url = "https://origin.publisher.example"
                 proxy_secret = "fictional-proxy-secret"
 
                 [ec]
@@ -1143,10 +1153,15 @@ mod tests {
     fn a_requests_services_carry_the_publishers_prefixes() {
         let settings = Settings::from_toml(
             r#"
+                [source]
+                module = "origin"
+
+                [source.origin]
+                url = "https://origin.publisher.example"
+
                 [publisher]
                 domain = "www.publisher.example"
                 cookie_domain = ".publisher.example"
-                origin_url = "https://origin.publisher.example"
                 proxy_secret = "fictional-proxy-secret-32-bytes-ok"
 
                 [geo]
@@ -1189,10 +1204,15 @@ mod tests {
     fn disabled_startup_accepts_dormant_multi_provider_auction_plan() {
         let mut settings = Settings::from_toml(
             r#"
+                [source]
+                module = "origin"
+
+                [source.origin]
+                url = "https://origin.publisher.example"
+
                 [publisher]
                 domain = "publisher.example"
                 cookie_domain = ".publisher.example"
-                origin_url = "https://origin.publisher.example"
                 proxy_secret = "fictional-proxy-secret"
 
                 [ec]
@@ -1214,10 +1234,15 @@ mod tests {
     fn startup_rejects_multi_provider_auction_plan() {
         let mut settings = Settings::from_toml(
             r#"
+                [source]
+                module = "origin"
+
+                [source.origin]
+                url = "https://origin.publisher.example"
+
                 [publisher]
                 domain = "publisher.example"
                 cookie_domain = ".publisher.example"
-                origin_url = "https://origin.publisher.example"
                 proxy_secret = "fictional-proxy-secret"
 
                 [ec]