Template cache key ignores gpt_bootstrap.js and other Rust-inlined head scripts #1198

Description

With [creative_opportunities] assembly_mode = "esi", the Fastly adapter stores the transformed document in a shared template cache. The stored bytes include every integration head insert. Several of those inserts are programs compiled into the binary:

• gpt_bootstrap.js, 894 lines, inlined with include_str!.
• Rust string literals for the Prebid, Didomi, DataDome and Sourcepoint initializers (including Sourcepoint's window._sp_ property trap) and the GPT enable and install flags.

The template cache key's template_fingerprint covers only the tsjs bundle hash and the serialized settings. The settings values inside these programs are covered. The program code is not.

A new binary that changes any of these programs, with the same settings and bundles, computes the same key. It serves templates stored by the previous binary, with the previous head code, until they expire, are evicted or purged, or someone bumps TEMPLATE_SCHEMA_VERSION. The version's doc comment asks for a bump on any change to what the transform emits. That rule is manual. The version has stayed at 4 since the cache landed in #1013 (squash commit d97bda6, 2026-08-22).

This has existed since #1013. It affects only Fastly deployments that opt into ESI assembly (the default is inline), and only requests eligible for a shared template.

Steps to reproduce

Add this temporary probe test inside mod template_cache_end_to_end_tests in crates/trusted-server-core/src/publisher.rs. It prints, so do not commit it.

#[tokio::test]
async fn probe_gpt_template_key_and_bytes() {
    let stub = Arc::new(StubHttpClient::new());
    let cache = Arc::new(MemoryTemplateCache::default());
    let services = services(Arc::clone(&stub), Arc::clone(&cache));
    let mut settings = settings_with_mode("esi");
    settings
        .integrations
        .insert("gpt".to_string(), serde_json::json!({ "enabled": true }));
    queue_shareable_html(&stub);
    println!(
        "PROBE template_fingerprint={} schema_version={}",
        template_fingerprint(&settings),
        crate::platform::TEMPLATE_SCHEMA_VERSION
    );

    let _ = run(&Arc::new(settings), &services, navigation_request()).await;

    let key = stored_cache_keys(&cache)
        .pop()
        .expect("should store one template");
    let entries = cache.entries.lock().expect("should lock entries");
    let body = &entries.get(&key).expect("should find the stored template").body;
    let text = String::from_utf8_lossy(body);
    println!(
        "PROBE stored_key={key}\nPROBE template_len={} template_sha256={}\n\
         PROBE template_contains_full_gpt_bootstrap_js={} \
         template_contains_edit_marker={}",
        body.len(),
        hex::encode(<sha2::Sha256 as sha2::Digest>::digest(body)),
        text.contains(include_str!("integrations/gpt_bootstrap.js")),
        text.contains("__tsjs_probe_bootstrap_edit"),
    );
}

Run it, append one statement to the bootstrap, and run it again. Revert the edit afterwards.

cargo test -p trusted-server-core --target wasm32-wasip1 --lib -- \
  publisher::tests::template_cache_end_to_end_tests::probe_gpt_template_key_and_bytes --exact --nocapture
printf 'window.__tsjs_probe_bootstrap_edit=1;\n' >> crates/trusted-server-core/src/integrations/gpt_bootstrap.js
cargo test -p trusted-server-core --target wasm32-wasip1 --lib -- \
  publisher::tests::template_cache_end_to_end_tests::probe_gpt_template_key_and_bytes --exact --nocapture

Observed at a4e01eb, before the edit:

PROBE template_fingerprint=249bc0e3161ced98eaf0206e61eb7d06271abe385801e54cf311f93b9ca7c14c schema_version=4
PROBE stored_key=ts-template-cache-v4-e39ba33be7d991e1fa9ac2a142e7158d3af087651500245a4ff9042a3c3d85bd
PROBE template_len=33977 template_sha256=f1c1da2a3ebfd8db964ffc1f22aa641e463f8cd3c820be6a4bf8633459b5212d
PROBE template_contains_full_gpt_bootstrap_js=true template_contains_edit_marker=false

After the edit:

PROBE template_fingerprint=249bc0e3161ced98eaf0206e61eb7d06271abe385801e54cf311f93b9ca7c14c schema_version=4
PROBE stored_key=ts-template-cache-v4-e39ba33be7d991e1fa9ac2a142e7158d3af087651500245a4ff9042a3c3d85bd
PROBE template_len=34015 template_sha256=7028acc5b1db3400fd58fbd17793811cad12ada44d6d4c383c7bde090aebeca6
PROBE template_contains_full_gpt_bootstrap_js=true template_contains_edit_marker=true

The stored template contains the whole bootstrap. The two binaries use the same fingerprint and the same cache key for different bytes. The 38-byte length difference is exactly the appended line. The hash values depend on the locally built bundles, so compare the two runs with each other.

Expected behavior

A binary that changes any head program that ends up in cached templates uses different template cache keys (or otherwise stops reading older templates) without anyone having to remember a manual version bump.

Actual behavior

The key depends only on the request dimensions, the bundle hash, the serialized settings and TEMPLATE_SCHEMA_VERSION. After a deploy that changes only inline head code, readers keep getting the previous binary's head bytes until each entry expires.

Root cause

template_fingerprint has two inputs (crates/trusted-server-core/src/publisher.rs:2073-2087):

let mut hasher = sha2::Sha256::new();
hasher.update(
    trusted_server_js::concatenated_hash(&trusted_server_js::all_module_ids()).as_bytes(),
);
// ...
let canonical = serde_json::to_value(settings)
    .and_then(|value| serde_json::to_vec(&value))
    .expect("serializing typed settings should be infallible");
hasher.update(canonical);

The key field is documented as "Digest of every setting that can shape the transformed template plus the tsjs bundle" (platform/template_cache.rs:72-74). Code bytes owned by the binary are neither. The only guard for them is the manual rule at platform/template_cache.rs:25-37:

/// Bump on **any** change to what the transform emits. Without it a deploy reads
/// yesterday's template shape and assembles against markers that moved, which fails
/// as a rendering bug far from its cause rather than as a cache miss.
...
pub const TEMPLATE_SCHEMA_VERSION: u32 = 4;

The head inserts are prepended to <head> during the transform (html_processor.rs:317-347). The template that is stored is that transformed output (publisher.rs:1834, let bytes = output.into_inner();, stored at publisher.rs:1901). On a hit, the only validity check is the seam marker (publisher.rs:4868-4881). Head bytes are never re-derived, so stale head code passes.

Inline head programs and whether they reach cached templates:

History

• d97bda6 (Add shared root caching with hybrid ESI and streaming assembly #1013, 2026-08-22) created the template cache with TEMPLATE_SCHEMA_VERSION = 4 and edited gpt_bootstrap.js in the same commit. No templates predate it, and the version has not changed since (git log -S'TEMPLATE_SCHEMA_VERSION: u32 = ' a4e01eb55 finds only d97bda6).
• gpt_bootstrap.js has changed once since then. 6cae7f5 (Prevent competing GPT first impressions and resize PUC shells #1079, 2026-09-16) added 324 lines to it and removed 15, with no schema bump. The same commit also changed JS bundle sources. I built the bundles at 6cae7f5^ and 6cae7f5 with the same node_modules (lib/package-lock.json is unchanged between them): tsjs-core.js, tsjs-gpt.js and tsjs-prebid.js changed, and the other ten modules were identical. concatenated_hash(all_module_ids()) hashes every module, so the fingerprint moved and templates stored before that deploy were not reused. The bootstrap change was masked by the bundle change. It was not handled.
• TS adInit overwrites publisher ads and APS replacements remain 1x1 #1078, the issue Prevent competing GPT first impressions and resize PUC shells #1079 fixed, notes that "a bundle-only change would leave the fallback path broken" and asks to "Mirror the same ownership contract in gpt_bootstrap.js". A follow-up that corrected only the bootstrap would have hit this bug.
• Rust-literal programs: git log -L over each head-insert function range, from d97bda6 to a4e01eb, shows changes only in Prebid, from dbc1012 (Add configuration-driven OpenRTB auction providers #1016) and 9558943 (Add managed LiveRamp RampID integration #1054). Both refactored the payload code, and both also changed crates/trusted-server-js/lib/src/integrations/prebid/index.ts in the same commit. Add configuration-driven OpenRTB auction providers #1016 moved the literals into free functions without changing their bytes. I found no change to the GPT, Didomi, DataDome, Sourcepoint or GPT-diagnostics ranges.
• I found no occurrence of stale head code reaching readers. The gap is latent.
• The same manual rule is the only guard for any other Rust-side change to template bytes, such as attribute rewriters, Next.js and GTM script rewriting, or script tag formatting. Six commits since the cache landed changed core HTML-transform or integration code with no JS bundle change in the same commit and no schema bump: a4e01eb (Make body hold parser-aware and stream Next.js processing #1135), d704d0a (Forward trusted geo to Didomi notice loaders #1127), 125ed24 (Add native secret-store config resolution #1036), e67fed3 (Harden the Google Tag Manager proxy's upstream and tag selection #1109), 22fbbf2 (Add JavaScript asset proxy integration #742) and b7fcb5d (Document the config template and harden config validation #870). I did not check whether each one changed emitted bytes for an unchanged configuration.

Impact

Only Fastly deployments with assembly_mode = "esi" are affected. That mode is opt-in and marked experimental; the default is inline (creative_opportunities.rs:209-220). Within those deployments, only requests eligible for a shared template are affected.

How long stale head code can be served after a deploy:

• A stored template lives for the smaller of the origin's remaining shared freshness and template_cache_max_age_seconds (publisher.rs:6253-6335, capped at 6331). The setting defaults to 60 seconds and accepts 1 to 86,400 (creative_opportunities.rs:19-20). The example config suggests 1200 (trusted-server.example.toml:357), and Increase template_cache_max_age_seconds to 1200s #1087 (closed as completed) asked to raise the configured value to 1200 seconds because "invalidation will force a refetch".
• A hit only reads the entry, so it does not extend the lifetime (trusted-server-adapter-fastly/src/template_cache.rs:164-188). Stale entries read as misses (same file, 76-78). Earlier eviction can shorten the window.
• So each URL variant can serve the old head for up to its remaining lifetime: at most 60 seconds by default, 20 minutes at 1200 seconds, and up to 24 hours at the maximum.
• While old and new versions both serve traffic during a rollout, each can read templates the other stored, because the keys are identical.

What readers see: the old head program, the current bundle, and the current per-request seam. The seam script from the running binary calls tsjs.scheduleInitialAdInit (publisher.rs:5788-5797), which the cached bootstrap defines (gpt_bootstrap.js:293) until the bundle replaces it. Suppose a release changes that contract, or the bootstrap's fallback first-impression behavior (the kind of change #1078 required), without also changing a bundle. Readers then get the new seam with the old bootstrap for the whole window.

Mitigations at a4e01eb:

• Bump TEMPLATE_SCHEMA_VERSION in the release (manual).
• Purge the Fastly surrogate key ts-template, which the adapter attaches at insert (trusted-server-adapter-fastly/src/template_cache.rs:141, 255-258). Trusted Server has no production caller of purge_all at a4e01eb, so this means a purge through Fastly's own tools. Open PR Split origin shareability from template eligibility, and add the readthrough evidence gate #1169 adds ts cache purge.

Proposed fix

Fold a build-time digest of the core crate's sources into template_fingerprint.

• In crates/trusted-server-core/build.rs, which is a stub today, hash every file under src/ in sorted relative-path order, hashing path and bytes. Emit cargo:rerun-if-changed=src and write pub(crate) const TEMPLATE_BUILD_DIGEST: &str = "<hex>"; to OUT_DIR.
• include! that file and add hasher.update(TEMPLATE_BUILD_DIGEST.as_bytes()); to template_fingerprint.

This covers gpt_bootstrap.js, every Rust-literal head program, and every other Rust change to emitted template bytes, with no list to maintain. The cost is over-invalidation: any core source change cold-starts templates at deploy. That is one fill per URL variant, the same cost as any JS bundle change today, and the key doc already states that over-invalidating is safe. There is no runtime cost beyond hashing 64 extra bytes.

Keep TEMPLATE_SCHEMA_VERSION for the stored entry format (marker and metadata layout), and update its doc to say that emitted-byte changes no longer need a manual bump. Dependency upgrades that change serialization (for example lol_html) are still not covered. Hashing Cargo.lock into the digest, when present, would cover them.

Alternatives, if the over-invalidation is unwanted:

• Hash only the inline program sources through a registered list, with a test that fails when a new include_str! asset or head injector is not registered. This is narrower but needs upkeep.
• Add only GPT_BOOTSTRAP_JS to the fingerprint. This is a one-line change that covers the largest and most frequently changed program, but leaves the Rust literals uncovered.
• Add a CI test that pins a digest of the inline program sources and fails until TEMPLATE_SCHEMA_VERSION and the pinned digest are updated. This keeps the runtime unchanged but still depends on a human bump.

Related direction: #855 proposes build-time tsjs hashes. The same build step could also replace the runtime concatenated_hash call in the fingerprint.

Compatibility: the first deploy with the fix changes every fingerprint once, which costs one cold fill per URL variant. No config or storage migration is needed.

Done when

• template_fingerprint includes a build-time digest that changes whenever gpt_bootstrap.js or any other source that emits template bytes changes.
• A unit test shows that two different build digests with identical settings produce different fingerprints (through a helper that takes the digest as a parameter).
• The probe procedure above prints a different stored_key after the one-line bootstrap edit.
• The TEMPLATE_SCHEMA_VERSION doc states what still requires a manual bump.

Affected area

HTML processing / JS injection

Version

main at a4e01eb

Related

• Add shared root caching with hybrid ESI and streaming assembly #1013 (merged, d97bda6): added template_fingerprint, the shared template cache and TEMPLATE_SCHEMA_VERSION = 4.
• Prevent competing GPT first impressions and resize PUC shells #1079 (merged, 6cae7f5): changed gpt_bootstrap.js without a schema bump. Masked by bundle changes in the same commit.
• TS adInit overwrites publisher ads and APS replacements remain 1x1 #1078 (closed): required the bootstrap to change in step with the bundle.
• Increase template_cache_max_age_seconds to 1200s #1087 (closed): asked to raise the configured template lifetime to 1200 seconds, relying on invalidation to force a refetch.
• Split origin shareability from template eligibility, and add the readthrough evidence gate #1169 (open PR): bumps TEMPLATE_SCHEMA_VERSION to 5 and adds purge tooling. A mitigation lever, not a fix.
• Per-request fixed cost: release profile, settings pipeline trim, build-time tsjs hashes #855 (open): build-time tsjs hashes. The same mechanism fits this fix.
• Reduce Fastly adapter per-request fixed cost: defer KV/geo work, cache settings/registry/template fingerprint #1192 (open) and Reduce time-to-first-byte across the Fastly adapter #1193 (open PR): cache the fingerprint per warm instance. They would cache the same incomplete value.
• Define integrations crate split and ordered configuration #1194 (open PR): its design spec proposes a document fingerprint over inline template bytes. This bug exists on main today regardless.
• Companion issue Template cache keys and config pushes vary per load when allowed_context_keys has 2+ keys #1197: the same fingerprint is nondeterministic when allowed_context_keys has two or more entries.