Repository navigation
Description
Describe the bug
Concurrent first requests on a freshly joined JS node can fail with HTTP 500 Exception: Unknown auth policy: jwt, even though jwt is a valid policy in the application's endpoint definition.
The ASAN job in Long Test run 10866 failed e2e_logging / js:worker-3 / test_long_lived_forwarding on 2026-10-07. At 00:32:43.577Z, POST /app/log/private returned:
{"error":{"code":"InternalError","message":"Exception: Unknown auth policy: jwt"}}The executed auth_policies_by_name implementation constructs an empty function-local static map, then populates it outside the static initializer using an unsynchronized empty() check and successive insertions. Another task worker can observe a nonempty but partially populated map before jwt is inserted. Concurrent insertion and lookup also access the container without synchronization.
This is a high-confidence explanation of the observed error, not an instrumented proof of the exact conflicting instruction schedule. No authentication bypass or exploitable memory-corruption path is established.
To Reproduce
Configure and build the failing revision with the ASAN Long Test flags below, then run:
cd build
./tests.sh --output-on-failure --timeout 1600 -R '^e2e_logging$' --no-tests=errorThe JS long-lived forwarding case creates a fresh node and starts five concurrent user clients. Its cold first-request ordering is the relevant trigger; an ordinary rerun is not guaranteed to reproduce the race. Add deterministic concurrent cold-start coverage for initialization and lookup.
Expected behavior
Every request observes a fully initialized named-policy registry. Valid configured policies must not intermittently produce HTTP 500 during concurrent first use.
Environment information
Main SHA 41d6c20c218666673415d258e7d9201e0891b8df, run attempt 1, ASAN job 112562527175, CTest 94 e2e_logging.
Azure Linux 3.0 container; Debug, LONG_TESTS=ON, SAN=ON, USE_SNMALLOC=OFF. The uploaded JS node configuration has worker_threads=1, which this revision normalizes to two task workers.
Additional context
Evidence is in the run's logs-asan artifact:
- CI job log lines 7072-7089 contain the response and assertion at
tests/e2e_logging.py:1665. js_e2e_logging_2/out:301-310records initial TLS sessions on task threads 1 and 2, followed by exception-driven connection closure.src/js/registry.cpp:995-1017constructs endpoint authentication policies during lookup; the app manifest listsjwtanduser_cert.- All three JS node stderr files are empty. The temporary node retires and shuts down normally after the client failure; no preceding node crash was found.
Current main was still the failing SHA at the audit on 2026-10-07. Open PR #8511 changes JS response conversion, not policy initialization. Open PRs #8445, #8478, and #8368 likewise do not synchronize this map. Closed #8256 concerns JWT refresh counters and FD suppressions, not this lookup failure. No exact matching open tracker was found.
Follow-up investigation (2026-10-07)
Assessment: A small initialization fix is supported. The source defect is demonstrated; attribution of the exact historical HTTP 500 remains a high-confidence inference because the CI instruction schedule was not captured.
Root cause and evidence
Only construction of the empty function-local static map is guarded. The empty() check and six insertions run after that guard has completed. A second caller can therefore observe a nonempty, incomplete registry before jwt is inserted; uncontrolled simultaneous insertion/lookup is also unsynchronized.
Controlled source-derived C++ witnesses on GCC 15.2.1 and Clang 21.1.8 reproduced two independent failures:
- Pause initialization after inserting
user_cert, then perform a concurrent lookup: the baseline returns nojwtpolicy. - Inject
bad_allocafter the first insertion: the baseline retains a one-entry registry, so subsequent calls do not retry population and still cannot resolvejwt.
Moving all insertions inside the guarded static initializer prevents partial publication and allows normal retry after an initialization exception. These witnesses use copied registry/lookup functions with stub policies and controlled scheduling; they are not real-header CCF unit tests or sanitizer/network passes.
The uploaded worker_threads=1 configuration is normalized to two task workers, so concurrent cold requests are possible. The original node artifacts show empty stderr and normal subsequent shutdown, not an antecedent crash. No authentication bypass or exploitable memory-corruption path was demonstrated.
Candidate fix
Fully populate the map in one function-local static initializer, using the existing policy singleton pointers and translation-unit-local linkage. Preserve exactly user_cert, member_cert, any_cert, jwt, user_cose_sign1, and no_auth, along with existing lookup failures and OR/AND semantics. Do not construct replacement policies, alter JWT validation, introduce a global request lock, or fall back to weaker authentication.
A const map is possible, but the installed helper currently returns a mutable reference. The essential initialization fix can retain that signature to avoid breaking external C++ callers; const qualification needs a separate source-compatibility decision.
Regression coverage and remaining work
- Exercise the real header under a controlled cold-start schedule, including initialization exceptions and singleton identity. Use a narrowly scoped test-only interception facility if required; do not add a production callback API merely to reproduce the race.
- Add short concurrent first-request JS coverage on a fresh node with at least two effective workers. Keep it outside the existing TSAN skip for long-lived forwarding, or TSAN will still miss this initialization path.
- Preserve valid JWT/certificate, unknown-policy,
no_auth, and combined-policy behavior.
No production patch, configured CCF build, official unit/e2e/sanitizer suite, or compatibility run was performed during this investigation. Those remain implementation-time requirements.
Subsequent PR assessment
The registry remained unchanged at main 0a742a75d5d3cc49567e2c45ac069087e38618f8. Inspected PRs #8511, #8445, #8478, and #8368 do not synchronize its initialization; newly merged #8366 adds recovery tracing only. No applicable fix was established.
Activity