Skip to content

fix: renew cached gateway leaf certificates before expiry - #176

Draft
xai wants to merge 2 commits into
mainfrom
fix/gateway-leaf-renewal
Draft

xai wants to merge 2 commits into
mainfrom
fix/gateway-leaf-renewal

Conversation

@xai

@xai xai commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

What it does

The gateway proxy could serve a cached leaf certificate past its expiry. Cache hits only refreshed file timestamps, so a frequently used host kept its three-month leaf for as long as the gateway container lived.

Cache hits now validate the certificate/key pair, the host name, the signing CA, and validity at least 24 hours ahead, and renew invalid or expiring pairs. Certificate loading stays under the cache lock so pruning cannot remove a pair before it is loaded, and pruning never evicts the certificate being returned, even when other entries have newer timestamps.

Shared CA access is serialized separately from the leaf cache. The CA pair is loaded and checked once per store, so cache hits only re-read and verify the leaf. Incomplete or mismatched CA pairs are rejected instead of being replaced or used, and the error names the files involved. Host-side CA setup does not access the proxy-owned leaf cache, preserving the layout introduced by PR #81.

How to test

Unit tests cover expired and near-expiry leaves, a malformed key, incomplete CA pairs, concurrent stores with a one-entry cache, and a requested leaf whose timestamp is older than another cache entry. Four of the five new tests fail on main; the fifth guards host-side CA setup against touching the leaf cache.

To reproduce, start a restricted session with --network-log=requests so HTTPS goes through the MITM proxy, then replace the gateway's cached leaf for api.github.com with one that expires within a day:

agent=enclave-claude-<project-hash>-<name>   # from `enclave ps`
host=api.github.com
key=$(printf %s "$host" | sha256sum | cut -c1-32)
tls=~/.local/state/enclave/tls
openssl req -new -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -subj "/CN=$host" \
  -addext "subjectAltName=DNS:$host" -keyout leaf.key -out leaf.csr
openssl x509 -req -in leaf.csr -CA $tls/ca.crt -CAkey $tls/ca.key -days 1 \
  -copy_extensions copy -out leaf.crt
docker exec -i -u gateway $agent-gateway sh -c "cat > /var/lib/enclave/proxy/tls/hosts/$key.crt" < leaf.crt
docker exec -i -u gateway $agent-gateway sh -c "cat > /var/lib/enclave/proxy/tls/hosts/$key.key" < leaf.key
docker exec $agent curl -sv -o /dev/null https://$host/zen 2>&1 | grep 'expire date'

On main curl reports the injected one-day expiry, and an already expired leaf fails with certificate has expired. On this branch the proxy mints a fresh three-month leaf.

Verified on Linux (Go 1.27.2; gateway packages also with the go.mod toolchain, Go 1.24.4):

make build
make test
make lint
go test -race ./internal/gateway/tlsstore
go test -race ./internal/gateway/tlsstore -run 'Test(ConcurrentStoresShareCAAndLoadBeforePruning|LeafPruningKeepsRequestedCertificate)' -count=20

End to end on Docker 29.1.3 and rootless Podman 4.9.3 (claude tool, --network-log=requests):

  • Leaves injected into a running gateway's cache that were expired, expiring within 24 hours, signed by another CA, or paired with a malformed key were replaced on the next handshake. A leaf with 25 hours left was kept. With the gateway built from main, the expired leaf was served and curl failed with certificate has expired.
  • A second session reused the existing CA. The host file, the gateway copy, and the agent's mounted CA kept the same fingerprint, and curl, Node fetch, and Python urllib verified through the proxy.
  • 40 parallel handshakes to 8 hosts all verified, with no proxy errors.
  • With either CA file missing, session start fails without modifying the remaining file, for example initialize gateway CA: incomplete trusted CA pair: ~/.local/state/enclave/tls/ca.key is missing; restore it or remove ~/.local/state/enclave/tls/ca.crt to generate a new CA. Removing the named file lets the next session generate a new CA. A mismatched pair fails with load trusted CA pair from ~/.local/state/enclave/tls: tls: private key does not match public key.
  • A legacy container-owned ~/.local/state/enclave/tls/hosts does not affect session start.

QEMU has no restricted-egress gateway and is not affected.

Follow-ups

None.

Breaking changes

  • This PR introduces breaking changes and has been coordinated with maintainers.

Review checklist

Signed-off-by: Olaf Lessenich <olessenich@eclipsesource.com>
@xai xai added the bug Something isn't working label Oct 22, 2026
@xai

xai commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

@EclipseSourceAI

@EclipseSourceAI EclipseSourceAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Autonomous AI review.

This review was done by an AI agent and therefore may contain mistakes. Feel free to ignore any comment you disagree with. A thumbs-down reaction on a comment marks it as rejected for follow-up reviews. Noting why in a reply helps, since replies are read too.

Resolving all AI comments does not lead to an automatic approval. A maintainer still needs to review and sign off on the overall architecture and design.

To get an updated review after pushing changes, a maintainer may re-request a review from this account.

Running in Eclipse Enclave, submitted via review-guard-mcp

This PR fixes the gateway MITM proxy serving cached leaf certificates past their expiry. On a cache hit the leaf is now checked against the CA, the host name, and a validity window of 24h or more, and is re-minted if any check fails. The leaf is loaded while the cache lock is still held, and pruning never evicts the entry being returned. CA setup now refuses incomplete or mismatched pairs instead of regenerating them, and a .ca.lock file lock serializes CA creation across processes.

The renewal logic looks correct. I ran the tlsstore tests locally with -race and they pass. Points for maintainers:

  • Leaf cache flock: the extra file lock on hosts/.cache.lock runs on every handshake. Only one process uses that container-private cache, so the in-process mutex is enough. The CA lock is justified.
  • CA loading: the CA is still loaded in two places (ensureCA and the old loadCA fallback), because the branch that generates a new CA never stores it on the store.
  • Duplicate work per handshake: on a cache hit the leaf pair is loaded twice, and there are extra x509.ParseCertificate calls that pair.Leaf makes unnecessary.
  • Behavior change: session start now fails on an incomplete CA pair. This isn't documented in docs/networking.md yet.

Comment thread internal/gateway/tlsstore/store.go Outdated
s.mu.Lock()
defer s.mu.Unlock()
return s.ensureCA()
return util.WithFileLock(filepath.Join(s.HostsDir, ".cache.lock"), fn)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The leaf cache is container-private and only one proxy process with one Store ever touches it (entrypoint, proxy), so s.mu already covers the load-before-prune race. This flock adds an open/lock/close on every TLS handshake just to make the multi-Store test pass; I'd drop it and keep the file lock only for the CA, where concurrent host enclave runs really can race.

Comment thread internal/gateway/tlsstore/store.go Outdated
certPath := filepath.Join(s.HostsDir, hostKey+".crt")
keyPath := filepath.Join(s.HostsDir, hostKey+".key")
if util.FileExists(certPath) && util.FileExists(keyPath) {
if s.caCert == nil || s.caKey == nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block only runs right after ensureCA generated a new CA, because the generate branch never sets s.caCert/s.caKey. Assign them there (the parsed DER and priv) and this fallback plus loadCA can go, leaving one CA loader instead of two with different key-format support, and no CA read outside .ca.lock.

Comment thread internal/gateway/tlsstore/store.go Outdated

// The CA never changes after it is loaded, so cache hits skip the file lock and
// the RSA key parse that would otherwise run on every TLS handshake.
func (s *Store) withCALock() error {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this mostly doesn't lock, it ensures the CA is loaded. Something like loadCA (once the old helper is gone) or ensureCALoaded reads better at the call sites.

Comment thread internal/gateway/tlsstore/store.go Outdated
return false
}
cert, err := tls.LoadX509KeyPair(certPath, keyPath)
leaf, err := x509.ParseCertificate(pair.Certificate[0])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With go 1.24 tls.LoadX509KeyPair already populates pair.Leaf, so this second parse (and the one in ensureCA) is redundant. Use pair.Leaf.

Comment thread internal/gateway/tlsstore/store.go Outdated
if err != nil {
return err
}
pair, err := tls.LoadX509KeyPair(certPath, keyPath)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a cache hit the pair is now read and parsed twice per handshake: once in validLeaf and again here. Have ensureLeaf/validLeaf return the loaded *tls.Certificate (and keep the root CertPool on the store next to caCert instead of rebuilding it each call).

if !util.FileExists(present) {
present, missing = missing, present
}
return fmt.Errorf("incomplete trusted CA pair: %s is missing; restore it or remove %s to generate a new CA", missing, present)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes user-visible behavior (an incomplete pair used to be silently regenerated, now session start fails). A line in the CA section of docs/networking.md about this and the recovery step would help, since that's where users look for the TLS dir layout.

…ificates

Signed-off-by: Olaf Lessenich <olessenich@eclipsesource.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants