[semyon-dev]

Summary

Coordinate daemon shutdown across native gRPC, HTTP, gRPC-Web, TLS, both cmux implementations, experimental split listeners and ACME. Stop admitting requests, drain active calls within a configurable shutdown_timeout (30 seconds by default), then cancel remaining calls and close their transports. Keep payment storage available until request handlers and their deferred commit/rollback and unlock operations finish.

Own and join background work for metering, notifications, training status updates, certificate reload and contract subscriptions. Close proxy clients and listeners, preserve request cancellation through backend calls, and bound external RPC/HTTP work and etcd transaction retries. Process backends now start before stdin is copied, avoiding an uncancellable wait on a full pipe.

Regression coverage

• Real signed payments, payment handlers, interceptors and locks, with only Ethereum replaced: success, backend failure and client cancellation across memory/etcd and both cmux implementations.
• Assert shutdown waits with the channel still locked; verify the final payment state, actual unlock and reuse of the same channel after restart.
• Linux child-process SIGINT/SIGTERM cases for success, failure, cancellation and grace-period expiry. Parent-owned etcd survives child exit and restart.
• Grace-period expiry while an actual etcd unlock is delayed: storage must remain open until cleanup completes.
• HTTP and gRPC-Web drain/deadline cases, TLS and traffic-split listeners, unclassified TCP connections, ACME and concurrent/idempotent stop.
• Cancellation of pending Ethereum RPCs, metering HTTP work, websocket contract subscriptions and process stdin; lifecycle-group and broadcaster cleanup.
• Linux CI runs the full suite and repeats race tests for the affected lifecycle packages.

Validation

• go test ./... -count=1 -timeout=180s passed on Windows.
• Full race tests passed for the affected daemon, training, metering, handler, escrow, contract listener, configuration, blockchain, lifecycle and storage packages on Windows.
• golangci-lint run --config=.golangci.yml --timeout=3m ./...: no issues.
• Linux CI passed: the full suite, all eight real SIGINT/SIGTERM subprocess cases, and the full affected-package race suite repeated twice. Lint and Sonar analysis also passed for the final commit.

Shutdown contract and limits

shutdown_timeout limits the graceful phase, not total process lifetime. After cancellation, payment cleanup uses independent bounded storage requests; shutdown waits for those handlers before closing etcd. These independent contexts are intentional, so caller cancellation does not prevent unlocking.

Cleanup cannot guarantee a successful unlock when etcd is unavailable. SIGKILL, crashes, power loss and arbitrary code that ignores cancellation are outside the cooperative shutdown guarantee. Persistent payment locks still have no TTL; crash recovery remains separate work.

Fixes #209

[semyon-dev]

Pre-merge review and validation

Reviewed the complete PR after merging the current dev into test/209-graceful-shutdown. The reviewed and tested head is efa8899d123b0745360c24c5bad58f95a8d25d41; its merged dev parent is c74c92e55989e34cc5e274f7308444e696bd0851. The four merge conflicts were resolved while preserving the lifecycle changes and the updated dev comments. No blocking findings remain from this review.

Windows, Go 1.27.3

• Regenerated protobuf and contract bindings with go generate ./....
• Full suite: go test ./... -count=1 -timeout=5m.
• Full race suite: go test -race ./... -count=1 -timeout=8m.
• go vet ./....
• golangci-lint run --config=.golangci.yml --timeout=5m ./...: zero issues with v2.15.0 built using Go 1.27.3.

WSL2, Ubuntu 24.04, Linux/amd64, Go 1.27.3

• Full suite: go test ./... -count=1 -timeout=8m.
• Full race suite: go test -race ./... -count=1 -timeout=8m.
• Repeated shutdown regressions: go test -race ./snetd/cmd -run '^TestDaemonShutdown' -count=5 -shuffle=on -timeout=5m.
• The repeated run includes 40 real SIGINT/SIGTERM scenarios, covering success, backend failure, client cancellation and grace-period expiry, with persistent etcd state and channel reuse after child-process restart.
• go vet ./... and golangci-lint v2.15.0 built using Go 1.27.3: passed, zero lint issues.

Docker

• Built the production image from docker/Dockerfile at the tested head using Go 1.27.3.
• Ran the daemon as PID 1 and the image's non-root user, without external network access or published host ports.
• HTTP and gRPC modes both exited successfully on SIGINT and SIGTERM: four runtime smoke cases.
• Two additional HTTP cases kept a request body incomplete. On SIGINT and SIGTERM, the configured 300ms grace period expired, the blocked request was terminated, and the daemon exited with code 0.
• All temporary review containers were removed.

CI at the reviewed head

• Full Linux suite and repeated lifecycle race tests: passed.
• Lint: passed.
• Sonar analysis: passed.

Validation note: running the complete Windows and WSL suites concurrently caused a fixed-port collision in existing etcd fixtures on localhost port 2380. Running the suites separately passed in both environments, including the complete race suites; the collision was not a shutdown regression.

Deployment contract

Existing config files remain usable. The new optional shutdown_timeout defaults to 30s and limits the graceful phase, not total process lifetime: payment cleanup must finish before etcd closes. Supervisors need additional termination allowance for bounded storage cleanup. Simple HTTP forwarding now honors service_timeout; deployments with long-running HTTP calls should review that setting.

This addresses the cooperative SIGINT/SIGTERM scenario in #209. SIGKILL, crashes, unavailable storage and recovery of persistent locks without TTL remain outside this guarantee.