Distributed observability service for AutoStream incidents, diagnostics, remediation suggestions, and notification delivery.
- Signal ingest for service health, metrics, warnings, and errors.
- Rule-based detection, active-incident deduplication, and signal-driven recovery.
- Diagnostic reports and remediation action records.
- Discord, Slack, generic webhook, and email notification channels.
- Secret-safe notification delivery history for the Control Panel.
AUTOSTREAM_NODE_CONFIG=/etc/autostream-observability/config.yml
DATABASE_URL=mysql://autostream_observability:<PASSWORD>@tcp(db.example.com:3306)/autostream_observability?parseTime=true
AUTOSTREAM_SECRET_ENCRYPTION_KEY=<32_BYTE_BASE64_OR_HEX_ENCRYPTION_KEY>
REMEDIATION_MODE=suggest_only
TZ=Asia/Tokyo
Panel-managed config.yml must select listener.credential: node-listener.json.
For systemd, LoadCredential exposes the strict four-field JSON from
/opt/autostream/local-executor/ports/observability.json; its bind_address
is the only local listener authority and its positive config_revision is
reported by /updater/version. The JSON also carries schema_version: 2 and
service_type: observability. Missing, malformed, writable, or mismatched
credentials fail closed before MariaDB access. There is no bind or revision
environment fallback.
api.host and api.port remain the public/reverse-proxy endpoint and do not
select the local socket. The listener bind_address must use an unprivileged
port from 1024 through 65535; the standard host value is
127.0.0.1:8082, while Compose generates 0.0.0.0:8080 by default.
例えば 127.0.0.1:18082 に変更した場合、/health と
/updater/version も同じ 18082 で待ち受けます。不正な形式、範囲外、
または特権ポートを指定した場合は Observability がデータベース接続前に
起動を停止します。
IPv6 loopback を明示的に使う場合は [::1]:18082 のように角括弧を含めて
指定し、プローブURLも http://[::1]:18082/... とします。
Docker 版ではホスト公開ポートを AUTOSTREAM_OBSERVABILITY_PORT、
コンテナ内の待受ポートを AUTOSTREAM_OBSERVABILITY_CONTAINER_PORT
で個別に変更できます。既定値はそれぞれ 8082 と 8080 で、どちらも
1024～65535 を使用してください。
Compose published ports are a host/reverse-proxy responsibility. The Control
Panel Updater manages only host ports 1024 through 65535; manually
publishing a privileged or conflicting Docker host port is outside the managed
update contract.
The production health authority is the host Local Executor. These Compose files
intentionally omit an in-container healthcheck: the runtime image has no
purpose-built HTTP probe client, and the image contract does not add or repurpose curl, wget, or another unrelated executable solely for container health.
For managed Docker changes, the Local Executor probes the loopback published port for both /health and /updater/version; the published port is the health port.
A recreate is accepted only when health, service identity, version, and
the listener credential's config_revision match; otherwise the executor rolls back or reports
rollback_failed.
$env:AUTOSTREAM_OBSERVABILITY_PORT = "18082"
$env:AUTOSTREAM_OBSERVABILITY_CONTAINER_PORT = "18080"
$env:AUTOSTREAM_CONFIG_REVISION = "1" # Compose JSON generation input only
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -dControl Panel calls Observability with the Node Runtime Token stored in the Panel-generated AUTOSTREAM_NODE_CONFIG. Do not configure a separate Observability admin token or direct ingest token. Email channels relay through Control Panel and therefore require the notifications.email.send scope. Current Control Panel migrations add this scope to existing active Observability tokens; newly issued tokens include it from the start.
The Observability service does not serve a browser UI. GET / and GET /status return safe operator status JSON. GET /updater/version returns the embedded release version together with the Control Panel service identity and applied config revision for the loopback local executor. API data such as GET /metrics is token protected and is normally read through the Control Panel /observability/metrics proxy. GET /metrics?range_sec=<seconds> returns bounded history for the requested 15-minute to 3-hour range instead of only process-local browser history. A browser request to /metrics without the Node Runtime Token should return an authorization error.
An incident dedupe key is unique only while the incident is active. A matching failure increments its occurrence count and last-seen timestamp. A matching healthy signal resolves the active incident with recovery provenance, clears the active dedupe key, and preserves the old incident row so a later recurrence creates a new incident instead of overwriting history. Terminal incidents cannot be reopened by a stale status update.
The configure subcommand reports the node version, hostname, OS, and arch to Control Panel when it consumes the Configure Token. After the service starts with AUTOSTREAM_NODE_CONFIG, heartbeat reports the same platform fields plus observability.goroutines, heap metrics, and observability.uptime_seconds.
If the Node registration screen shows OS未取得 / Arch未取得 or no Metrics, verify that the latest autostream-observability configure command wrote the same path used by AUTOSTREAM_NODE_CONFIG, that the config is readable by the autostream user, and that heartbeat is succeeding in journalctl -u autostream-observability.
Webhook URLs are stored encrypted with AUTOSTREAM_SECRET_ENCRYPTION_KEY. Email channels store recipients and a non-secret control_panel/global_smtp reference; SMTP credentials remain exclusively in Control Panel. API responses and delivery history expose only non-secret policy, masked targets, fingerprints, status, and timestamps.
go test ./...
go build ./...- Do not log raw tokens, webhook URLs, SMTP passwords, credential-bearing URLs, or provider secrets.
- Register the Observability node in Control Panel and keep the generated
config.ymlreadable only by the service user. - Keep dangerous remediation actions manual unless a separate approval policy explicitly allows them.
- Apply request size limits and rate limits to sensitive endpoints.
Detailed deployment and security documentation is maintained in the autostream-docs repository.