Skip to content

fix(cel): tolerate exited processes during environment lookup - #1022

Merged
matthyx merged 2 commits into
mainfrom
fix/process-env-exited
Oct 8, 2026
Merged

matthyx merged 2 commits into
mainfrom
fix/process-env-exited

Conversation

@matthyx

@matthyx matthyx commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

When a process exits before an exec rule reads /proc/<pid>/environ, process.get_process_env currently returns a CEL error. R1180 attempts this lookup for every evaluated exec event, turning normal process churn into error-level rule-evaluation logs.

In a read-only 10-minute sample from an affected EKS cluster, 463 environment lookup errors appeared across 3 of 27 node-agent pods. All were R1180 failures caused by stat /proc/<pid>: no such file or directory; none were permission failures. Increased CPU was reported alongside the errors, but the CPU cost attributable to logging versus live reads has not been measured.

Treat ENOENT from the target PID lookup or environment read as an exited process. After the function cache, convert that specific error to an empty environment so rules with variable-presence guards evaluate to false. Keeping the conversion after caching prevents caching a missing process as an empty map for a subsequently reused PID. Preserve permission, procfs-initialization, and other failures as errors.

This reduces expected error-log noise; it does not eliminate per-exec environment I/O or the detection gap for processes that have already exited.

Validation:

  • Added a regression using a real exited process; it failed with the original failed to get process environment error before the fix and passes afterward.
  • Added coverage for preserving other errors, retrying missing PIDs, and caching successful lookups.
  • go test ./pkg/rulemanager/cel/... passes.
  • go test -race ./pkg/rulemanager/cel/libraries/process ./pkg/rulemanager/cel/libraries/cache -count=1 passes.
  • make binary passes.
  • Targeted golangci-lint reports zero new issues with --new-from-rev=HEAD; the unrestricted targeted run reports two existing QF1003 findings in unchanged process_test.go.
  • git diff --check passes.

Follow-up path/permission experiment:

  • All three affected pods have hostPID: true and run as root with SYS_PTRACE.
  • An ephemeral BusyBox container in an affected pod, using the agent's security context and a read-only host mount, confirmed that /proc/1 and /host/proc/1 are systemd in the same host PID namespace.
  • Both /proc/<kubelet-pid>/environ and /host/proc/<kubelet-pid>/environ were readable; contents were discarded.
  • A controlled child process had a readable environment through both paths while alive, and was absent through both paths after it exited and was reaped.
  • Checks executed directly in the agent container's mount namespace also confirmed matching host PID namespaces and readable kubelet environments through both paths.
  • The ephemeral container completed with exit code 0. No application configuration was changed.

These checks rule out a general proc-path or permission mismatch in the tested pod and strongly support the exit-race explanation. They do not correlate every original failing event with an observed process exit.

Signed-off-by: Matthias Bertschy <matthias.bertschy@gmail.com>
@coderabbitai

coderabbitai Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 8 minutes.

Check out review usage here.

View limit details

Limit details: You鈥檝e used the included review currently available.

Learn how review limits work.

Review configuration:

鈿欙笍 Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 00e1390e-922a-4dda-aa7d-34fbae0c2092
馃摜 Commits

Reviewing files that changed from the base of the PR and between e33c1ff and d06bbd5.

馃搾 Files selected for processing (3)
  • pkg/rulemanager/cel/libraries/process/process.go
  • pkg/rulemanager/cel/libraries/process/process_env_exited_test.go
  • pkg/rulemanager/cel/libraries/process/processlib.go
  • Autopilot 路 Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

鉂わ笍 Share

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: Matthias Bertschy <matthias.bertschy@gmail.com>
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown

Performance Benchmark Results

Node-Agent Resource Usage
Metric BEFORE AFTER Delta
Avg CPU (cores) 0.000 0.000 N/A
Peak CPU (cores) 0.000 0.000 N/A
Peak CPU p95 (cores) 0.000 0.000 N/A
Avg Memory (MiB) 0.000 0.000 N/A
Peak Memory (MiB) 0.000 0.000 N/A
Dedup Effectiveness

No data available.

@matthyx
matthyx marked this pull request as ready for review October 7, 2026 15:57
@matthyx
matthyx requested a balanced review from Copilot October 7, 2026 15:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

馃煝 Approval recommended

The implementation matches the described behavior and includes focused regression coverage.

0 open findings

What changed in this PR

Handles expected /proc/<pid> races when processes exit before CEL environment lookup.

Changes:

  • Converts exited-process errors to empty environments after caching.
  • Preserves permission and unrelated failures.
  • Adds regression and cache-behavior tests.
File Description
processlib.go Applies the fallback after cache lookup.
process.go Classifies ENOENT and converts exited-process results.
process_env_exited_test.go Tests exited processes, error preservation, and caching.

馃 Review effort: Balanced


馃挕 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@matthyx matthyx added the release Create release label Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown

Performance Benchmark Results

Node-Agent Resource Usage
Metric BEFORE AFTER Delta
Avg CPU (cores) 0.146 0.145 -0.3%
Peak CPU (cores) 0.156 0.154 -1.6%
Peak CPU p95 (cores) 0.156 0.153 -1.4%
Avg Memory (MiB) 376.011 311.901 -17.1%
Peak Memory (MiB) 379.598 315.805 -16.8%
Dedup Effectiveness

No data available.

@matthyx matthyx removed the release Create release label Oct 8, 2026
@matthyx
matthyx merged commit 901ad23 into main Oct 8, 2026
72 of 106 checks passed
@matthyx
matthyx deleted the fix/process-env-exited branch October 8, 2026 05:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants