Repository navigation
fix(sandbox-e2b): reliable exec timeout with server-side kill - #2976
Conversation
75090ec to
b64ed41
Compare
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
CryoThrust
left a comment
There was a problem hiding this comment.
One small robustness concern: res.body().string() is now unconditional in the non-success response path. OkHttp permits a response with a null body (for example a 204/transport edge or a test double), so this can replace the intended SandboxRuntimeException with an NPE. Keeping the previous null-safe fallback (res.body() != null ? res.body().string() : "") would preserve the error contract while still including the server body when present. A regression test with a non-success response and no body would pin this.
|
@CryoThrust Thanks for flagging this. The null guard was removed deliberately:
|
- agentscope-ai#2196: pass username query param in e2b filesystem REST calls (filesUrl helper plus URL assertions and workspace state setup) - agentscope-ai#2555: drop dead snapshot-timestamp helpers from E2bPlatformHttp, restore coverage tests plus InputStream close fix - agentscope-ai#2976: reliable e2b exec timeout with server-side kill (callTimeout plus readTimeout(0), Connect-Timeout-Ms header, interrupt-bit handling) plus its tests - E2bEnvdProcessClientTest: align three stale drain tests with the merged upstream agentscope-ai#2828 reject-without-exit-code behavior
oss-maintainer
left a comment
There was a problem hiding this comment.
Summary
The timeout model is right: readTimeout(0) hands total-duration ownership to callTimeout, Connect-Timeout-Ms stops leaking the remote process, and the interrupt bit is no longer swallowed. CI is green, CLA signed, head mergeable — nothing blocking.
Five points, in priority order:
- [Warning]
Thread.currentThread().isInterrupted()(line 171) reads ambient thread state, so a stale interrupt bit can turn a genuine exec timeout into an unmappedInterruptedIOException. Snapshot/clear the bit around the call instead. - [Warning]
timeoutSeconds <= 0now throwsIllegalArgumentExceptionwhere it previously meant "unbounded"; that type is not mapped bySandboxBackedFilesystem.execute()and diverges from the daytona/agentrun backends, which clamp. - [Warning] Client
callTimeoutand the serverConnect-Timeout-Msdeadline are identical, so on expiry the client still wins the race and the partial stdout/stderr already drained is discarded. - [Info] On the
res.body()null-guard exchange with @CryoThrust: the production-path argument is convincing for realCall.execute()responses —Objects.requireNonNull(res.body(), ...)documents that without reintroducing the ternary, and survives a NullAway/ErrorProne rollout or a synthesising interceptor. - [Info] Good coverage on the header and the fail-fast path; a
MockWebServerstall case is the one thing that would actually pinreadTimeout(0)itself, since the app-level interceptors short-circuit before OkHttp timeouts engage.
Automated review by github-manager-bot
| throws Exception { | ||
| if (timeoutSeconds <= 0) { | ||
| throw new IllegalArgumentException( | ||
| "timeoutSeconds must be positive: " + timeoutSeconds); |
There was a problem hiding this comment.
[Warning] Fail-fast on timeoutSeconds <= 0 changes the effective contract, and it diverges from the sibling sandbox implementations.
AbstractBaseSandbox.exec() only substitutes a default when the argument is null (int timeout = timeoutSeconds != null ? timeoutSeconds : getDefaultExecTimeoutSeconds()), so an explicit 0 used to mean "no callTimeout" and now aborts before the request is issued. DaytonaHttp.execute() (Math.max(1, timeoutSeconds)) and AgentRunMcpChannel.exec() (Math.max(1, ...)) clamp instead of throwing, so the same input now behaves differently per backend.
It also escapes the tool-layer error mapping: SandboxBackedFilesystem.execute() catches SandboxException.ExecTimeoutException / ExecException only, so an IllegalArgumentException surfaces as an unmapped failure rather than an ExecuteResponse with a non-zero exit code.
Either clamp to keep the backends consistent, or throw a SandboxException so the harness renders it as a tool error:
if (timeoutSeconds <= 0) {
timeoutSeconds = getDefaultExecTimeoutSeconds(); // or throw SandboxException.ExecException
}If throwing is deliberate, it is worth a line in the PR description plus a javadoc note on runShell/runShellBinaryStdout, since both are called with hardcoded constants today.
| .addHeader("Authorization", basicAuthUser(opt.getRunUser())) | ||
| // Ask envd to kill the remote process when this lapses; otherwise a | ||
| // client-side timeout only drops our HTTP stream and leaks the process. | ||
| .addHeader("Connect-Timeout-Ms", String.valueOf(timeoutSeconds * 1000L)); |
There was a problem hiding this comment.
[Warning] The server-side deadline and the client-side callTimeout are set to exactly the same instant, so on timeout the two abort races and the client usually wins — which means the process is reaped server-side (good, that is #2974) but the terminal frames from envd are still lost, and the caller sees a generic ExecTimeoutException without the partial output/exit status the server already flushed.
Give the server a small head start so its close frame can arrive first:
long timeoutMs = TimeUnit.SECONDS.toMillis(timeoutSeconds);
rb.addHeader("Connect-Timeout-Ms", String.valueOf(timeoutMs));
// client waits slightly longer than envd's own deadline
OkHttpClient callClient = http.newBuilder()
.callTimeout(timeoutMs + 5_000L, TimeUnit.MILLISECONDS)
.readTimeout(0, TimeUnit.SECONDS)
.build();Also timeoutSeconds * 1000L overflows silently for very large values from a caller-supplied timeout — TimeUnit.SECONDS.toMillis(...) saturates instead. Note TAR_TIMEOUT_SECONDS and the 120s workspace calls are unaffected; this only matters for the LLM-supplied path.
| } catch (InterruptedIOException e) { | ||
| // External cancellation surfaces here too; don't misreport it as a timeout | ||
| // and don't swallow the interrupt bit. | ||
| if (Thread.currentThread().isInterrupted()) { |
There was a problem hiding this comment.
[Warning] Restoring the interrupt bit and rethrowing is the right call, but the exception type now leaves the SandboxException vocabulary. runShellCapture is reached from E2bSandbox.doExec, and SandboxBackedFilesystem.execute() maps only ExecTimeoutException / ExecException; a raw InterruptedIOException propagates past both handlers.
Two knock-on effects worth deciding on explicitly:
- A cancelled exec produces an unstructured failure for the agent instead of an
ExecuteResponsethat says the command was interrupted. - The check is a heuristic — any stale interrupt bit on a pooled thread (e.g. after
shutdownNow(), or an interceptor that interrupts and does not clear it) turns a genuinecallTimeoutinto "cancellation". Thefinallyin the new test hints at how easy that is to leak. If the goal is to distinguish the two reliably, an explicitcall.isExecuted()/canceledflag or a wrapper exception (SandboxException.ExecInterruptedException) is more robust than reading global thread state.
The second catch clause of DockerSandbox.doExec has the same shape, so a shared convention would keep the backends aligned.
| int len = ByteBuffer.wrap(lenB).order(ByteOrder.BIG_ENDIAN).getInt() & 0x7FFFFFFF; | ||
| if (len < 0 || len > 64 * 1024 * 1024) { | ||
| if (len > 64 * 1024 * 1024) { | ||
| throw new IOException("Invalid connect frame length: " + len); |
There was a problem hiding this comment.
[Info] Dropping len < 0 is correct — the & 0x7FFFFFFF on line 199 already makes the value non-negative, so the branch was dead. But that same mask is why a corrupt header with the high bit set is silently truncated rather than rejected: 0x80000001 becomes 1, i.e. a plausible one-byte frame instead of an error.
If you want the length guard to be meaningful, validate the unsigned value before masking:
int raw = ByteBuffer.wrap(lenB).order(ByteOrder.BIG_ENDIAN).getInt();
long len = Integer.toUnsignedLong(raw);
if (len > 64L * 1024 * 1024) {
throw new IOException("Invalid connect frame length: " + len);
}Pre-existing behaviour, not introduced here — mentioning it only because this hunk touches the validation.
| // would preempt callTimeout (both surface as InterruptedIOException), | ||
| // misreporting a short idle stall as a full exec timeout (#2974). | ||
| // Total-duration semantics is owned solely by callTimeout. | ||
| .readTimeout(0, TimeUnit.SECONDS) |
There was a problem hiding this comment.
[Info] readTimeout(0) is the actual fix for #2974, but nothing pins it: the new tests use application interceptors, which throw before OkHttp's timeouts engage, so the assertions really cover "header is set" and "interrupt is not swallowed". If someone restores the inherited readTimeout, the suite still passes.
A MockWebServer case that writes one frame, stalls longer than opt.getReadTimeoutSeconds(), and asserts ExecTimeoutException would lock the behaviour in; asserting callClient.readTimeoutMillis() == 0 && callClient.callTimeoutMillis() == timeoutSeconds * 1000 via a small package-private seam is the cheap version.
|
@oss-maintainer Thanks for the thorough review — all five points addressed in ec8cf44 (19/19 tests green, spotless clean). Point by point:
|
oss-maintainer
left a comment
There was a problem hiding this comment.
Summary
Re-review after the force-push that addresses my previous five points. All five landed: the interrupt bit is snapshotted and restored instead of being read ambiently, the fail-fast path now explains itself and is pinned by a "no HTTP request issued" assertion, Objects.requireNonNull settles the res.body() debate, and the new Connect-Timeout-Ms → deadline_exceeded mapping plus mapPrematureEnd close the "server killed it but we call it a stream error" gap with real coverage on both codecs (JSON and hand-decoded proto). CI green, CLA signed, mergeable; the two points below are not blockers for me, they are the race and the pre-existing-bit case that the new logic still leaves open.
- [Warning]
E2bEnvdProcessClient.java:167— clearing a pre-existing interrupt bit means an already-cancelled caller now blocks inexecute()for the fulltimeoutSecondsbefore the bit is restored; suggest failing fast whenpreInterruptedis true, plus a test for that path. - [Warning]
E2bEnvdProcessClient.java:190— clientcallTimeoutand the server deadline are the same duration, so thedeadline_exceededbranch will rarely win the race in production (the canned-body tests bypass it), and partialstdout/stderrdrained before the kill is still discarded byExecTimeoutException. Offsetting one side by ~500 ms and carrying the captured output would make the new mapping actually observable. - [Info]
:129— good resolution of theIllegalArgumentExceptiondebate, but the cross-backend divergence (daytona/agentrun clamp) is now documented rather than removed, and the message embeds raw model-generated command text that may carry inline tokens; worth a redaction pass or a why-not-clamp comment. - [Info]
:281— the hand-rolled proto end-stream decode is correct forEndStreamResponse{error=1}; keep the broad catch and consider a debug log when an unparsable frame falls through toMissingExitCodeException.
Automated review by github-manager-bot
| int exit; | ||
| // Snapshot the interrupt bit: a stale bit set before this call must not turn a | ||
| // genuine exec timeout into a fake cancellation below. | ||
| boolean preInterrupted = Thread.interrupted(); |
There was a problem hiding this comment.
Snapshot-and-restore fixes the misclassification from my previous point 1, but it also removes the only fast-abort a caller has: Thread.interrupted() clears the bit, so a thread that was already cancelled before entering runShellCapture now issues the HTTP call and blocks in execute() for up to timeoutSeconds (default is a full exec timeout) before the bit is re-set in finally. OkHttp's blocking execute() does not poll the interrupt bit, so restoring it afterwards is not an early return — it's a delayed one.
Suggest failing fast on a pre-existing bit instead of absorbing it:
boolean preInterrupted = Thread.interrupted();
if (preInterrupted) {
Thread.currentThread().interrupt();
throw new InterruptedIOException("caller thread already interrupted; aborting exec before Start");
}That keeps the guarantee you documented (a stale bit must not fake a cancellation of this call) while not making an already-cancelled caller wait out the timeout. Either way, a short test asserting the pre-existing-bit path's behaviour would be good — interruptionIsRethrownNotWrappedAsTimeout only covers an interrupt raised from inside the call.
| throw new SandboxException.ExecTimeoutException(shellCommand, timeoutSeconds); | ||
| } catch (ConnectStreamException e) { | ||
| if (CONNECT_DEADLINE_EXCEEDED.equals(e.code())) { | ||
| throw new SandboxException.ExecTimeoutException(shellCommand, timeoutSeconds); |
There was a problem hiding this comment.
The deadline_exceeded mapping is the right shape, but as long as client callTimeout (line 135) and server Connect-Timeout-Ms (line 156) are the same duration, this branch is mostly unreachable: the client aborts the stream at T and the server writes the end-stream frame at T too, so in practice the connection is closed before the frame is read and InterruptedIOException (or the new MissingExitCodeException heuristic) wins the race. Two consequences:
- The error code you are mapping — and the
mapPrematureEndfallback below — will fire rarely in production and mostly in tests (the canned-body tests bypass the race entirely, since the frame is already in memory). - The partial
stdout/stderralready drained at that point is still discarded:ExecTimeoutExceptioncarries only the command and the timeout, so the operator sees no output for a long-running command that produced megabytes before being killed. That was the substance of my previous point 3 and it is not yet addressed.
Cheap improvement: give the server a slightly shorter deadline (e.g. timeoutSeconds * 1000L - 500) or extend callTimeout by a small grace (timeoutSeconds + 1s) so the terminal frame can arrive, and consider a ExecTimeoutException(cmd, timeout, stdout, stderr) variant (or attaching the captured output to the existing type) so the killed run's partial output survives to the harness error path.
| "[e2b] timeoutSeconds must be positive, got " | ||
| + timeoutSeconds | ||
| + " for command \"" | ||
| + truncateCommand(shellCommand) |
There was a problem hiding this comment.
Fair resolution of the previous point 2 — the [e2b] prefix plus "this is a caller bug, not a sandbox failure" makes the intent unmistakable, and pinning "no HTTP request must be issued" in the test is the part that actually matters.
One thing to note for whoever reads this later: the IllegalArgumentException type is still not mapped by SandboxBackedFilesystem.execute(), and daytona/agentrun clamp instead of throwing, so the cross-backend divergence you and @CryoThrust discussed is now documented in the message rather than removed. If that is the deliberate call, a one-line comment at the throw site saying why clamping was rejected would save the next reviewer re-litigating it.
Also mind that the message now embeds up to 200 chars of the raw command. Shell commands in this path are model-generated and routinely contain inline tokens (curl -H "Authorization: Bearer ..."), and an IllegalArgumentException message is exactly what lands in logs and exception metrics. If the goal is just operator context, consider logging the command at warn behind the existing redaction helper (or quoting only the first argv word) instead of putting it in the throwable.
| continue; | ||
| } | ||
| if (field == 1) { | ||
| code = in.readString(); |
There was a problem hiding this comment.
Two small robustness notes on the hand-rolled wire decode:
in.skipField(tag)on an outer tag of unknown wire type, combined with theWIRETYPE_LENGTH_DELIMITEDguard, is safe forEndStreamResponse{ Error error = 1 }, but if the field is ever reused with a different type the loop silentlyskipFields and returnsnull, which turns a real server error back intoMissingExitCodeException. Alogger.debug("unparsable connect end-stream error ({} bytes)", data.length)beforereturn nullwould make a server-side format change diagnosable.readRawVarint32()as the sub-message limit followed bypushLimitis correct, but a hostile/truncatedlenyieldsNegativeRuntimeLengthException— anInvalidProtocolBufferExceptionthat is not anIOExceptionsubclass here... actually it is, but it escapes throughparseProtoEndStreamError'scatch (Exception e) { return null; }only because the catch is broad. It is fine today; please keep the broad catch, since a future narrowing toInvalidProtocolBufferExceptionwould let a malformed frame abort the drain loop.
|
@oss-maintainer Re-review addressed in d995191 (23/23 tests green, spotless clean). Point by point:
Coverage follow-ups included: malformed JSON end frame, proto frame with unknown top-level fields (skip path), empty proto end frame, and both |
oss-maintainer
left a comment
There was a problem hiding this comment.
Summary
Delta re-review of d9951910 (my previous review was COMMENTED on ec8cf440). Both non-blocking warnings from that review are now closed, and the fixes match what was asked:
- Pre-cancel fail fast (
:167) — the interrupt-bit snapshot moved ahead of the call; it re-arms the bit and throwsInterruptedIOException("exec cancelled before start")instead of letting an already-cancelled caller block for the fulltimeoutSeconds.alreadyCancelledCallFailsFastWithoutBlockingpins both the fast return and that no HTTP request is issued. - Server-wins race (
:190) —callTimeoutis nowtimeoutSeconds*1000 + 500mswhileConnect-Timeout-Mskeeps the exact server deadline, sodeadline_exceededdeterministically wins and a retry never overlaps a still-running process.timeoutSeconds * 1000Lpromotes before the add, so no int overflow at realistic values. truncateCommandremoval — command text is gone from the invalid-timeout error, with the rationale (model-generated commands can carry inline tokens) documented at the call site; grep over head confirms no dangling references.- Debug-only partial-stream logging —
logPartialStreamsis guarded byisDebugEnabled()and capped at a 512-byte tail, so the hot path allocates nothing when debug is off, and the addedparseProtoEndStreamErrordebug line makes the silent fall-through observable.
Findings
- [Info]
E2bEnvdProcessClient.java:204— only thedeadline_exceededandMissingExitCodeExceptionpaths log partial stdout/stderr tails; the genericConnectStreamExceptionpath still surfaces code + message only. Reasonable scope choice, noting it just so the asymmetry is deliberate. - [Info] CI:
build (ubuntu-latest)andbuild (windows-latest)are still queued ond9951910as of this review, so I am not approving on this commit yet.license/cla,Check License,Check Module Syncare green and the branch isMERGEABLE/CLEAN.
The remaining check is just CI going green; nothing here needs another code change from my side.
Automated review by github-manager-bot
|
Thanks for the careful read — P1's mechanism is correct, P2's premise needs a correction, and both are addressed in a007a21: P1 (backup can fire before the server deadline) — accepted, and the fix is already shaped by it. You're right about the two clocks: The slack is now documented as a fault-grading threshold rather than a race constant: within T the server kills on time and the client observes P2 (timeout still embeds the command) — the premise doesn't hold, but the conclusion is now implemented anyway. No code, PR description, or comment in this PR ever stated 'errors carry no command text' as a goal — the IAE comment was scoped to that single new message. That said, the official SDKs settle it: Python's |
|
@oss-maintainer Follow-up in a007a21 (25/25 green, spotless clean), addressing your re-review plus a local review round:
|
oss-maintainer
left a comment
There was a problem hiding this comment.
Summary
Delta re-review of a007a211 (previous COMMENTED review was on d9951910). Both previous warnings stay closed and this round adds the two things the timeout path needed: Connect-Timeout-Ms so envd kills the remote process instead of leaking it, an end-stream deadline_exceeded decode (JSON and proto) that maps a server-side kill to ExecTimeoutException, a T + 5s client backup with the fault-grading rationale written down, and ExecTimeoutException no longer echoing the command text (secret hygiene). readTimeout(0) plus the ConnectStreamException / MissingExitCodeException split are covered by new tests, including the no-HTTP-request-on-invalid-timeout and interrupt-bit cases.
LGTM — approving. Remaining items are non-blocking notes: the requireNonNull fail-loud shape on the non-2xx branch, the timeoutSeconds + 5s wall-clock tail for callers above doExec, and the truncating elapsed-time boundary in mapPrematureEnd. CI is green on a007a211 (build ubuntu + windows, codecov/patch) and CLA is signed.
Automated review by github-manager-bot
| try (Response res = callClient.newCall(req).execute()) { | ||
| if (!res.isSuccessful()) { | ||
| String err = res.body() != null ? res.body().string() : ""; | ||
| String err = Objects.requireNonNull(res.body(), "envd response body").string(); |
There was a problem hiding this comment.
Dropping the command text from ExecTimeoutException is the right call — model-generated commands routinely inline API keys, and the message ends up in logs and error events. Keeping command as an unused constructor parameter (documented as signature compatibility) is acceptable, but a reader of SandboxException.ExecTimeoutException may still assume getMessage() is diagnostic; the javadoc paragraph added here mitigates that.
One behavioural note on this line though: Objects.requireNonNull(res.body(), ...) inside the non-2xx branch turns a hypothetically null body into an NullPointerException that is not caught by the InterruptedIOException / ConnectStreamException / MissingExitCodeException handlers below, so it escapes as an unexpected error instead of SandboxRuntimeException(WORKSPACE_START_ERROR). OkHttp never returns a null body for a real network response, so this is theoretical — but the previous res.body() != null ? res.body().string() : "" shape preserved the classification. If the intent is fail-loud, consider throwing the typed SandboxRuntimeException with "<no body>" instead of relying on requireNonNull.
| * setup distribution short of extreme degradation; larger values would approach the | ||
| * agent retry interval and buy nothing further. | ||
| */ | ||
| private static final long CLIENT_TIMEOUT_SLACK_MILLIS = 5000; |
There was a problem hiding this comment.
The 500ms → 5000ms slack plus the T / T+slack / >T+slack fault-grading rationale is exactly the reasoning I asked for in the previous round, and clientBackupToleratesServerKillMargin pins the ordering so the server deadline wins the race.
Worth remembering the tail cost: with readTimeout(0) the call timeout is the only guard, so a hung-but-open stream now blocks the calling thread for up to timeoutSeconds + 5s. That is a deliberate trade-off versus misreporting an idle stall as an exec timeout, but the sandbox-level wall-clock budget (and any tool-level timeout above doExec) should be > timeoutSeconds + slack or the caller gives up first and observes a cancellation instead of an ExecTimeoutException.
|
|
||
| static Exception mapPrematureEnd( | ||
| MissingExitCodeException e, String shellCommand, int timeoutSeconds, long startNanos) { | ||
| if (elapsedSeconds(startNanos) >= timeoutSeconds) { |
There was a problem hiding this comment.
mapPrematureEnd grades by elapsed time only, so a stream that dies at T + 3s (inside the slack window, server already gone) is reported as ExecTimeoutException rather than a transport failure. That is the documented intent of the slack band, and it keeps callers getting an exec timeout where the process did run to its deadline — just flagging that elapsedSeconds() truncates, so a stream ending at T - 0.4s classifies as a premature end rather than a timeout. Using the same TimeUnit.MILLISECONDS comparison as the call-timeout boundary would make the boundary exact.
oss-maintainer
left a comment
There was a problem hiding this comment.
Summary
This PR makes the E2B exec timeout deterministic: readTimeout(0) so total duration is owned solely by callTimeout, a Connect-Timeout-Ms header so envd kills the remote process server-side, a 5s client-backup slack so the server deadline deterministically wins the race, fail-fast on non-positive timeouts, and Thread.interrupt() cancellations rethrown instead of being misreported as exec timeouts. I verified the concurrency fix empirically against the repo's OkHttp 5.3.2 and against connect-go/envd sources: callTimeout surfaces as InterruptedIOException("timeout") with the interrupt bit clear (so the new catch maps it to ExecTimeoutException as intended), an application-interceptor stall is subject to callTimeout (so clientBackupToleratesServerKillMargin really does pin the T vs T+5s ordering), the interrupt snapshot/restore touches only the current thread -- no shared mutable state, no locks, no visibility hazard -- and envd reads Connect-Timeout-Ms as milliseconds and uses it as the process kill deadline, so the server-wins design holds. The one real defect is that the end-stream error frame is decoded as protobuf under the default E2bCodec.PROTO although the Connect protocol always sends it as JSON, which makes the new deadline_exceeded mapping and its two proto tests inert for the codec that ships by default (I confirmed by feeding real frames through the method: it returns null). Otherwise the racing path has no wire-level coverage -- every test uses in-memory canned frames -- so a MockWebServer-style case that stalls a real socket past T and asserts the server-wins classification would still be worth adding.
Re-reviewed at head a007a21136; CLA: signed.
Automated review by github-manager-bot
| } | ||
| } | ||
|
|
||
| private ConnectError parseEndStreamError(byte[] data) { |
There was a problem hiding this comment.
The end-stream error frame is decoded as protobuf whenever the codec is PROTO, but the Connect protocol always sends that frame as JSON — so the new deadline_exceeded mapping never fires for the default codec.
parseEndStreamError (line 272) switches on the request codec, and both codec() and E2bSandboxClientOptions default to E2bCodec.PROTO, so the default path goes to parseProtoEndStreamError (line 296). The Connect protocol does not use the RPC's codec for the end-of-stream envelope: connect-go (which serves process.Process/Start in envd — connectrpc.com/connect v1.18.1 in e2b-dev/infra's envd go.mod) marshals it with json.Marshal in connectStreamingMarshaler.MarshalEndStream from connectEndStreamMessage, a struct that has JSON tags only (error, metadata) and no protobuf tags, and connect-go's own client parses it with json.Unmarshal unconditionally. connect-python (_protocol_connect.py: json.dumps on send, json.loads on receive) and connect-es (protocol-connect/end-stream.ts, JSON-only serialization) behave the same. A real application/connect+proto stream therefore ends with a 0x02 frame whose payload is {"error":{"code":"deadline_exceeded","message":"context deadline exceeded"}}.
I fed those exact frames through this method: the leading { is read as tag field 15 / wire-type 3, the loop skips, and the broad catch (Exception) swallows the resulting InvalidProtocolBufferException and returns null (verified for deadline_exceeded, unavailable and empty-error frames). Consequences for the default codec:
parseEndStreamErroralways returnsnull, soConnectStreamExceptionis never thrown and thedeadline_exceeded→ExecTimeoutExceptionbranch added by this PR is dead code; classification falls back tomapPrematureEnd's whole-secondelapsedSeconds >= timeoutSecondsheuristic (which usually still reports a timeout because the server kills at T, but not reliably).- Non-timeout server errors (
unavailable,internal, "sandbox gone") lose their code and message and surface as the generic "envd process stream ended before receiving a process exit code" — the casejsonNonTimeoutErrorFrameKeepsStreamErrorpins for JSON is unreported and differently classified for PROTO. - The two new proto tests (
protoDeadlineExceededEndFrameMapsToExecTimeout, line 464, andprotoEndFrameWithUnknownFieldsStillMaps, line 538) build their frames viaprotoEndStreamError(line 618) using the sameerror = 1assumption, so decoder and fixture share one wrong premise and pass while production stays broken.
Suggested fix: decode the end-stream payload as JSON for both codecs (always call parseJsonEndStreamError), delete parseProtoEndStreamError and the protoEndStreamError fixture, and pin the behaviour with a test that feeds a JSON end-stream frame to a PROTO-coded client — the configuration that ships by default. If a proto fallback is kept for robustness it needs a captured frame from envd rather than a hand-built one. The rest of the frame handling is correct: END_STREAM_FLAG = 0x02 matches connect-go's connectFlagEnvelopeEndStream, and the per-call callTimeout + readTimeout(0) + interrupt-snapshot design verifies out against OkHttp 5.3.2.
|
@oss-maintainer You found a real defect — confirmed and fixed in 298b138 (24/24 green, spotless clean). Root cause. Verified live, not just from source. Ran a real envd (0.6.10) with Fix. Deleted the proto decoder, the codec dispatch, the No behavior regression. When the server kills on time the client still reports On the wire-level socket test you also raised: agreed it is the real gap, but it needs |
|
@oss-maintainer You were right that the racing path had no wire-level coverage, and that the app-level interceptors couldn't provide it. Added in da3eb1b — I first tried the The cases, with the base client deliberately set to
While writing this I hit a genuine bug in my own first draft worth flagging: I initially wrapped the end event in a Connect end-stream frame (flags |
oss-maintainer
left a comment
There was a problem hiding this comment.
Summary
Delta re-review at head da3eb1b8 (previous round was on a007a211). Both open items from the last round are closed, and closed in the way the protocol actually works:
- End-stream decoding is now JSON-only.
parseEndStreamErrorno longer branches oncodec(), and the proto decoder is gone. The javadoc records why — connect-go'sMarshalEndStreamhardcodesjson.Marshal, so the end-stream frame is not routed through the negotiated codec — and reports it verified against a live envd, whereapplication/connect+protoandapplication/connect+jsonreturn the same 76-byte{"error":{"code":"deadline_exceeded",…}}frame. That was the defect: under the defaultE2bCodec.PROTOthe newdeadline_exceededmapping could never fire, so the server-wins classification was inert for the codec that ships. Dropping the branch also deletes ~44 lines of hand-rolled wire parsing that nothing else needed. - The racing path now has wire-level coverage.
E2bEnvdProcessClientSocketTest(210 lines, 3 cases) drives a real socket through OkHttp rather than canned in-memory frames: a start event followed by silence pastTasserts theExecTimeoutExceptionmessage and classification; a second case pins the client-backup ordering (elapsed >= Tand< 30 s, so the backup bounds the call without preempting the server deadline); a third keeps the normal completion path honest. This is exactly the gap I flagged — every previous test fed pre-encoded buffers, so nothing exercised howcallTimeout(0-read-timeout)behaves against an actual stalled peer.
CI is fully green on this head (build (ubuntu-latest) 19m18s, build (windows-latest) 27m27s, License, Module Sync, codecov), CLA is signed, and the PR is MERGEABLE/CLEAN. Nothing in the delta reopens the earlier analysis: the interrupt-bit handling, Connect-Timeout-Ms propagation and the mapPrematureEnd fault grading are unchanged since the last round.
Approving. Thanks for the live-envd verification note — that is the kind of detail that saves the next person from re-introducing a "symmetric" proto decoder.
One forward-looking note (not a blocker, no action needed here): #3341, #3343 and #3344 are a split of this same area by the same author and carry 064c77b5 / 8fd39b32 in their branches. Whichever lands first, the others will need re-anchoring, and #3341 currently still logs envdAccessToken in E2bEnvdProcessClientIntegrationTest — worth a glance when you reconcile the series.
Automated review by github-manager-bot
…l reason, fail-fast message
…t, no command text in errors
da3eb1b to
7afa257
Compare
|
The latest Windows CI failure is outside this PR's changed modules: |
Buktal
left a comment
There was a problem hiding this comment.
-
E2bEnvdProcessClient.java:125— rejectingtimeoutSeconds <= 0breaks the merged integration test and reverses a working contract.E2bEnvdProcessClientIntegrationTest.java:77(merged via #3341–#3344) callsrunShell(..., "sleep 120", 0)as "blocks until killed"; with the fail-fast it throwsIllegalArgumentExceptionimmediately, so thecause instanceof SandboxExceptionassertion fails for anyone running the live suite (E2B_API_KEY-gated, which is why CI doesn't catch it). The justification comment ("this transport cannot model an unbounded stream") doesn't hold: the baselinetimeoutSeconds > 0 ? callTimeout(...) : httpdid model it, and the official SDKs map 0/None to "no deadline" (timeout_to_msin e2b-python). The issue itself scopes the header to "whentimeoutSeconds > 0", implying 0 stays valid. Suggestion: treat 0 as "no deadline" (skipcallTimeoutand the header for that call;readTimeout(0)still applies), or move the rejection up to a layer that maps into theSandboxExceptionvocabulary — today a raw IAE renders as"Internal sandbox error"with exit -1 inSandboxBackedFilesystem.execute(), while daytona/agentrun clamp viaMath.max(1, ...). -
E2bEnvdProcessClientSocketTest.java:116— the lower bound doesn't pinreadTimeout(0). IfreadTimeout(0)were reverted, the inherited 1s read timeout kills the call at ~T andelapsedMs >= 1000still passes;!(e.getCause() instanceof SocketTimeoutException)(:119) is vacuously true becauseExecTimeoutExceptionnever carries a cause. Verified on the pre-fix base: the test fails only on the message assertion (:112), not on timing. Raise the bound to ~T+slack (e.g.>= TIMEOUT_SECONDS * 1000L + 4_000) so reverting the fix turns this red; same forclientBackupFailsafeEndsHungSocket(:135). -
SandboxException.java:288— the message change silently spills to Docker and is undeclared.DockerSandbox.java:185constructs the same exception, so Docker timeout messages also lose the command text — arguably an improvement, but a cross-backend user-visible change the PR body doesn't mention, anddocs/v2/{en,zh}/docs/others/release-notes.mdhas no entry (cf. the #2828/#2609 entries for comparable behavior changes). Please declare the Docker impact and add the release-notes entry.
…ance 0 now means "no deadline", matching the official E2B SDKs and the E2bEnvdProcessClientIntegrationTest contract that runs `sleep 120` with 0 to block until killed. Negative values still fail fast. - omit Connect-Timeout-Ms for 0: the protocol reads 0 as "deadline already exceeded", so an unbounded call must not send it - explicitly set callTimeout(0) so an injected client's callTimeout is not inherited; readTimeout(0) already covered the read path - guard the InterruptedIOException, deadline_exceeded and mapPrematureEnd paths so an unbounded call never reports "timed out after 0s" - socket tests: base client gets 1s readTimeout and 2s callTimeout, and the timeout=0 case delays the response by 3s, so either inherited timeout turns the test red - add mapPrematureEnd coverage for the 0 branch - document 0 as implementation-specific on Sandbox.exec and declare the Docker impact of dropping command text from ExecTimeoutException
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
|
All three addressed in 1. You were right about the premise: the baseline does model an unbounded call (
2. Socket test lower bound didn't pin Correct — with
Verified by reverting 3. Message change spilled to Docker + release notes Declared in the PR body under "Cross-backend impact", and added entries to Also updated |
Buktal
left a comment
There was a problem hiding this comment.
All three points from the previous round are resolved and verified: timeoutSeconds = 0 now means no deadline with bounded guards on every timeout path (the merged integration test's contract is restored), the socket tests' new bounds turn red when readTimeout(0) / the callTimeout override are reverted (verified locally by reverting: 3 red / 1 green; full module suite green on this head), and the Docker spillover is declared with EN/ZH release-notes entries and an implementation-specific javadoc for 0.
Buktal
left a comment
There was a problem hiding this comment.
Follow-up verified on 272eab12b: CLIENT_TIMEOUT_SLACK_MILLIS is back to private (1ca1f37 had widened it to package-visible, but no test references it). Full module check on this head is green — mvn -B -T1C -pl agentscope-extensions-sandbox-e2b -am spotless:check test → BUILD SUCCESS, 0 failures / 0 errors.
Fixes #2974.
readTimeout=0) so total-duration semantics is owned solely bycallTimeout; previously a small basereadTimeoutpreemptedcallTimeoutand misreported a short idle stall as a full exec timeout (both surface asInterruptedIOException).Connect-Timeout-Msso envd kills the remote process server-side; previously a client timeout only dropped our HTTP stream and leaked the process.deadline_exceededand maps toExecTimeoutException; a stream that ends without an exit code past the deadline degrades to the same type.Thread.interrupt()) is rethrown as-is with the interrupt bit restored instead of being wrapped asExecTimeoutException.timeoutSeconds = 0means "no deadline", matching the official E2B SDKs and the contract already exercised byE2bEnvdProcessClientIntegrationTest(runShell(..., "sleep 120", 0)blocks until killed). For that callcallTimeoutandConnect-Timeout-Msare omitted (Connect-Timeout-Ms: 0would mean "deadline already exceeded"), whilereadTimeout(0)still applies. Negative values fail fast withIllegalArgumentException.T + 5s, so the server-side kill wins the race and the slack only absorbs connect/setup jitter.Cross-backend impact
SandboxException.ExecTimeoutExceptionno longer includes the command text in its message.DockerSandboxconstructs the same exception, so Docker sandbox timeout messages change as well (commands may embed credentials; callers already hold the command). Recorded indocs/v2/{en,zh}/docs/others/release-notes.mdunder 2.0.4.Sandbox.execjavadoc now states thattimeoutSeconds = 0is implementation-specific: E2B treats it as no deadline, Docker as a zero-wait poll.Tests
E2bEnvdProcessClientTest57/57,E2bEnvdProcessClientSocketTest4/4; fullagentscope-extensions-sandbox-e2bmodule 149/149.timeout = 0case delays the response by 3s. RevertingreadTimeout(0)orcallTimeout(0)makes the call die at ~1s / ~2s instead of succeeding (verified locally).E2bEnvdProcessClientIntegrationTest#signalKillViaRunShell(E2B_API_KEY-gated) passes withtimeout = 0, exercising the no-deadline path against real envd.