Repository navigation
Conversation
…nections (#9) * feat(output): add chunk-safe output params parser and output limiter OutputParamsParser accumulates OUTPUT PARAMS values across output chunks and finds a line split between two chunks. OutputLimiter caps the stored output and emits a truncation marker once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(handler): run jobs without open transactions and survive lost connections The handler kept the job entity managed and flushed it from a busy loop, rewriting the whole output on every iteration. A connection dropped inside such a flush left pdo believing in a transaction DBAL had forgotten, and the worker crashed in the retry send with "There is already an active transaction"; the message was redelivered every redeliver_timeout. - JobOutputWriter writes output, status and result through DBAL in autocommit and resets the connection after a lost connection or a stale native transaction - the job is claimed atomically (planned -> running), redelivered messages of running/finished jobs do not run the command again - the wait loop sleeps 100 ms and writes at most once per poll interval; database errors are tolerated up to db_failure_tolerance - a failed command is a job status; only an unstorable result is thrown, as UnrecoverableMessageHandlingException - a missing job is rejected without retry - new config job_queue.processing: poll_interval_ms, output_max_bytes, db_failure_tolerance, rerun_on_redelivery Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(handler): add SQLite integration tests for job processing Runs the real handler with real subprocesses against a temporary SQLite database. A DBAL driver middleware injects lost connections (including the stale pdo transaction of TT-SERVER-CQ) and the DBAL logging middleware counts statements and transactions. Adds pdo_sqlite to the CI matrix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: document job processing options and messenger recommendations Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(handler): store only valid UTF-8 and never let rejected output block the result - Utf8Stream holds back a character split between two pipe reads and scrubs bytes which are not UTF-8; OutputLimiter cuts at a character boundary. A strict MySQL utf8mb4 column rejected the broken character (1366) forever. - Output a working database rejects (not a lost connection) is replaced by a note in the loop and in finalize, so it can neither kill a healthy command nor keep the result from being stored; a rejected result write is retried without output params and status message. Stored messages are bounded. - The writer keeps the caller's transaction nesting level for every reset, not only in the handler's finally block. - Finalize appends its status message only below the output cap. - Loading and claiming the job are retried (a lost claim response is verified by status + started_at) and end in a recoverable exception; a missing job is looked up again before it is rejected. - A rerun claim appends its note in the claim UPDATE, so MySQL always reports the changed row. - A job deleted mid-run stops its command; Process output buffers are cleared after each read; stdout and stderr are parsed separately. - The controller cancels by a conditional UPDATE, so it never overwrites a stored result. - Tests: handler-level TT-SERVER-CQ guard with a non-ConnectionLost driver error plus a stale native transaction (loop and finalize), multibyte cap and split reads, rejected output, caller transaction, deleted job, lost claim race, pre-claim retries. - CI: symfony/flex so SYMFONY_REQUIRE applies, DBAL 3 / ORM 2 legs; conflict with symfony/dependency-injection 8. Docs and changelog updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
OutputParamsParser accumulates OUTPUT PARAMS values across output chunks and finds a line split between two chunks. OutputLimiter caps the stored output and emits a truncation marker once.
The handler kept the job entity managed and flushed it from a busy loop, rewriting the whole output on every iteration. A connection dropped inside such a flush left pdo believing in a transaction DBAL had forgotten, and the worker crashed in the retry send with "There is already an active transaction"; the message was redelivered every redeliver_timeout.
Runs the real handler with real subprocesses against a temporary SQLite database. A DBAL driver middleware injects lost connections (including the stale pdo transaction of TT-SERVER-CQ) and the DBAL logging middleware counts statements and transactions. Adds pdo_sqlite to the CI matrix.
docs: document job processing options and messenger recommendations
fix(handler): store only valid UTF-8 and never let rejected output block the result