Skip to content

fix(handler): run jobs without open transactions and survive lost connections (#9) - #10

Merged
Matys333 merged 1 commit into
mainfrom
dev
Oct 8, 2026
Merged

Matys333 merged 1 commit into
mainfrom
dev

Conversation

@Matys333

@Matys333 Matys333 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Changes

  • feat(output): add chunk-safe output params parser and output limiter

OutputParamsParser accumulates OUTPUT PARAMS values across output chunks and finds a line split between two chunks. OutputLimiter caps the stored output and emits a truncation marker once.

  • fix(handler): run jobs without open transactions and survive lost connections

The handler kept the job entity managed and flushed it from a busy loop, rewriting the whole output on every iteration. A connection dropped inside such a flush left pdo believing in a transaction DBAL had forgotten, and the worker crashed in the retry send with "There is already an active transaction"; the message was redelivered every redeliver_timeout.

  • JobOutputWriter writes output, status and result through DBAL in autocommit and resets the connection after a lost connection or a stale native transaction
  • the job is claimed atomically (planned -> running), redelivered messages of running/finished jobs do not run the command again
  • the wait loop sleeps 100 ms and writes at most once per poll interval; database errors are tolerated up to db_failure_tolerance
  • a failed command is a job status; only an unstorable result is thrown, as UnrecoverableMessageHandlingException
  • a missing job is rejected without retry
  • new config job_queue.processing: poll_interval_ms, output_max_bytes, db_failure_tolerance, rerun_on_redelivery
  • test(handler): add SQLite integration tests for job processing

Runs the real handler with real subprocesses against a temporary SQLite database. A DBAL driver middleware injects lost connections (including the stale pdo transaction of TT-SERVER-CQ) and the DBAL logging middleware counts statements and transactions. Adds pdo_sqlite to the CI matrix.

  • docs: document job processing options and messenger recommendations

  • fix(handler): store only valid UTF-8 and never let rejected output block the result

  • Utf8Stream holds back a character split between two pipe reads and scrubs bytes which are not UTF-8; OutputLimiter cuts at a character boundary. A strict MySQL utf8mb4 column rejected the broken character (1366) forever.
  • Output a working database rejects (not a lost connection) is replaced by a note in the loop and in finalize, so it can neither kill a healthy command nor keep the result from being stored; a rejected result write is retried without output params and status message. Stored messages are bounded.
  • The writer keeps the caller's transaction nesting level for every reset, not only in the handler's finally block.
  • Finalize appends its status message only below the output cap.
  • Loading and claiming the job are retried (a lost claim response is verified by status + started_at) and end in a recoverable exception; a missing job is looked up again before it is rejected.
  • A rerun claim appends its note in the claim UPDATE, so MySQL always reports the changed row.
  • A job deleted mid-run stops its command; Process output buffers are cleared after each read; stdout and stderr are parsed separately.
  • The controller cancels by a conditional UPDATE, so it never overwrites a stored result.
  • Tests: handler-level TT-SERVER-CQ guard with a non-ConnectionLost driver error plus a stale native transaction (loop and finalize), multibyte cap and split reads, rejected output, caller transaction, deleted job, lost claim race, pre-claim retries.
  • CI: symfony/flex so SYMFONY_REQUIRE applies, DBAL 3 / ORM 2 legs; conflict with symfony/dependency-injection 8. Docs and changelog updated.

…nections (#9)

* feat(output): add chunk-safe output params parser and output limiter

OutputParamsParser accumulates OUTPUT PARAMS values across output chunks
and finds a line split between two chunks. OutputLimiter caps the stored
output and emits a truncation marker once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(handler): run jobs without open transactions and survive lost connections

The handler kept the job entity managed and flushed it from a busy loop,
rewriting the whole output on every iteration. A connection dropped inside
such a flush left pdo believing in a transaction DBAL had forgotten, and the
worker crashed in the retry send with "There is already an active
transaction"; the message was redelivered every redeliver_timeout.

- JobOutputWriter writes output, status and result through DBAL in
  autocommit and resets the connection after a lost connection or a stale
  native transaction
- the job is claimed atomically (planned -> running), redelivered messages
  of running/finished jobs do not run the command again
- the wait loop sleeps 100 ms and writes at most once per poll interval;
  database errors are tolerated up to db_failure_tolerance
- a failed command is a job status; only an unstorable result is thrown,
  as UnrecoverableMessageHandlingException
- a missing job is rejected without retry
- new config job_queue.processing: poll_interval_ms, output_max_bytes,
  db_failure_tolerance, rerun_on_redelivery

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(handler): add SQLite integration tests for job processing

Runs the real handler with real subprocesses against a temporary SQLite
database. A DBAL driver middleware injects lost connections (including the
stale pdo transaction of TT-SERVER-CQ) and the DBAL logging middleware counts
statements and transactions. Adds pdo_sqlite to the CI matrix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: document job processing options and messenger recommendations

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(handler): store only valid UTF-8 and never let rejected output block the result

- Utf8Stream holds back a character split between two pipe reads and scrubs
  bytes which are not UTF-8; OutputLimiter cuts at a character boundary. A
  strict MySQL utf8mb4 column rejected the broken character (1366) forever.
- Output a working database rejects (not a lost connection) is replaced by a
  note in the loop and in finalize, so it can neither kill a healthy command
  nor keep the result from being stored; a rejected result write is retried
  without output params and status message. Stored messages are bounded.
- The writer keeps the caller's transaction nesting level for every reset,
  not only in the handler's finally block.
- Finalize appends its status message only below the output cap.
- Loading and claiming the job are retried (a lost claim response is
  verified by status + started_at) and end in a recoverable exception; a
  missing job is looked up again before it is rejected.
- A rerun claim appends its note in the claim UPDATE, so MySQL always
  reports the changed row.
- A job deleted mid-run stops its command; Process output buffers are
  cleared after each read; stdout and stderr are parsed separately.
- The controller cancels by a conditional UPDATE, so it never overwrites a
  stored result.
- Tests: handler-level TT-SERVER-CQ guard with a non-ConnectionLost driver
  error plus a stale native transaction (loop and finalize), multibyte cap
  and split reads, rejected output, caller transaction, deleted job, lost
  claim race, pre-claim retries.
- CI: symfony/flex so SYMFONY_REQUIRE applies, DBAL 3 / ORM 2 legs;
  conflict with symfony/dependency-injection 8. Docs and changelog updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@Matys333
Matys333 merged commit 4a5fb07 into main Oct 8, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants