[DiegoTavares]

Follow-up to #2506: a restarted RQD answers kill-before-release with NOT_FOUND, which was wrapped as a failure, so lostProc deferred provably dead frames indefinitely and re-logged the "Deferring release of lost proc" WARN every 30s.

• RqdClientGrpc.killFrame now treats NOT_FOUND as confirmed-stopped (matching isFrameRunning); only unknown outcomes throw RqdClientException.
• Bound the deferral: after dispatcher.lost_proc_max_defer_ms (default 20 min) without a ping proving the frame alive, release the proc and fail the frame closed to DEAD (manual retry) instead of deferring again. Negative disables the bound.
• Boot-time fence: a host whose reported boot time is later than the proc's dispatch (plus a clock-skew margin) cannot be running the frame, so release it to WAITING even when the kill cannot be delivered.
• Orphaned-proc reaper: order candidates oldest-ping first (deterministic batch turnover) and make the batch size configurable (maintenance.orphaned_proc_batch_size, default 100) so unconfirmable procs cannot starve releasable orphans.

New unit tests cover the fail-closed bound, the boot fence, and killFrame status classification against a real localhost gRPC server; DAO tests cover the new queries.

LLM usage disclosure

Claude Opus was used for investigating the logs and implementing this fix.

Summary by CodeRabbit

• Bug Fixes

  • Improved handling of lost processes by detecting stale pings and host reboots.
  • Automatically marks frames as DEAD when lost-process recovery cannot be confirmed within the configured time limit.
  • Treats already-missing frames as successfully stopped during kill operations.
  • Orphaned-process cleanup now processes oldest entries first.

• Configuration

  • Added settings for lost-process deferral limits and orphaned-process cleanup batch size.

[coderabbitai]

📝 Walkthrough

Walkthrough

The change adds proc ping and reboot checks, bounds deferred lost-proc handling, treats RQD NOT_FOUND as a successful stop, and makes orphaned-proc batch size configurable.

Changes

Proc recovery and maintenance

Layer / File(s)	Summary
Proc detection queries cuebot/src/main/java/com/imageworks/spcue/dao/ProcDao.java, cuebot/src/main/java/com/imageworks/spcue/dao/postgres/ProcDaoJdbc.java, cuebot/src/test/java/com/imageworks/spcue/test/dao/postgres/ProcDaoTests.java	The DAO detects stale pings and host reboots after dispatch. Orphaned-proc results are ordered by oldest ping. PostgreSQL tests cover both checks.
Bounded lost-proc release cuebot/src/main/java/com/imageworks/spcue/dispatcher/DispatchSupportService.java, cuebot/src/main/resources/opencue.properties, cuebot/src/test/java/com/imageworks/spcue/test/dispatcher/DispatchSupportServiceLostProcTests.java	Lost-proc handling supports reboot confirmation, configurable deferral bounds, indefinite deferral when disabled, and DEAD state assignment after expiry.
RQD kill outcome contract cuebot/src/main/java/com/imageworks/spcue/rqd/RqdClient.java, cuebot/src/main/java/com/imageworks/spcue/rqd/RqdClientGrpc.java, cuebot/src/test/java/com/imageworks/spcue/test/rqd/RqdClientGrpcTests.java	RQD NOT_FOUND now confirms that a frame is stopped. Other failures still raise RqdClientException. Integration-style tests cover these outcomes.
Configurable orphaned-proc batches cuebot/src/main/java/com/imageworks/spcue/service/MaintenanceManagerSupport.java, cuebot/src/main/resources/opencue.properties	clearOrphanedProcs reads maintenance.orphaned_proc_batch_size and falls back to 100.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant DispatchSupportService
  participant ProcDaoJdbc
  participant RqdClientGrpc
  DispatchSupportService->>ProcDaoJdbc: Check ping age and host reboot
  DispatchSupportService->>RqdClientGrpc: Kill frame
  RqdClientGrpc-->>DispatchSupportService: Return success, NOT_FOUND, or error
  DispatchSupportService->>DispatchSupportService: Stop frame as WAITING or DEAD

Loading

Possibly related PRs

• AcademySoftwareFoundation/OpenCue#2441: Both changes modify DispatchSupportService.lostProc and related tests.
• AcademySoftwareFoundation/OpenCue#2471: This change extends the same deferred-release logic with ping and reboot checks.
• AcademySoftwareFoundation/OpenCue#2473: Both changes cover orphaned-proc cleanup and terminal frame states.

Suggested reviewers: lithorus, ramonfigueiredo

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary change: preventing lost processes from being deferred indefinitely.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cuebot/src/main/java/com/imageworks/spcue/dao/postgres/ProcDaoJdbc.java`:
- Around line 582-586: Update the orphan query’s ORDER BY clause in ProcDaoJdbc
to sort first by proc.ts_ping and then by proc.pk_proc, preserving oldest-first
ordering while making limited batches deterministic for equal timestamps.

In
`@cuebot/src/main/java/com/imageworks/spcue/service/MaintenanceManagerSupport.java`:
- Around line 206-208: Validate batchSize in checkHardwareState() before calling
procDao.findOrphanedVirtualProcs(); when maintenance.orphaned_proc_batch_size is
non-positive, replace it with DEFAULT_ORPHANED_PROC_BATCH_SIZE so the cleanup
query always receives a valid limit.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b2f8c61b-17d0-4fe6-aa67-44fa3a27c6aa

📥 Commits

Reviewing files that changed from the base of the PR and between d9f5ba2 and 2600eb5.

📒 Files selected for processing (10)

• cuebot/src/main/java/com/imageworks/spcue/dao/ProcDao.java
• cuebot/src/main/java/com/imageworks/spcue/dao/postgres/ProcDaoJdbc.java
• cuebot/src/main/java/com/imageworks/spcue/dispatcher/DispatchSupportService.java
• cuebot/src/main/java/com/imageworks/spcue/rqd/RqdClient.java
• cuebot/src/main/java/com/imageworks/spcue/rqd/RqdClientGrpc.java
• cuebot/src/main/java/com/imageworks/spcue/service/MaintenanceManagerSupport.java
• cuebot/src/main/resources/opencue.properties
• cuebot/src/test/java/com/imageworks/spcue/test/dao/postgres/ProcDaoTests.java
• cuebot/src/test/java/com/imageworks/spcue/test/dispatcher/DispatchSupportServiceLostProcTests.java
• cuebot/src/test/java/com/imageworks/spcue/test/rqd/RqdClientGrpcTests.java