[NopeCharon]

Problem

Deleting a job (DELETE /api/v1/namespaces/{namespace}/jobs/{job}) only marks the job as hidden; its current rows in job_versions_io_mapping are kept. LineageDao.getLineage() built its recursive traversal from those rows without checking whether the job was still visible, so a deleted job kept connecting the datasets it consumed and produced. The final join on jobs_view then dropped the deleted job itself, which left datasets and jobs in the graph with no visible edge linking them to the requested node.

For d1 → job1 → d2 → job3 → d3 → job2 → d4, after deleting job3 the lineage for d1 still contained d3 → job2 → d4.

Closes: #3085

Solution

• LineageDao.getLineage(): the job_io CTE now joins jobs_view (resolving symlinked jobs to their target, the same way the final join already does), so only visible jobs are traversed and the graph stops at the deleted job.
• LineageDao.getJobFromInputOrOutput(): skip hidden jobs when choosing the job to start from for a dataset node. Without this, lineage for a dataset whose producer was deleted (d3 above) would start from the deleted job and come back as an orphan graph. It now starts from a visible consumer, so d3 → job2 → d4 is returned.

No schema changes.

One-line summary: API: Exclude deleted jobs from lineage graph traversal

Checklist

• You've signed-off your work
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
• You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
• You've included a header in any source code files (if relevant)

Testing

• New LineageDaoTest.testGetLineageWithDeletedJob and LineageDaoTest.testGetJobFromInputOrOutputIgnoresDeletedJob; both fail without this change.
• LineageDaoTest, LineageServiceTest, JobDaoTest, ColumnLineageDaoTest, OpenLineageServiceIntegrationTest, OpenLineageIntegrationTest, and JobResourceIntegrationTest pass.
• Replayed the scenario from the issue against /api/v1/lineage in a local integration run: after deleting job3, lineage for d1 only contains d1 → job1 → d2, and lineage for d3 contains d3 → job2 → d4.
• ./gradlew :api:spotlessJavaCheck passes.

[boring-cyborg]

Thanks for opening your first pull request in the Marquez project! Please check out our contributing guidelines (https://github.com/MarquezProject/marquez/blob/main/CONTRIBUTING.md).