[noorcyang]

Draft for team discussion, not ready to merge. Jira: BH-1670

Problem

DagRun.find_duplicate runs WHERE dag_id = ? AND (run_id = ? OR execution_date = ?) on every trigger: experimental API, TriggerDagRunOperator, and the UI trigger. MySQL can't serve that OR from the (dag_id, run_id) / (dag_id, execution_date) unique indexes. It does a ref on the dag_id prefix and filters every run of the DAG.

Measured read-only on the US metadb (Aurora MySQL 8.0.42), 2026-10-28:

query	time
current OR (lake-chrono-deduplication-dag, 2,218,928 runs)	15.0s
rewritten (dag_id AND run_id) OR (dag_id AND execution_date)	13.4s
dag_id AND run_id / dag_id AND execution_date alone	1ms each
UNION of the two	1ms

That's ~98% of each ~14s trigger call for the chrono dedup scheduler (~280 triggers/hr), measured from the webserver's execution_date → queued_at timestamps.

Upstream

Unchanged in 2.9.3 and 2.10.5; the stable API's post_dag_run has its own copy of the OR. apache#34713 was closed as expected behaviour. Airflow 3.0 changes find_duplicate to dag_id + run_id only, so this patch can be dropped once we're on 3.x.

Change

• find_duplicate: look up by run_id, and if nothing matches, by execution_date. Each is a unique-index probe.
• post_dag_run (stable API) calls find_duplicate instead of its own OR query.
• Test for a run_id match and an execution_date match landing on different runs.

Behaviour change (edge case): if run_id matches run A and execution_date matches a different run B, the old code raised MultipleResultsFound (500 from the API). It now returns A, so callers raise DagRunAlreadyExists (409). The stable API already used .first(), so nothing changes there.

Verification

Patched source run against a fresh SQLite metadb: the existing test_dagrun_find_duplicate assertions plus the new split-match case all pass, and ruff --select F is clean. I haven't run the full fork test suite or tested on MySQL; CI / a stage deploy should cover that.

Open for discussion

• Version bump (+affirm5 / +affirm2) and release: not included.
• Carry it on both 2.5.1 and 2.9.3, or only 2.9.3 ([BATCH-4364] Port Affirm fork patches onto upstream 2.9.3 (2.9.3+affirm1) #35)?
• Separately: prune old lake-chrono-deduplication-dag runs.

🤖 Generated with Claude Code