test: a simulation soak for query-level AD, and three bugs it found (M4 5)

[alxmrs]

Adversarial property/fuzz testing for the M3/M4 stack (#69–#83): a generator of loss queries, a finite-difference oracle that shares nothing with ddx, a set of metamorphic relations, and a nightly soak. Stacked on #82.

What it checks

crates/ddx-datafusion/tests/ad_simulation.rs generates loss queries as DAGs of CTEs over small random tables. It uses every primitive in design.md §4.3 and the shapes around them: maps (including about 10% drawn from ddx-core's own gen_expr), joins on shared dims and cross joins, LEFT JOIN with COALESCE, CASE on data, grouped SUM/AVG/MAX/MIN, repeated identical aggregates, HAVING, dim and value filters, IN/NOT IN/EXISTS semi-joins, rank filters, ORDER BY … LIMIT, UNION ALL, DISTINCT, softmax with and without ddx_stop_gradient on the shift, and scalar subqueries over data. Tables get NULLs, orphan rows, repeated keys in data, shuffled rows, and BIGINT/INT/VARCHAR dims.

Oracle. The loss is a query DataFusion can run without ddx, so each gradient is checked against (L(θ+hd) − L(θ−hd))/2h. It uses random directions and single entries. The loss is screened at kinks (the second difference must shrink like h²) and Richardson-extrapolated. A screened point is counted, and the bounded suites assert how many cases were accepted.

Metamorphic relations, where ddx is compared with itself:

• the calculus: ∇(cL), ∇(L+c), ∇(L²), ∇sin L, a loss CTE read twice, L·sg(L), L + sg(L);
• grad = vjp seeded with 1 (and 2.5); vjp(R, c) = grad(Σ R·c); vjp is linear in c;
• invariance to CTE vs inline subqueries, the unoptimized plan, wrt order, case and splitting, row order, and partitions 1/4/7;
• the program contract: a program built at θ₀ and run at θ₁ (sometimes with extra rows) equals a fresh one; repeated dims are refused by the checks; after run only the value and gradients remain, after release nothing does, and user tables are untouched; three programs spawned concurrently on one context don't interfere (S11);
• the SQL surface: grad(loss, t.val) equals the program, and an SGD join equals θ − 0.1·∇L. The statement is spelled with GRAD/Grad (, comments holding grad( and multibyte text, a quoted "Loss Fn" CTE, W.VAL, and extra CTEs around the loss;
• shape: one gradient row per table row, Float64, NULL exactly where the value is NULL.

A refusal is always allowed and is tallied by kind. A panic, an Internal or InvalidPlan error, a program that is accepted but fails to run, or a wrong number is a failure.

Running it. Six bounded suites run under cargo test (24 seeds each, about 16s in debug). soak_v2_query_ad is #[ignore]d. It uses the same DDX_SOAK_* knobs and log format as ddx-core's soak, and now runs nightly as a second job. .github/scripts/report_fuzz_findings.sh now reads each soak's own REPRO line, and no longer counts a top-level log twice. DDX_V2_SEED=<n> with replay_one_seed prints a case's SQL and tables. DDX_V2_NULL_PCT=0 turns off NULL injection, to hunt past the NULL bug; it replays the same case minus its NULLs, which is how every NULL failure below was attributed.

What it found

Soaks totalled about 6,500 cases, of which about 84% were accepted. That gave about 17k finite-difference comparisons (fewer than 0.1% screened) and about 100k metamorphic comparisons. Every failure maps to one of three bugs, pinned in tests/ad_findings.rs as #[ignore = "known bug: …"] repros with hand-worked gradients (CONTRIBUTING's failing-test-first convention; -- --ignored shows all four failing):

1. Silently wrong gradient at NULL rows. SUM, AVG, MAX and MIN skip a row whose argument is NULL, but ddx still sends the group's cotangent through that row's other inputs. In SUM(p.val + q.val) with q NULL at row 1, ddx gives ∂/∂p(1) = 1 where the answer is 0. This happens whether the NULL is in a parameter or in constant data, and it accounts for every finite-difference failure (all pass once the case's NULLs are removed).
2. Two rank filters in one recomputed region are accepted, then fail to run. The optimizer gives both window columns the same name, and the rebuilt region keeps both.
3. An unoptimized plan with a constant join condition under a CASE is accepted by grad_plan (which takes any LogicalPlan, a DataFrame's included), and then its backward step fails DataFusion's schema check.

No failures came from the calculus identities, vjp, wrt handling, the SQL rewriter, dim types, program reuse, concurrency or catalog hygiene.

Also worth knowing, though not a failure: ddx refuses some rankings the generator makes total over the CTE's dims ("a window function whose PARTITION BY and ORDER BY…", "over rows joined to…"). These are conservative refusals, not wrong answers, but they are coverage the rank-select rule could have.

Not covered

The Python ddxdb.ad layer (it wraps the same Rust calls), Float32 value columns (the finite-difference oracle would need f32 tolerances), and WITH RECURSIVE, which ddx refuses.

🤖 Generated with Claude Code

https://claude.ai/code/session_012NE18ox5ivwTc7ZSbHZUGu

[alxmrs]

Thanks: all three findings were real ddx bugs, and they're fixed in the stack below. This branch is rebased onto it, and ad_findings.rs now runs the repros as ordinary tests, each naming its fix.

Finding	Fix
NULL rows leak gradient	#74, #76: each reduce rule's seed is NULL where its argument is NULL. Later soaks found three more ways the same leak got through, all fixed: vjp's seed (#74), a NULL term added to a real one in a multi-aggregate loss (#74), and a MAX/MIN row that doesn't attain the extreme (#76).
Two rank filters collide	#73: each window column is renamed in place as soon as it's computed
Unoptimized CASE over integer data	#79: a step's table takes the schema of the physical plan that ran

My commits on this branch:

• un-ignore: takes the three repros off #[ignore].
• stopped-softmax fix: a softmax whose shift is stopped no longer offers its exponentials and their sum for reuse. Only the ratio is shift-invariant, so reading Σ exp(u − sg(max)) directly makes stop-gradient's gradient differ from the finite difference by definition (seed 100607; JAX agrees with ddx).

🤖 Generated with Claude Code

[alxmrs]

Composability review: the oracles are an asset, make them reusable

Praise: the finite-difference oracle, the simulation generator shared with ddx_core::test_utils, and the mutation-testing pass (#93) are the real evidence that this AD is correct. They are also the most valuable thing a second adapter could inherit, because "matches jax.grad/finite difference on random plans" is the conformance test for any engine.

Suggestion: ad_simulation.rs (about 5,000 lines) is coupled to SessionContext. If the harness took a small engine trait (run SQL, produce Substrait, run a program, return a table) the same suite could run against the DuckDB adapter and any other, and would double as the "is your Substrait producer good enough for ddx?" test that other projects would want. Cheap to do now, expensive once a second engine has its own copy.

[alxmrs]

Re: reusable oracles

Agreed it's valuable, but I'd do it once a second adapter is in the repo, not now. The harness's engine surface is wider than run-SQL / produce-Substrait / run-a-program: optimizer variants, Substrait mutations, partition and row-order rewrites. The right trait is easier to cut against a real second engine than to guess at. The first half exists now: ddx_ad::Backend (#79) is the 'run a program' part, and tests/backend.rs already drives DataFusion through it. Recorded in design.md S14 (#82). One small change here: the refusal classifier treats an AdError kind it doesn't know as a bug (14f7d87), since AdError is now non_exhaustive.

🤖 Generated with Claude Code

[alxmrs]

LGTM