test: a forward-mode oracle, a cost bound and a SQL-text fuzz; what they found (M4 10)

[alxmrs]

Stacked on #93. This round tests the fixed stack: every soak and every earlier finding passes on it. The goal was to find what the existing oracles can't see.

New techniques

• Forward-mode oracle (exact). Every generated relation gets a hand-written twin: the same SQL with a tangent column dv beside each value. It uses JAX's rule for MAX/MIN (the mean of the tangents of the rows that attain the extreme exactly) and ddx's documented conventions at kinks. DataFusion runs it, and reverse mode's ⟨∇L, d⟩ must match it to 1e-8. It shares no code with the transposes, so it's exact to rounding rather than to a finite difference's step. That lets it see conventions at points a step would cross, and errors a step would hide.
  • Teeth: on its own it kills every value mutant from test: mutation testing for the v2 soak, and the blind spots it found (M4 9) #93, several faster than the finite difference (unreached-one in 3 s instead of 43 s).
  • Result on the fixed stack: about 11,400 comparisons, zero disagreements. The transposes are right on everything the generator reaches.
• Cost bound (cost). The plans DataFusion builds for a program's steps must stay within 200× the forward query's. A superlinear blowup then fails as a finding instead of killing the process. A new fan node (one relation read by several columns) feeds it.
• SQL-text fuzz (tests/ad_sql_text.rs). Valid grad(…) statements with whitespace, CRLF, comments (some holding (, ), quotes or multibyte text), and changes of case and quoting between tokens. Each spelling must give the plain statement's rows or fail loudly.
• Targeted probes past the generator: function semantics (log, atan2, cbrt, round, …), filters pushed into a custom provider's scan (Exact pushdown), window aggregates as values, DISTINCT aggregates, GROUP BY expressions, CASE forms, decimals, computed and inequality join keys, deep and wide plans, and vjp cotangent edge cases.

Found

Each is pinned in ad_findings.rs as an ignored known bug:

finding	severity	notes
Fan-in is exponential. A value read by N columns gets N cotangent terms, folded to skip NULLs (80ca10f). Each fold names the accumulator three times, and DataFusion's Substrait consumer names a column by its expression.	high	10 columns: 117 MB of plan; 12 columns: over 4 GB, on a two-row table. Introduced by a NULL fix.
Deep row-wise chains are exponential. Maps between two aggregates are rebuilt as unnamed appended columns, and names double per layer where a map reads its input twice.	high	14 layers: 61 MB; 20 layers exhaust 13 GB. GELU/LayerNorm-style layers have this shape.
Stack overflow in ddx_ad::grad. Transposer::region clones a rebuilt region recursively, hundreds of relations deep.	high	On a 2 MB stack (a tokio worker's), a 66 KB plan aborts the whole process; nothing can catch it.
Panics and internal errors in grad(…) in SQL. call_span counts parentheses inside comments.	medium	Of 3,000 valid spellings: 953 raised ddx's Internal error, 18 panicked in GradCalls::rewrite, and none returned wrong rows. grad /* c */ (…) isn't recognised either.
MAX/MIN near-ties. The 8-ulp window added against jitter also shares the cotangent between exact values a few ulps apart, where MAX is differentiable.	low	MAX(1, 1+2ulp) gives (0.5, 0.5); jax.grad gives (0, 1). Invisible to any finite difference.
vjp adds up repeated cotangent keys.	low	wrt tables get a uniqueness check; cotangents get none, and a row with a NULL key is dropped silently.
A simple CASE x WHEN … is reported as an invalid plan, which the harness treats as ddx misreading its producer.	low

Not bugs, for the record: filters inside a scan are honoured; unsupported functions are refused rather than mis-derived; NULL-skipping functions (greatest/least with NULL, COALESCE, NVL) match the finite difference. Coverage gaps (refusals of queries ddx could support): LIMIT … OFFSET when the ORDER BY dim isn't selected, and power(v, e) with e built from dims.

Running it

The bounded exact suite runs in CI. cost and the text fuzz are ignored until their bugs are fixed; the soak runs cost every night. A soak can now use a machine's memory, so CONTRIBUTING.md says to run it under systemd-run --user --scope -p MemoryMax=8G. Big mode and the fan node are sized to stay within memory meanwhile.

🤖 Generated with Claude Code

https://claude.ai/code/session_012NE18ox5ivwTc7ZSbHZUGu

[alxmrs]

Thanks: all seven were real, and they're fixed in the stack. This branch is rebased onto it. The findings run as ordinary tests, and cost and the SQL-text fuzz are back in the PR gate.

Finding	Fix
Fan-in exponential	#74: a column's cotangent terms are one flat NULL-skipping sum, with no nested running sum
Deep chains exponential	#79: ad::logical_plan consumes a step with short computed-column names (__ddx_c{n}). The names are DataFusion's, and ddx's plans refer to columns by position. 20 layers now plan small, and the gradient matches the chain rule to 1e-12.
Stack overflow	#74: cotangents are projected in batches, so a step is as deep as the chain of columns reading each other, not as the region is wide. Your 9-layer repro runs on a 2 MB stack.
Comments in grad(…)	#83: GradCalls::find tokenizes with the dialect's tokenizer. It finds grad, then (, across comments, counts parenthesis tokens, and refuses overlapping spans rather than slicing
MAX/MIN near-ties	#76: the tie tolerance applies only to arguments that can jitter (ones that read a saved aggregate or constant data). Table values compare exactly, so MAX(1, 1+2ulp) gives (0, 1)
vjp repeated cotangent keys	#74: a check that the cotangent has one row per key, like a wrt table's dims check. The adapters (#79, #81) now bind a check's reads.
Simple CASE x WHEN …	#71: DataFusion writes it as an IfThen whose first clause is the operand with no result, and a Substrait switch is the standard form. Both are now read as the CASE on x = v.

Changes to your harness, in my commit on this branch:

• consumed_plan_bytes and cost_checks measure plans through ad::logical_plan, which is what ad::run builds, not DataFusion's default consumer.
• The stack-overflow repro's child process no longer passes --ignored; now that the test is un-ignored, that flag would filter it out.

One limit remains: datafusion-python's Substrait consumer can't be given short names, so a very deep chain still makes a large plan from Python. It's documented in ddxdb.ad and design S13.

Verified here:

• All workspace tests pass, including the exact and cost bounded suites.
• ddxdb (68) and the JAX oracle (137) pass.
• Capped soaks found nothing: 1,143 general cases from base 20,000,000 and 540 extreme-value cases from 21,000,000.

🤖 Generated with Claude Code

[alxmrs]

LGTM