docs(design): record what building M3 settled (M3 10)

[alxmrs]

Stacked on #77. Docs only; closes out M3.

Rewritten after the ontology review on #80. The design now describes per-primitive rules and records why the tagging markers were dropped.

The design's v2 sections were written before any of v2 existed, and several things changed on contact with real plans and with review. This brings the doc in line and records why:

• §2, principle 3 was "tag explicitly; never infer", which asked for a tag on every contraction and reduction. It now reads "never guess; derive, or be told". ddx derives a derivative from what an operator computes, and is told only what it cannot read off the plan: grad/jvp in v1, ddx_stop_gradient in v2.
• §4.2 keeps the Substrait argument and the spike evidence that a ddx-claimed function survives the cross-engine round-trip, without the marker mechanism.
• §4.3 is one transpose rule per relational primitive (map, select, broadcast, reduce), with SUM, AVG, MAX, MIN and COUNT under reduce. A contraction is those rules composed. Both argmax idioms are covered: MAX shares ties like jax.grad, while a rank filter gives the tie to the row its ORDER BY kept. The DuckDB window round-trip bug and its workaround stay. Stop-gradient is the one thing a query tells ddx.
• §4.4 shows grad/vjp as built: dims and values, saved aggregates and recomputed regions (the checkpointing choice), fan-in as UNION ALL plus SUM, gradients shaped like their tables, and late-bound reads.
• §4.5's worked example is now nn.py's own loss query, unchanged: there is no migration cost.
• §4.6 and §8 record what M3 built and how it was checked.
• Decision log S6–S10:
  • S6: save aggregates, recompute regions;
  • S7: bind step reads late;
  • S8: relations are dims and values;
  • S9: functions matched by name;
  • S10: the markers were dropped, and why.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CvszJhH8pn8H9G69wEPMU2

[alxmrs]

🤖😈 Adversarial review — #78 (design doc for M3)

The rewrite reads well, and dropping the markers is the right call. §4.3's per-primitive table is much easier to trust than the five special-cased rules. Three places where the doc now says something the code doesn't back up:

1. Principle 5 vs. S8. §4.4 and S8 make "dims identify a row" the foundation of the gradient's shape, and relation.rs concedes that a table breaking it gets summed gradients. That is a silent-wrong mode, and the doc should name it as one, not bury it. It is demonstrated on feat(ddx-ad): one transpose per relational primitive; grad and vjp (M3 6) #74 (a gradient of 6, 6, 6 instead of 2, 4, 6, and one SGD step turns 3 rows into 5). Either add a run-time uniqueness check (my preference; see feat(ddx-ad): relations as dims and values; the forward pass read (M3 5) #73) or state it in §4.6 as a known silent-wrong mode, next to v1 differentiates a cast to an integer type as the identity #87.
2. §4.1 "GELU/ReLU … reduce to the same primitive set." Not today: CASE and greatest over a varied value are refused (details on feat(ddx-ad): the map primitive's local derivatives, via ddx-core (M3 3) #71). Only abs-based or filter-based formulations work.
3. §4.4 "A region is rebuilt with the same relations in the same order, so it produces the same rows." That holds for DataFusion today; I tried and failed to break it with rank ties across 8 partitions (see feat(ddx-ad): AVG, MAX and MIN rules; rank-select and stop-gradient (M3 8) #76). But it is an assumption about engines, not a property of plans, and M5's engine is the one to worry about. Please list it under §4.6 "genuinely open", with the mitigation: save window columns instead of recomputing them, or refuse ORDER BYs that aren't total.

Also missing from §4.6: the tape currently lives in the caller's catalog under global names, which makes concurrent use of one context unsafe (#79). Whatever the fix is, it deserves a decision-log entry beside S6/S7.

[alxmrs]

All four recorded in 6b5d753, with decision-log entry S11.

1. Dims identify rows: now a run-time check, not a silent-wrong mode (feat(ddx-ad): relations as dims and values; the forward pass read (M3 5) #73/feat(ddx-ad): one transpose per relational primitive; grad and vjp (M3 6) #74). §4.4 says so, and S8 records the change.
2. ReLU: the claim is true again, since CASE/greatest/least have rules (feat(ddx-ad): the map primitive's local derivatives, via ddx-core (M3 3) #71). §4.1 and the map row name them.
3. Region rows: §4.4 states the assumption and the ranking-totality check that enforces it (feat(ddx-ad): AVG, MAX and MIN rules; rank-select and stop-gradient (M3 8) #76). §4.6 keeps the part that is still open, volatile functions and scans that aren't repeatable.
4. The tape: S11 records the per-program prefix, and that an adapter drops a program's intermediates once it has run (feat(ddx-datafusion): grad, vjp and run, the v2 API on DataFusion (M4 1) #79).

[alxmrs]

Added docs(design): record the v2 soak's findings and the fixes (S12). It covers:

• the MAX/MIN windows, the NULL-skip mask, and the wider "same rows" checks (§4.3, §4.4);
• volatile functions leave §4.6's open list, and near-tie jitter joins it;
• a new decision-log entry, S12.

🤖 Generated with Claude Code

[alxmrs]

Added docs(design): record the second soak round (S13): size, not calculus, along with each fix and the one remaining limit, deep chains from Python.

🤖 Generated with Claude Code

[alxmrs]

Added a sentence to §4.4 stating which MAX/MIN arguments get the 8-ulp tie tolerance (only those read from an aggregate's output), and sharpened the S13 note to match. Follows #76's fix for #102.

🤖 Generated with Claude Code

[alxmrs]

😈🧪 Adversarial tester: ✅ approved for correctness. Reviewed at d2c7bdc (identical tree to today's rebase febfc15). Workspace tests, doctests, a 20-min soak (0 failures) and a mutation spot-check all pass. The one blocker in the stack is on #79 (#79 (comment)); the full report is in #102 (comment).

[alxmrs]

TL;DR: Design update is fine; this is basically internal only.