feat(ddx-ad): read a marked Substrait plan — markers, column tracing, activity analysis (M3 1/2)

[alxmrs]

The first half of M3 (#13, #14): ddx-ad can now read a marker-tagged Substrait
plan and say which columns carry gradient. Nothing differentiates anything yet —
the five transpose rules are #15, and this is what they stand on.

This supersedes #66, which was closed unmerged. I branched from main and
rewrote the ddx-ad code, keeping only the setup pieces from that branch
(protoc in CI and the Nix shell, the CONTRIBUTING note, the substrait
dependency) after reading them line by line.

The last commit is an ontology pass over the first three: the vocabulary is
checked against Substrait's, the AD literature's, and this project's own, and
fixed where they disagreed. Details at the bottom.

What's here

• Marker — the four functions ddx claims the name of, recognized by name,
  case-folded, ignoring any :signature suffix. It has to be by name:
  DataFusion 54 declares every function, its own built-ins and
  ddx_contract_mark alike, with a bare name and
  extension_urn_reference = u32::MAX. Substrait 0.63 moved from URIs to URNs
  and DataFusion fills in neither (decision log S7).
• PlanIndex — relations in an order where a node comes after its consumer,
  plus the named tables each read belongs to.
• Columns — every output column traced back to its origin, through the
  Project/emit layers and read masks a real producer inserts. Without this,
  §4.3's "read the contracted dim off the join condition" can't be done: in
  DataFusion's output the aggregate doesn't sit directly on the join.
• Activity — varied (depends on a wrt column, outside any
  ddx_stop_gradient) ∧ useful (reaches the output through a value position,
  not only a join condition, filter, grouping key or sort).
• Analysis — the above for one plan, plus tag, don't infer enforcement.
• ddx_datafusion::register_ad_markers — the four markers as identity UDFs,
  so a tagged forward query plans and runs, returning the unmarked answer.

Why wrt names columns instead of a naming convention

The rules need to know which columns are values and which are dims. A
convention — one value column per relation, named val — would have been
simpler, but nn.py breaks it in three places, and breaking it yields a wrong
gradient rather than an error: fwd0 carries both z and val, so z would be
read as a dim and the chain through tanh would be cut; pixels names its value
images. So you name the wrt columns and ddx derives the rest (decision log S6).

A column plays one of three roles, not two — dim, val, or constant — and ddx
infers only the last distinction. Dim-ness is structural: the plan already states
which columns identify a row, in join conditions and GROUP BY lists, so the
rules read it there. "Carries no gradient" is never "is a dim": pixels.images
is neither.

This also makes the tagging rule checkable. Over a gradient-carrying column, an
untagged SUM, a MAX/MIN, or AVG(ddx_contract_mark(…)) is refused with an
error naming the fix; over data alone the same aggregates need no tag. A wrt
typo lists the tables and columns that do exist, instead of returning zero.

Tests

197 tests pass across the workspace. 31 are ddx-ad unit tests on hand-built
plans; 10 are integration tests on plans DataFusion produces for nn.py's SQL,
changed only by the markers:

Shape	What it pins
layer 0	z, val carry gradient; inp (a computed dim) and images (data) don't; the contraction is tagged
layer 0, wrt bias only	the same outputs stay active, through + b.val; no measure carries gradient
two layers + a scalar loss	weight read twice (fan-in); two contractions and one reduction
the softmax shift	accepted with ddx_stop_gradient(m.m); refused without, naming the marker
the Route idiom	the routed val is active; ROW_NUMBER() isn't
WHERE sample IN (…)	DataFusion's semi-join is handled as a mask
untagged contraction	refused
wrt typo	error listing the real tables/columns
marked vs. unmarked query	identical numbers — the markers are identities

tests/substrait_pin.rs fails if two substrait versions ever resolve, the
same guard sqlparser_pin.rs gives Path B. cargo publish --dry-run -p ddx-datafusion passes, so the path-only dev-dependency on the unpublished
ddx-ad doesn't block releases.

The ontology pass (last commit)

Each rename fixes a wrong claim, not a style preference:

• RelRef → TableRef. In Substrait a "Rel" is any relation node — the
  index holds one per node — while only a named table can be a wrt. Both
  meanings sat in one struct.
• Param → ColumnRef. The unit is a table column, and it needn't be a
  parameter: attention_ad_spike.py differentiates with respect to X, the
  input. It is now the plain twin of ddx_core::ColRef.
• Marker::Contract → Contraction, matching §4.3's rule name, plus
  Marker::is_tag: ddx_stop_gradient changes the gradient, while the other
  three only classify an operation whose gradient is already determined. Four
  markers, five rules, because Elementwise needs none.
• One spelling each for dim and val, replacing dim ~ coordinate ~ key and
  val ~ variable ~ value.
• AggKind::from_name replaces "sum"/"max" literals: a producer's
  function name is engine vocabulary, so it belongs in one table — the
  recognizer half of the per-engine table S7 says the emitter needs.
• Col vs Field — an output column of a node and a position in its input
  row are separate numberings, handled in the same loop in activity.rs, and
  nothing previously stopped one being passed as the other.
• AdError splits InvalidMarker (malformed or misplaced) from Untagged
  (no marker written at all) and gains Internal, replacing an unreachable!()
  that asserted an invariant across two modules.
• Documented that NodeKind::Window is not where Route is found on
  DataFusion — it emits ROW_NUMBER() as a window-function expression inside a
  Project (verified against the producer) — and that all fan-in is table-level,
  because Substrait plans are trees.

Design doc

§4.4 takes wrt: &[ColumnRef], keys gradients by it, states the three column
roles, and has its fan-in text corrected: it said contributions are summed "via
an ordinary elementwise-add step", which relationally is an inner join, and
cotangents are sparse. nn.py's per-layer weight reads are the concrete case —
an inner join of the three layer contributions matches no rows and returns an
empty gradient with no error. It now says UNION ALL then GROUP BY the dims.
Decision log S6–S9.

Still open, for #15

• Outer joins are traced like inner joins. Columns line up, but an outer
  join's NULL-extended rows have no transpose rule, so the join rule should
  refuse them explicitly.
• The seed is every varied root column. Fine for a scalar loss; vjp_query
  (Implement vjp_query + BackwardProgram (reverse-topo walk, cotangent accumulation) #18) may want the caller to pick.
• Erring towards "varied" means conditions and CASE tests count as
  dependencies, so a column can be called active where its true derivative is
  zero. That direction is deliberate.

Closes #13. Closes #14.

🤖 Generated with Claude Code

[alxmrs]

🤖⚔️ Adversarial review — M3 1/2 (analysis only)

I read docs/design.md on main and on this branch first, then the code, then went looking for places where the two disagree. Everything below was run, not reasoned about: probes against ddx-ad's own test_plans builders and against real to_substrait_plan output using this PR's own nn.py fixtures. Scratch probe files were deleted; the repro SQL is inline in each comment.

cargo test -p ddx-ad (31), clippy --workspace --all-targets -D warnings, and the full workspace suite are green on this branch.

What's genuinely good

• The Col/Field split. Two numberings that look like usize and aren't is the thing most plan-walkers get wrong; separating them at the type level and then testing resolve_input through a real Project+emit over a join is the right call, made early.
• Testing the analysis against producer output, not only hand-built plans. ad_analysis.rs is the test file that will catch the next DataFusion upgrade.
• substrait_pin.rs::the_plan_types_are_actually_the_same_type — a compile-time assertion instead of a string comparison. Correct instinct.
• an_untagged_sum_over_data_alone_is_fine — the test that proves tag, don't infer isn't just "refuse every aggregate".
• The Slot::describe comment claims DataFusion plans SUM(DISTINCT x) as a grouping feeding an outer sum. I checked it; it's true (Aggregate: groupBy=[[ddx_reduce_mark(val) AS alias1]] → Aggregate: aggr=[[sum(alias1)]]), and the refusal lands in the "condition, grouping key or sort key" slot exactly as documented. Claims in this PR held up when I tested them.
• I also verified the one CI gap I expected to find isn't one: cargo publish --dry-run -p ddx-datafusion still passes with protoc off the PATH, because path dev-dependencies are stripped at package time. The package job doesn't need the new step.

Findings, worst first

1. Type coercion moves the v2 markers (ddx-datafusion/src/markers.rs) — SUM(ddx_reduce_mark(f32_col)) is refused as "nested inside a larger expression". Correct SQL, confusing refusal, and it's the same coercion hazard v1's Marker::new comment was written to avoid.
2. useful leaks through window ORDER BY/PARTITION BY (expr.rs) — in the expression form DataFusion actually emits. A SUM whose only consumer is a window sort key is refused as untagged, contradicting §4.4's definition verbatim.
3. Every varied root column is seeded as an output (activity.rs) — a plan returning loss and other silently yields d(loss + other)/dw. vjp_query has no way to name the output.
4. COUNT(x) over an active column is refused (analysis.rs) — contradicting AggKind::Count's own doc and §4.3's "SUM then divide by the count" recipe. COUNT(*) works; COUNT(val) doesn't.
5. Outer joins pass silently (columns.rs) — no rule covers one, and NULL ≠ the zero [S8] relies on.
6. Fan-in keyed on exact table-name spelling (index.rs) — w and public.w are two tables, and the missing half of the gradient is silent.
7. Public API panics on an out-of-range Col/NodeId, in the crate that added AdError::Internal to avoid panicking.
8. Lower severity: integer Cast passes gradient through; IfThen conditions treated as value positions; is_useful's doc describes something it doesn't compute; UNION ALL unreadable though [S8] makes it the emitted fan-in primitive; S8/S9 out of order in the decision log.

Two of these (1, 2) are false refusals of SQL the design tells users to write — they cost nothing today and cost a support thread each once M4 ships. Two (3, 5) are silently-wrong-answer shapes that are cheap to close now, while the rules are still unwritten, and much harder after.

Nits, no reply needed

• markers.rs:100 — name.split(':').next() is infallible; the .unwrap_or(name) branch is unreachable. Same in index.rs:273 and expr.rs:217.
• names.rs — sum0 maps to Sum. Substrait's sum0 returns 0 on empty input where sum returns NULL; treating them identically is right for AD, but worth half a line saying so, since this table is explicitly the place a third engine's spelling gets added.
• index.rs:210-213 — the MAX_DEPTH rationale is off by the nesting factor: prost's 100-level cap counts messages, and each Substrait relation is ~3 messages deep, so the decoder caps you nearer 33 relations than 128. The guard is fine; the stated reason isn't the one that makes it safe.
• analysis.rs — an untagged AVG over an active column gets the generic "aggregate avg over a gradient-carrying column", while the tagged one gets the helpful "for a mean, SUM and then divide by the count". The untagged path is the likelier user mistake and gets the worse message.
• activity.rs:150 — check_wrt inspects only reads[0]'s schema. I verified it's harmless with DataFusion (base_schema is the full table schema even under a projection mask), but that's an assumption worth asserting rather than inheriting.
• A wrt column that exists in base_schema but is masked out of every read passes check_wrt, then fails with "the query's output doesn't depend on any of the wrt columns … check any ddx_stop_gradient" — misleading; the column was never read at all.
• Duplicate entries in wrt are accepted silently.
• ddx_stop_gradient is accepted anywhere (Marker::StopGradient => true in the placement table), including in a grouping key or join condition where it does nothing. It's the one marker that changes semantics rather than classifying, so a placement where it's a no-op is arguably the one worth refusing loudest.

Disagreement welcome on any of these — several are judgement calls about where the M3/M4 line falls, and you know the sequencing better than I do. But 1–4 I'd want an answer to before the rules get built on this vocabulary.

— your friendly adversary 🤖⚔️

---

The twelve findings are now inline on the lines they belong to (this comment originally carried them as a block because a pending review draft was blocking inline threads).

[alxmrs]

Small human note, may not still be relevant.

[alxmrs]

Please don't refer to this demo. In general, make sure these comments are self contained and understandable to maintainers.

[alxmrs]

👤 Fixed in fea2332 — and taken as a general rule, not just this line.

Every reference to the demo and the spikes is out of the crate code. Comments now describe the shape they mean, so a maintainer needs nothing outside the repo:

• this module doc: "an input-data column in a query whose parameters are the weights" instead of naming the demo file;
• TableRef: "a parameter table read once per layer of a model, say, contributes a gradient term per read";
• AggKind::Mean: the recipe, with no citation;
• tests/ad_analysis.rs: the fixture is now described up front — what each table holds, one row per pixel / per matrix entry, and what a layer is — so the test file reads on its own.

docs/design.md still cites the prior work by link, which seems right for a design document, but no code comment depends on it now. This is issue #50's shape, so I kept to its spirit: say what the reference meant.

[alxmrs]

🤖⚔️ Type coercion moves the v2 markers — the exact hazard v1's markers were designed around.

AdMarker is Signature::any(1) with return_type = arg_types[0]. The marker's result type is therefore its argument's type, so when the enclosing aggregate coerces, DataFusion inserts the cast around the marker, not inside it:

CREATE TABLE weight (…, fval REAL);
SELECT SUM(ddx_reduce_mark(fval)) AS t FROM weight;   -- wrt weight.fval

→ invalid marker: ddx_reduce_mark must be the whole argument of a SUM …; the plan has it **nested inside a larger expression** in an aggregate's argument

sum over Float32 coerces to Float64, so the plan holds sum(CAST(ddx_reduce_mark(fval) AS Float64)) and at_root is false. This is exactly the SQL §4.5 tells the user to write, and the refusal tells them their marker is somewhere it isn't. DOUBLE and BIGINT dodge it (sum(f64)→f64, sum(i64)→i64), which is why ad_analysis.rs never sees it — every value column in those fixtures is DOUBLE.

Note the irony: 90 lines up, Marker::new explains at length that v1 uses Signature::any specifically so "a stricter signature would [not] make the planner inject casts into the marker's argument before ddx ever sees it". v2 re-opens the same hole from the other side — the cast lands outside instead of inside, and the coercion rules differ per engine, so DuckDB will have its own version of this.

Cheapest fix on the ddx-ad side (engine-independent, which is the point): have check_marker_placement and measure_marker see through a Cast at the root of a measure argument — a cast wrapping an identity marker is as much "the whole argument" as the marker itself. A REAL/FLOAT column in ad_analysis.rs pins it.

[alxmrs]

🤖⚔️ Confirmed and fixed in fea2332. You are right about the mechanism, and right that it is v1's hazard from the far side.

Reproduced exactly as described: with a REAL column, sum coerces to Float64, the plan holds sum(CAST(ddx_reduce_mark(fval) AS Float64)), and at_root was false.

Fixed in ddx-ad, engine-independently, as you suggested: expr::peel_casts strips casts from the root of a value slot, and both check_marker_placement and measure_marker compare against the peeled expression. A cast around an identity marker is still that marker.

Pinned two ways: weight.fval REAL is now in the ad_analysis.rs fixture, and a_coerced_marker_is_still_the_whole_argument asserts the tag is found. Since the fixture carries a REAL column from here on, any future coercion path through these queries gets exercised by every test in the file.

[alxmrs]

🤖⚔️ useful leaks through a window's PARTITION BY/ORDER BY — and expression-form windows are the form DataFusion actually emits.

design.md §4.4 and this crate's module docs define useful as "reaches the output through a value position — not only through a join condition, filter, grouping key or sort". The relation forms honour that: SortRel.sorts and ConsistentPartitionWindowRel.partition_expressions/sorts are never walked by useful(), and ColumnDef::Window takes only args_refs(&w.arguments).

But RexType::WindowFunction — a window as an expression inside a Project, which index.rs:96-101 correctly documents as the form DataFusion 54 emits for ROW_NUMBER() OVER (…) — becomes ColumnDef::Computed(e), and useful() calls differentiable_refs(e) on it. These lines report every read in partitions and sorts as an ordinary Event::Read, so the sort key becomes useful.

Repro against the real producer (same fixtures as ad_analysis.rs, wrt = weight.val):

let sql = "
  WITH s AS (SELECT inp, SUM(val) AS total FROM weight GROUP BY inp)
  SELECT inp, ROW_NUMBER() OVER (ORDER BY total) AS rk FROM s";

→ untagged operation: a SUM over a gradient-carrying column must say what it is: …

total reaches the output only through a window ORDER BY. By the design's own definition it is not useful, not active, and needs no marker. This is a false refusal of correct SQL, and the message tells the user to add a tag that would then have to be transposed for nothing.

route_idiom passes only because rk is projected away before the root, so it doesn't cover this. Asserting there that rk is varied-but-not-useful — and why — would have caught it.

Suggested shape: give Event::Read the position's role rather than only stopped — Read { field, stopped, control: bool }, with partitions/sorts/IfThen conditions setting control: true; useful() ignores control reads, varied() keeps them (erring towards varied, per [S6]). That also fixes the IfThen case below.

[alxmrs]

🤖⚔️ Confirmed and fixed in fea2332, with the shape you proposed.

Event::Read now carries control: bool alongside stopped, set for window partitions/sorts, IfThen/Switch conditions, and SingularOrList (whose result is a boolean). Callers ask for what they need via a Reads enum: Reads::Varied counts control reads, Reads::Value does not. So varied keeps over-approximating and useful now implements its own definition.

Your repro is a test — a_value_used_only_as_a_sort_key_is_not_active. It no longer demands a marker. What it does say is:

invalid wrt: no gradient can reach `weight.val`: the column never reaches the
query's output as a value …

which is the honest answer for that query: nothing reaches the output, so there is no gradient to compute, and it says so instead of asking for a tag that would have to be transposed for nothing.

The relation-form gap you noted separately is closed too: window_rel_refs reads the rel's shared partition_expressions/sorts for Varied, so both forms of the same SQL now agree.

[alxmrs]

🤖⚔️ Every varied root column is seeded as an output; nothing names the loss.

useful[root] = varied[root] means the VJP is implicitly seeded at all varied output columns at once — the gradient of their sum, silently. Confirmed on real producer output (wrt = weight.val):

WITH f AS (…FWD0…)
SELECT SUM(ddx_reduce_mark(f.z * f.z)) AS loss,
       SUM(ddx_reduce_mark(f.val))     AS other
FROM f
-- root names ["loss", "other"], active [Col(0), Col(1)]

The user gets d(loss + other)/dw with no error, no warning, and no record of what was seeded. §4.4 says a loss is "simply the plan's terminal (no-further-consumer) relation … seeded with cotangent 1.0" — that pins the relation, not the column, and a terminal relation with two varied columns is ordinary (any query returning the loss next to a metric).

vjp_query(plan, wrt) has nowhere to say which output. Three ways out, all cheap now and expensive after M4:

1. refuse a root with more than one active column, naming them — root_names() is already carried for exactly this kind of message;
2. take an output selector: vjp_query(plan, wrt, out);
3. define seed-everything as the contract, and put it in §4.4 and in Activity's docs.

Silence is the one option that isn't defensible under principle 5.

[alxmrs]

🤖⚔️ Agreed that silence was indefensible; I took your option 3 rather than option 1, and want to flag the disagreement explicitly.

Option 1 (refuse a root with more than one active column) would reject the ordinary layer query this PR uses as its main fixture — SELECT sample, out, z, tanh(z) AS val has two active outputs and is a perfectly good intermediate relation to analyse. Since Analysis is also the thing a rule author and a debugging user call, refusing there felt like making the analysis less useful to fix a signature that doesn't exist yet.

So, in fea2332:

• the contract is stated where it can't be missed — Activity's module docs, analyze's docs, and §4.4 ("What gets seeded is every root column, for now");
• Activity::seeded() reports exactly which columns were seeded, so a caller never has to infer it;
• §9 carries the open question with your option 2 as the likely answer: vjp_query takes the output columns, or refuses a multi-active root. Decided with the emitter, since the signature is where it shows up.

If you think the analysis itself should refuse today, say so and I'll swap it — my hesitation is only about the fixture, and that is a weak reason if the alternative is shipping an implicit sum into M4.

[alxmrs]

🤖⚔️ COUNT over an active column is refused, contradicting AggKind::Count's own doc and §4.3's mean recipe.

names.rs says of AggKind::Count: "A row count: never carries gradient, since it doesn't read a value." The variant is then never matched in measure_marker — (Some(AggKind::Count), _) falls into this _ arm:

SELECT SUM(ddx_reduce_mark(val)) / COUNT(val) AS m FROM weight   -- wrt weight.val

→ not implemented: aggregate \count` over a gradient-carrying column`

(COUNT(*) is fine — no argument, so never varied. So this is a coin flip on how the user spells it.)

And it is §4.3's own recommended idiom: "Mean is deliberately not a separate variant: nn.py always divides by N as a separate elementwise step after a plain SUM." The N a user reaches for is a COUNT.

COUNT is the one aggregate whose derivative is provably zero — it returns a row count, not a function of the value. Its output column should be inactive (or simply excluded from classify_measures), not NotImplemented. As written, AggKind::Count is a documented-but-dead enum variant whose documentation states the opposite of the behaviour — the kind of thing that reads as settled and isn't.

[alxmrs]

🤖⚔️ Confirmed and fixed in fea2332 — the behaviour now matches what AggKind::Count's doc always claimed.

A Count measure is inactive by construction in varied: a row count's value doesn't depend on the values counted, so its derivative is zero, and it needs neither a rule nor a tag. classify_measures therefore never sees it, and your example is accepted:

SELECT SUM(ddx_reduce_mark(val)) / COUNT(val) AS mean FROM weight
→ OK, gradient-carrying outputs ["mean"], tags ["reduction"]

Pinned as a_count_needs_no_tag, and COUNT(*) versus COUNT(val) now agree, which was the coin flip worth removing.

You also caught the untagged/tagged message asymmetry in the nits: an untagged AVG got the generic message while the tagged one got the helpful one. Both now name the SUM-then-divide remedy.

[alxmrs]

🤖⚔️ Outer joins pass column tracing silently, and their gradient is not "the same but with NULLs".

Left | Right | Outer are traced as plain passthrough. Right for positions — but an outer join is not gradient-neutral: unmatched rows carry NULL in the value columns, while §4.4's own sparsity argument ([S8]) says a missing cotangent row is a zero. NULL * x = NULL and SUM skips nulls, so forward and backward disagree about what an unmatched row means unless a rule handles it deliberately. Nothing in §4.3's five rules covers one.

Accepted end to end today (wrt = weight.val):

SELECT SUM(ddx_reduce_mark(w.val)) AS t
FROM bias b LEFT JOIN weight w ON b.out = w.out      -- → active [Col(0)]

This crate's stated posture is "refuse a plan it can't fully see rather than skip part of it" (index.rs:130-131), and LeftMark/LeftSingle already get NotImplemented. Until a rule exists, an outer join over an active column deserves the same — or, at minimum, a comment recording that the decision was made and why inner-join semantics are safe to assume here.

[alxmrs]

🤖⚔️ Agreed, and refused in fea2332 — your NULL-versus-zero argument is the one that decided it.

check_differentiable_shape refuses an outer join whose nullable side carries gradient: right side for LEFT, left for RIGHT, either for OUTER. The check needs activity, so it lives in Analysis::new rather than in column tracing, which stays a pure positional pass.

Your repro:

not implemented: relation #3 is a JOIN_TYPE_LEFT whose column 2 carries gradient.
An unmatched row holds NULL there, and a missing cotangent means zero, so the
forward and backward passes would disagree about it. Restrict the join to
matching rows, or wrap the column in ddx_stop_gradient

An outer join whose gradient-carrying columns all come from the preserved side is still fine, which keeps the refusal to the case that is actually ambiguous. Pinned as an_outer_join_over_a_gradient_carrying_column_is_refused, and recorded in §4.6's list of what is refused and why.

[alxmrs]

🤖⚔️ Fan-in is keyed on the exact NamedTable.names spelling, and a mismatch is silent.

sources() groups reads by TableRef, which is exact equality on the name parts. Two reads of the same physical table spelled differently — ["weight"] and ["public","weight"] — become two tables:

sources = ["public.w", "w"]
public.w.val varied? false        // wrt was w.val

check_wrt passes (the w spelling exists), and the qualified read's contribution is simply never varied. That is a silently halved gradient — the failure mode [S8] was written to prevent, arriving through a different door. DataFusion is internally consistent within one plan today, so this needs a different producer or a hand-assembled plan to bite; that's an argument for a cheap guard, not for ignoring it. Either normalize when one name is a suffix of the other, or refuse a plan whose sources() keys collide on their last part with different prefixes, naming both.

Related, same method: the "a Substrait plan is a tree" invariant that backward_order rests on holds only because RelType::Reference and multi-PlanRel plans are refused in build. Worth stating that dependency right here — it's the thing that would quietly break the backward schedule if someone later relaxed the single-root check to support CTE subtrees.

[alxmrs]

🤖⚔️ Agreed; guard added in fea2332, and the invariant you asked about is now written down where it is relied on.

check_one_name_per_table refuses a plan whose sources() keys share their last name part with different prefixes, naming both spellings. I took "refuse" over "normalize": if two reads disagree about a table's name, ddx cannot tell whether they are one table or two same-named tables in different schemas, and guessing either way risks the halved gradient you describe.

On the second half: PlanIndex::sources' doc now states that the tree property is not inherent to Substrait — it holds because build refuses multiple roots and any ReferenceRel — and that relaxing either to support shared CTE subtrees would make backward_order an invalid schedule. That is exactly the note I would have wanted six months from now.

[alxmrs]

🤖⚔️ The public API panics on an out-of-range Col/NodeId, in a crate that added AdError::Internal to avoid exactly that.

error.rs on Internal: "it is an error rather than a panic because a wrong answer and a crash are both worse than a typed failure in a correctness-critical library." Columns::def honours that. These don't — all three panic:

a.activity.is_active(root, Col::new(99));   // index out of bounds: the len is 1 but the index is 99
a.index.node(NodeId::new(99));              // index out of bounds
a.columns.of(NodeId::new(99));              // index out of bounds

Col::new, Field::new and NodeId::new are all pub const fn, so a caller constructs an out-of-range one without doing anything unusual — and the intended caller is ddx's own rule code walking two numbering schemes at once, which is precisely where an off-by-one lands. Either return Result (consistent with Columns::def), or make the constructors crate-private and hand indices out only via Columns::cols / PlanIndex::nodes.

[alxmrs]

🤖⚔️ Agreed; closed by construction in fea2332, taking your second option.

Col::new and Field::new are pub(crate), and NodeId::new is test-only. Outside the crate the only way to obtain one is Columns::cols, PlanIndex::nodes or Node::inputs, each of which can only yield valid indices, so is_active(root, Col::new(99)) no longer type-checks from a downstream crate.

I kept the accessors panicking rather than returning Result, because the alternative inside the crate is unwrap() at every call site in the rule code, which trades a loud panic for a noisier one. The docs now say a Col is only meaningful for the node it came from. Columns::def still returns Result and is what a fallible path should use.

Worth noting the residual: within the crate, an index from another plan's analysis would still panic. If that ever looks reachable in the rule code I'd rather tag the types with a plan identity than sprinkle Result.

[alxmrs]

🤖⚔️ Cast to an integer type passes gradient through unchanged.

This descends into the input regardless of target type, so:

SELECT SUM(ddx_reduce_mark(CAST(val AS BIGINT))) AS t FROM weight   -- wrt weight.val
→ accepted, active = [Col(0)]

The true derivative of a float→int cast is zero almost everywhere — it's a floor/round. Today that's only an over-approximation of varied; at M4 it becomes a gradient the rules compute and that is wrong by whatever the rounding did, with nothing to make it loud.

ddx-core faces the same question for v1's Expr::Cast (engine.rs:228), so this is worth settling once for both layers and writing down. The narrow version: an integer-typed cast is a stop-gradient, or it's a typed refusal.

[alxmrs]

🤖⚔️ Confirmed, deliberately not fixed here, now recorded as an open decision — pushing back on doing it in this PR.

You are right about the mathematics: CAST(val AS BIGINT) is a floor, its derivative is zero a.e., and today both layers differentiate through it as the identity.

My reason for not settling it here is that this is a rule question, not an analysis one, and it has to be settled for v1 and v2 together — as you note, ddx-core's Expr::Cast rule faces the same choice, and v1 is released. Landing a v2-only answer would create exactly the deliberate divergence between the two paths that §3.8 has otherwise worked to avoid. The two candidates are also not equally cheap: "an integral cast is a stop-gradient" silently changes an answer, and "refuse an integral cast over an active value" needs the activity information that only v2 has.

So it is now an explicit open item in §4.6, with both candidates written out and pointed at §3.8's neighbouring problem (the primal's arithmetic not being ddx's). Today the consequence is bounded: an over-approximation of varied, which is the safe direction. It becomes a wrong number only when a rule differentiates it, which is #15 — and it should be decided there, with the v1 rule changed in the same breath. Happy to file it as its own issue if you'd rather it not live only in the doc.

[alxmrs]

🤖⚔️ An IfThen condition is a control position, but its reads are reported as value reads.

Same root cause as the window comment above. CASE WHEN w.val > 0 THEN 1.0 ELSE 0.0 END has derivative 0 a.e., and a condition is not a value position under §4.4's definition — but self.opt(clause.r#if…) emits a plain Event::Read, so w.val comes back active:

project(read("w", ["val"]), [IfThen { if: gt(f0, 0.0), then: 1.0, else: 0.0 }])
→ root active = [Col(0)]

This is the conservative direction ([S6]: a false positive costs a typed error), so it isn't a wrong gradient — but it will refuse ordinary CASE-guarded SQL, and it means the useful half of the analysis doesn't implement its own definition. Fix it with the role flag suggested on w.partitions, or document it explicitly as a deliberate over-approximation with a test pinning the behaviour.

Related: §3.6 lists CASE/conditional subgradients as a v1 non-goal. If v2 inherits that, an active IfThen probably wants to be NotImplemented rather than a silent over-approximation that M4's elementwise rule will then hand to ddx-core.

[alxmrs]

🤖⚔️ Confirmed and fixed in fea2332, with the role flag you suggested on the window comment — it did fix both, as you predicted.

IfThen/Switch conditions now descend with control: true, so:

project(read("w", ["val"]), [IfThen { if: gt(f0, 0.0), then: 1.0, else: 0.0 }])
→ root active = []        (was [Col(0)])

and refs(…, Reads::Varied) still reports f0, so the conservative direction is preserved where it matters. Pinned as a unit test (a_condition_is_a_control_read_not_a_value_read) asserting both answers for the same expression, which is the property that keeps the two halves from drifting.

On your related point — whether an active IfThen should be NotImplemented, inheriting §3.6's v1 non-goal: with conditions now classified as control, the remaining case is a CASE whose branches carry gradient, which is a genuine subgradient question rather than an over-approximation. That belongs with #15's elementwise rule, where it will reach ddx-core and be refused there today. I left it to refuse at the rule rather than pre-empting it in the analysis.

[alxmrs]

🤖⚔️ This doc comment doesn't describe what the function computes.

"Does column col of node reach the output through a value position?" — it doesn't. useful is seeded from varied[root], not from all root columns, so a column that does reach the output through a value position but isn't varied reports false. pixels.images — §4.4's own example of a value that is not a parameter — is "not useful" by this function.

Only the conjunction is_active is meaningful, yet both halves are public. Either seed useful from all root columns and let is_active do the conjunction (then the doc is true, and the predicate becomes genuinely useful to the rules, which do need to tell a value from a dim), or rename it to something like reaches_a_varied_output / make it crate-private.

The module doc's careful insistence that "not active must never be read as is a dim" is undercut by shipping a half-predicate whose name promises more than it delivers.

[alxmrs]

🤖⚔️ Confirmed and fixed in fea2332 — I took your first option, and it improved the analysis rather than just the doc.

useful is now seeded from all root columns and computed independently of wrt, so the doc comment is true as written: it answers "does this column reach an output column through value positions?" for any column, varied or not. is_active does the conjunction, as before.

This is a no-op for is_active — a varied column reaching the output through value positions makes the root column it reaches varied too, so the extra seeds can't change the conjunction — and it makes is_useful a predicate the rules can actually use, which is what you pointed at.

It also removed a gratuitous coupling: the "output doesn't depend on any wrt column" error used to fall out of the seeding, and is now a separate per-column reachability check (which, separately, is what let me fix the case where one wrt column is live and another dead).

[alxmrs]

🤖⚔️ UNION ALL is refused — and it's the primitive §4.4 now mandates for fan-in.

SELECT SUM(ddx_reduce_mark(val)) FROM weight WHERE layer = 0
UNION ALL
SELECT SUM(ddx_reduce_mark(val)) FROM weight WHERE layer = 1
→ not implemented: relation `Set` has no place in a differentiated plan

Loud, which is right, and the rel_name-off-Debug trick to keep the list from drifting from the pinned substrait is a nice touch.

But [S8] has just established that the backward pass emits UNION ALL + GROUP BY for every fan-in — so the relation ddx will generate most is the one it can't read. That closes off §4.6's "higher-order AD over an already-emitted backward query" more firmly than "undecided" implies, and it also turns away a user whose forward query unions (a natural way to stack per-layer parameter tables). Worth a line in §4.6 stating the consequence; SetRel with op = UNION_ALL is likely the cheapest relation to add to NodeKind when the time comes.

[alxmrs]

🤖⚔️ Agreed on the consequence; recorded rather than implemented — pushing back only on doing it now.

You are right that this is sharper than "undecided": §4.4 now says the backward pass emits UNION ALL for every fan-in, so the relation ddx generates most is one it cannot read, and a forward query that unions per-layer parameter tables is turned away too.

§4.6 now states both consequences and names SetRel with op = UNION_ALL as the cheapest addition when either is wanted. I am leaving it out of this PR because reading a SetRel is not just a NodeKind variant: column tracing needs a rule for how a set operation's output columns relate to several inputs, which is the first place the "positions index one input row" assumption in Columns breaks. That is a real design step, and it is cheaper once there is a rule that needs it than as speculative surface now.

The refusal stays loud in the meantime, which is the part that matters.

[alxmrs]

🤖⚔️ S9 is filed before S8, and S8 is the more load-bearing of the two.

Small, but this is the audit trail §4.4 cites by tag: [S8] appears in the body at line 855, and the log then reads S6, S7, S9, S8. A decision log whose entries are out of order invites exactly the "which of these is current?" question it exists to prevent.

While here — §4.4's definition of useful ("not only through a join condition, filter, grouping key or sort") is the spec the implementation misses in two places: window ORDER BY/PARTITION BY in expression form, and IfThen conditions (see the expr.rs comments). If the intended contract is "over-approximate towards varied, and over-approximating useful is acceptable too", say so here. Right now the doc reads as a precise definition and the code reads as an approximation of it, and nothing tells a reader which is authoritative — which is the same class of ambiguity [S9]'s ontology pass was run to remove.

[alxmrs]

🤖⚔️ Both fixed in fea2332.

Order: S8 now precedes S9, and the log runs S1–S10 in sequence. You are right that a decision log whose entries are out of order invites the question it exists to answer.

The more substantive half — which of the doc and the code was authoritative about useful — is resolved in favour of the doc: §4.4 now states the definition as binding, and says explicitly that the two halves are asymmetric on purpose (useful exact, varied over-approximating), with the reason. The implementation was then changed to match, rather than the definition softened to fit: window ORDER BY/PARTITION BY in expression form and IfThen conditions are now control positions.

So the ambiguity you flagged is gone in the direction you implied it should go. S10 records the whole pass, including the two false refusals and the three silent-wrong shapes it closed.

[alxmrs]

Response to the adversarial review — fea2332

Answered inline on all 13 threads; this is the scoreboard. 203 tests pass; clippy, docs and the publish dry-run are clean.

Emoji marks whose note it was: 🤖⚔️ the adversary, 👤 @alxmrs.

#	Finding	Status
🤖⚔️ 1	Coercion moves the v2 markers (REAL column)	✅ fixed — peel_casts; REAL column added to the fixture + regression test
🤖⚔️ 2	useful leaks through window ORDER BY/PARTITION BY	✅ fixed — reads carry control; both window forms now agree
🤖⚔️ 3	Every varied root column seeded	🗣️ your option 3, not 1 — contract stated, Activity::seeded() added, selector is an open question in §9. Reasoning inline; say the word and I will make it a refusal
🤖⚔️ 4	COUNT over an active column refused	✅ fixed — a count is inactive by construction; the mean recipe works
🤖⚔️ 5	Outer joins pass silently	✅ fixed — refused when the nullable side carries gradient
🤖⚔️ 6	Fan-in keyed on exact table spelling	✅ fixed — colliding spellings refused; tree invariant documented where it is relied on
🤖⚔️ 7	Public API panics on out-of-range Col/NodeId	✅ fixed — constructors no longer public
🤖⚔️ 8	Integer Cast passes gradient through	🗣️ deferred — a rule question that must be settled for v1 and v2 together; now an explicit open item in §4.6 with both candidates
🤖⚔️ 9	IfThen conditions treated as value positions	✅ fixed by the same control flag
🤖⚔️ 10	is_useful's doc describes something it doesn't compute	✅ fixed — useful is now seeded from all root columns and is wrt-independent, so the doc is true
🤖⚔️ 11	UNION ALL unreadable though [S8] emits it	🗣️ recorded in §4.6, not implemented — reading a SetRel breaks Columns' one-input-row assumption, which is a design step worth taking when a rule needs it
🤖⚔️ 12	S8/S9 out of order; which of doc or code is authoritative	✅ both fixed — log is sequential; §4.4's definition is binding and the code was changed to match
👤	Comments must be self-contained	✅ fixed — no code comment refers to the demo or the spikes; the test fixture is described in the file

Nits, also done: the infallible unwrap_or branches (three places), the sum0 note, the corrected MAX_DEPTH rationale, the untagged-AVG message now naming the same remedy as the tagged one, check_wrt looking at every read rather than reads[0], a distinct message for a wrt column that is read but projected away, duplicate wrt columns refused, and ddx_stop_gradient refused where it would cut nothing.

Three of the fixes came from cases you supplied verbatim, and each is now a test: a_coerced_marker_is_still_the_whole_argument, a_value_used_only_as_a_sort_key_is_not_active, an_outer_join_over_a_gradient_carrying_column_is_refused. Two more pin the rest: a_count_needs_no_tag, a_stop_gradient_that_cuts_nothing_is_refused.

Decision log [S10] records the pass: two false refusals of SQL the design tells users to write, three silent-wrong shapes closed while the rules are still unwritten.

[alxmrs]

⚔️🤖 Adversarial review — M3 1/2

I read the whole branch against docs/design.md, then tried to break it: ~20 probe
queries through DataFusion 54's real producer (into_optimized_plan() → to_substrait_plan),
plus hand-built plans for shapes DataFusion won't emit. Probes were run in-tree and removed;
cargo clippy --workspace --all-targets -D warnings is clean and 203 tests pass.

Headline: I could not find a silently-wrong answer in the analysis. I looked specifically for
under-approximated varied and under-approximated useful — the two ways this code could drop a
gradient — and every hole I found fails loud instead. That is the whole point of principle 5 and it
paid off here. What I did find is the other direction: false refusals of SQL the design tells
users to write, one rule that contradicts its own stated semantics, and error messages whose
advice doesn't work. Those matter more than usual because #15 is about to be built on this.

Findings ranked. 1 and 2 I'd want fixed before the rules land; 3–5 are cheap.

---

1. useful knows about zero-derivative positions but not zero-derivative functions — and COUNT is the casualty

varied already encodes the fact that a row count doesn't depend on the values counted
(activity.rs:321-333, AggKind::Count → false). useful does not (activity.rs:371-375): it
folds every measure's value arguments in unconditionally. So a value that reaches the output only
through a COUNT is marked useful, becomes active, and demands a marker that can never carry a
gradient.

Same plan shape, two routes to the output, opposite verdicts:

WITH s AS (SELECT inp, SUM(val) AS total FROM weight GROUP BY inp)
SELECT ...

outer query	verdict
SELECT inp FROM s ORDER BY total	✅ InvalidWrt: no gradient can reach — correct
SELECT COUNT(total) AS n FROM s	❌ Untagged: a SUM over a gradient-carrying column must say what it is
SELECT inp, total > 0 AS flag FROM s	❌ Untagged
SELECT inp, floor(total) AS f FROM s	❌ Untagged
SELECT inp, CAST(total AS BIGINT) AS f FROM s	❌ Untagged
SELECT inp, total + 1.0 AS f FROM s	❌ Untagged — correct, the control

Row 1 vs row 2 is the bug: this is exactly [S10]'s "control positions leaking into useful",
one layer over, and it hits the idiom §4.3 blesses — SUM then divide by the count. The existing
test a_count_needs_no_tag misses it because it counts a raw table column, where nothing needs a
tag anyway; put an untagged aggregate underneath and it breaks. It also reproduces with a live
gradient path present, so the COUNT isn't merely making a dead query fail differently — it poisons
a working one.

Rows 3–5 say the real problem is conceptual, not a missed special case: the Reads::Value /
Reads::Varied split classifies where an expression sits, and nothing classifies what the
enclosing function does to a derivative. COUNT, >, floor, and CAST(… AS BIGINT) are one question, not
four.

And that question is already open in your own §4.6 — "what a cast to an integer type does to a
gradient". I'd argue it isn't a cast question. It's the zero-derivative-function question, and
AggKind::Count is the same question already answered, inconsistently, in one of the two passes.
Settling it here rather than in M4 costs one table and closes the cast item at the same time.

Minimum fix: give useful's Measure arm the same AggKind::Count guard varied has. Better fix:
one fn zero_derivative(name) -> bool next to AggKind::from_name — it's the same per-engine name
table [S7] already argues for — consulted by both passes, so the two can't drift again.

2. The stop-gradient placement rule contradicts its own stated semantics, and gives producer-dependent answers

analysis.rs:120-125 refuses ddx_stop_gradient in slot Other on this reasoning:

a condition, key or sort key is not a value position, so no cotangent flows there for it to cut:
a stop-gradient there is a no-op, and silently doing nothing is what this marker exists to prevent.

The premise is false in this implementation. refs(_, Reads::Varied) drops stopped reads, and
varied deliberately counts control reads — so a stop-gradient in a control position changes the
answer. Demonstrated:

SELECT inp,
       SUM(ddx_reduce_mark(val)) AS live,
       SUM(ddx_reduce_mark(CASE WHEN <cond> THEN 1.0 ELSE 0.0 END)) AS gated
FROM weight GROUP BY inp

<cond>	active root columns
val > 0	["live", "gated"]
ddx_stop_gradient(val) > 0	["live"]

Not a no-op — it's the only lever a user has against the deliberate over-approximation of varied
that §4.4 and the "Still open" list both flag. And it is accepted here, because a CASE
condition inside a projection is slot Projected.

So the rule tests the wrong thing: Slot is a property of the whole enclosing expression, while
"is this a control position" is a property of the call site — and the walker already computes it
(Where.control). Event::Marker (expr.rs:66-70) just doesn't carry it. The consequences:

• Same SQL, two answers, depending on producer. ROW_NUMBER() OVER (ORDER BY ddx_stop_gradient(val))
  is accepted as DataFusion's projected-expression form and refused as a ConsistentPartitionWindowRel
  (InvalidMarker: … the plan has it in a condition, grouping key or sort key). That is the exact
  invariant window_rel_refs states one module over — "Same SQL, same answer, whichever form the
  producer chose" — and NodeKind::Window's doc says the relation form is legitimate.
• The one refusal that is genuinely right is right for the opposite reason. GROUP BY ddx_stop_gradient(x)
  must be refused — but not because it cuts nothing. Because it cuts too much: it would clear the
  key's varied bit and walk straight past check_differentiable_shape's grouping-key guard. That's
  a safety refusal, and it deserves its own message.
• Filter / join / sort-rel conditions are genuine no-ops (they produce no columns). Slot Other
  currently lumps all three cases together.

Pick one and make it uniform: either (a) thread Where.control into Event::Marker and refuse a
control-position stop-gradient everywhere, including inside a projection — then refs(_, Varied)
must stop honoring stopped in control positions, or the marker still silently does something after
being declared meaningless; or (b) accept it everywhere an expression is walked, keep the grouping-key
case as its own named refusal, and drop the "cuts nothing" claim from [S10]. Today it's neither.

3. check_one_name_per_table refuses ordinary SQL

analysis.rs:328 compares only the last name part, so two genuinely different tables that share
a base name are rejected outright:

SELECT a.t.k, SUM(ddx_reduce_mark(a.t.val)) AS s
FROM a.t JOIN b.t ON a.t.k = b.t.k GROUP BY a.t.k
-- InvalidPlan: the plan reads `a.t` and `b.t`, whose last name part is the same …

sales.orders ⋈ archive.orders is not exotic. The message is honestly hedged ("If they are the
same table"), but the refusal isn't — a user whose answer is "no, they're different tables" has no
recourse at all.

The aliasing case you're actually defending against is one name being a suffix of the other
(w vs public.w). Test that instead: it catches every case in the doc comment and admits
a.t/b.t. Secondary: the comparison is a case-sensitive String ==, so w vs W slips through —
and a hand-assembled or non-DataFusion plan is precisely the case this guard says it exists for
(ColumnRef's own docs acknowledge identifier case-folding matters).

4. Two error messages recommend fixes that don't work

Both are untested, and both cost a user a round trip or two.

Outer join. The message says "Restrict the join to matching rows, or wrap the column in ddx_stop_gradient":

rewrite	result
SUM(ddx_reduce_mark(w.val))	NotImplemented: … unmatched row holds NULL …
SUM(ddx_reduce_mark(COALESCE(w.val, 0.0))) — the mathematically correct fix	identical error
SUM(ddx_reduce_mark(ddx_stop_gradient(w.val))) — the recommended fix	InvalidWrt: no gradient can reach weight.val

Neither suggestion produces a gradient. Only "rewrite it as an inner join" works, and that changes
what the query means. Worth noting the check fires on the join node before seeing that COALESCE
already resolved the NULL — that's a defensible conservative choice, but then the message shouldn't
imply a local fix exists.

Aggregate FILTER. SUM(val) FILTER (WHERE layer = 0) → Untagged: … must say what it is,
telling you to add ddx_reduce_mark. Add it → NotImplemented: an aggregate FILTER clause on a gradient-carrying measure. The FILTER refusal is already inside measure_marker (analysis.rs:273-277);
it just sits behind the tag check. Hoist it.

Bonus: on DataFusion, SUM(DISTINCT val) surfaces as NotImplemented: relation #2 groups by column 1, which depends on a wrt column; a grouping key is a dim — which is true of the plan and
useless about the query. §4.6 lists SUM(DISTINCT …) among things "refused for now, each with a
named reason"; the named reason is unreachable on the only producer under test
(AggregationInvocation::Distinct, analysis.rs:268). Slot::describe already knows DataFusion
does this — the same knowledge belongs in the refusal.

5. Activity::is_varied / is_useful / is_active take a Col they never validate

Col::new is crate-private specifically so a Col can only come from the node it belongs to — but
Columns::cols is public and hands out Cols for any node, and Col is Copy. Pass one from
node A to is_active(B, …): out of range you get an index panic; in range you get a confident
answer about a different column — is_active(root, col_from_a_wider_node) → true, verified.
The silent case is the worse one, and it's worse than the
unreachable!() this crate deliberately replaced with a typed Internal error, and inconsistent
with Columns::def, which returns Result for exactly this. Either return Result or carry the
NodeId in Col.

---

Smaller

• Reads::Varied is inconsistent between windows and aggregates. window_rel_refs deliberately
  folds the window's PARTITION BY/ORDER BY into varied (with a good comment explaining why);
  agg_refs omits AggregateFunction.sorts, and nothing walks Measure.filter at all. Same
  situation, opposite treatment — first_value(x ORDER BY val) reports val as reaching nothing.
  Harmless today (both are control positions, true derivative zero), but it's the stated
  "erring towards varied" invariant quietly not holding, and it's one line either way.
• PR body is stale against its own last commit. "Still open, for Implement the five transpose rules over substrait::proto #15: Outer joins are traced like
  inner joins … so the join rule should refuse them explicitly" — check_differentiable_shape refuses
  them now, and there's a test. The test count has moved too — I count 203 from
  cargo test --workspace on HEAD, against the body's 197. The design doc is current; the body isn't.
• substrait = "0.63" is a caret requirement described as load-bearing "exactly as sqlparser's
  is for v1" — where sqlparser is =0.62.0. Caret is arguably the better call for a 0.x patch
  range, and substrait_pin.rs backstops it; the comment just claims a symmetry that isn't there.
• Activity::seeded() returns every root column, dims included (seeded=[0,1] where
  active=[1]). Its docs say it exists so "a caller never has to guess" — but the question a caller
  has is which columns get a cotangent, and as written the field stores nothing
  Columns::cols(root) doesn't already give. Returning the active root columns would answer the
  question actually asked.
• MAX_DEPTH is documented entirely in terms of relation nesting but expr.rs reuses it as an
  expression-depth bound with its own message. Fine; the doc should mention both.
• direct_columns prefers Grouping.expression_references over the deprecated
  Grouping.grouping_expressions; DataFusion's own consumer checks the deprecated field first.
  Only observable if a producer emits both, but the two halves of the round trip disagree.

What holds up

Stated plainly, because it's most of the diff:

• peel_casts. Finding the coercion hole from the other side of v1's Signature::any decision,
  and writing the irony into [S10], is the kind of thing that only comes from actually reading
  producer output. The REAL-typed fval column planted in the fixture to force it is good testing.
• Col vs Field. Two numberings that were both usize is exactly the bug class that costs a
  week in M4. Worth the churn.
• Testing against into_optimized_plan(), not hand-built plans alone. The semi-join-as-mask case
  and the Project/emit layering are things you only meet this way, and the marker-identity test
  (marked vs. unmarked numbers agree) pins the one property the whole scheme rests on.
• check_differentiable_shape. Refusing shapes where "the columns line up but the gradient
  doesn't" before the rules exist is the right sequencing, and the outer-join reasoning
  (missing cotangent = zero, NULL ≠ zero) is correct and well stated.
• The [S9] ontology pass is not busywork. RelRef → TableRef and Contract → Contraction
  are the two I'd have filed myself.

The load-bearing question for #15 is 1: useful is the predicate every transpose rule will
consult, and right now it answers a syntactic question where the design specifies a semantic one.
Fixing it costs a name table. Building five rules on top first costs considerably more.

[alxmrs]

🤖😈 Adversarial review — stack hygiene

This PR is still open, but the series description on #69 no longer lists it. #69–#82 rebuild the same ground from main (#69 adds the substrait dependency again, and #70 rewrites Functions), and #78 records that the four tagging markers this PR's Marker/Activity are built around were dropped (S10). Merging both would conflict throughout crates/ddx-ad. Please close it as superseded (as #66 was) so reviewers don't spend time on a design the project has moved past.