Conversation
…pd_non_anticipativity) Two always-on derived frames in _derived_branch.py, consumed by nothing (recourse plan §6b Slice A; design specs/sliceA_lineage_frames_design.md r3): - dd_same_scenario (d, d_other): symmetric/reflexive pairs sharing a scenario leaf-path; domain = period_in_use only; all-pairs degenerate content for deterministic solves; leaf rule with the F6 authority convention (branch-local member displaces the anchor). - pd_non_anticipativity (d, b): period-level invest-NA pairs (strictly-pre-reveal anchor ties), parameterized on an optional branch_start frame (Slice E hook); provably empty for every currently-constructible solve under the min-anchor fallback. Wiring: unconditional assignment in apply_branch_cluster (step 2b), guarded-ctx input acquisition per the dtttdt_from_source null-guard precedent, ValueError failure semantics on inconsistent per-solve artefacts, Utf8-construct -> non-strict cast_dim -> live null assertion -> deterministic Utf8-key sort. FlexData registration (two typed fields, None only pre-cascade), d_other axis synonym. No consumers, no schema change, no migration, no autoscale entries (nothing enters the LP), no dump-side change (_copy_meta already glob-copies the rebuild inputs). Tests: T1-T12 per design §4/§5 (17 tests) — five exact shape tables, both mid-horizon shapes incl. the branch_start-driven [(p1, p1_low)] pair, symmetry property, metadata exclusion, failure semantics, _drop_master_rows pass-through, integration + enum hygiene on the stoch_two_period fixture, and the first stochastic dump/reload round-trip (pins the _copy_meta glob). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Slice B of the recourse-invest plan (specs/sliceB_walker_lineage_design.md). Add a keyword-only lineage=pl.DataFrame|None=None parameter to the shared period_walk_iterator, applied PRE-aggregation under the Slice A semi-join contract (remove (e,d,d_all) iff both ends in period_in_use and (d,d_all) not in lineage; history-anchored rows pass unconditionally). - _derived_walks: _apply_lineage_filter (anchor_col/dall_col params, fixed-direction lineage-side non-strict cast, mandatory .unique(), marker left-join) + _assert_lineage_castable null guard; zero-cost None path behind a single if. - Thread lineage=None through inventory #1-17 (cohort/variant/from_source chain in _derived_existing, _derived_npv, _derived_params) and apply the shared helper in the one non-walker pair-former #18 edd_divest_active_from_source (anchor_col="d_divest", dall_col="d"). - _derived_branch: check_recourse_npv_preconditions + assert_recourse_npv_preconditions — the three named NPV proof obligations (leaf-weight constancy, cohort sum-to-1, byte-equal fan-member year rows), shipped as a pure checker with the F4 soundness-scope docstring; added to __all__. Inert: every production call passes lineage=None; no flag, no consumer activation, no schema change, no autoscale entries, no LP change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Slice B test suite (specs/sliceB_walker_lineage_design.md §9). Capability harness (solver-free, hand-built frames + stub source): - W1 set-shape (history passes, cross-leaf both-ends-PIU pairs removed) - W2 factor-side PRE-aggregation proof (sums 7.0/3.0/4.0 vs unfiltered 7.0 — a post-aggregation filter cannot produce 3.0/4.0) - W3 lineage=None 12-row unfiltered pin - W4/W4b empty-frame contract (set: 4 history rows; factor: single (h2020,7.0) row, anchors absent not zeroed) - W5a duplicate-pair .unique() guard; W5b uncastable-token ValueError; W5c Enum x Enum missing-token raises / all-present identical Stochastic fixture pins (solver-required fixture, solver-free asserts): - W6 edd_invest_set today-bug witness ((wind,p2040,p2040_low) absent = erratum E1 first-stage-visibility break) - W7 edd/divest non-vacuity + repeat-determinism; NPV variant vacuity pins + invest-method overlay leg - P1/P2 obligation checker clean on stoch/deterministic workdirs - P3a-d hand-seeded obligation violations (i)/(ii)/(iii) - P4 walker year-source blindness diagnostic (erratum E1 witness pair) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The Slice B NPV obligation checker's year-row reader (`_read_solve_data_csv`) fell back to a bare `pl.read_csv(...)` on the workdir CSV when the Provider didn't carry the frame. That disk arm tripped the post-Step-2 cascade invariant (test_meta_provider_invariants::test_no_disk_csv_reads_in_cascade, Rule 1) — the seed-funnel bug class the invariant guards. Make the reader Provider-only (`_provider_get` -> `provider.get`), matching every other reader in `_derived_branch.py`. The checker now does no disk I/O; the caller owns placing the true emitted year CSVs on the Provider (tests seed via seed_provider_from_dir / _mk_provider; the Slice D activation site inherits the same solve_data frames the rest of the cascade already consumes). Obligation (iii) semantics preserved. SOUNDNESS SCOPE docstring + design (local) sec 6.2 updated with the deviation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Add the opt-in solve.stochastic_invest_method enum (value list
stochastic_invest_methods: none/recourse, default none) via migration
v70. 'none' is byte-identical to prior behaviour; 'recourse' is
reserved for Slice D and hard-rejected by the Slice C guards.
- FLEXTOOL_DB_VERSION 69 -> 70; _migrate_v70_stochastic_invest_method
mirrors the v64 solve.scaling exemplar (value list + definition +
guarded solve_advanced grouping).
- PARAMETER_TYPES row ('stochastic_invest_method','solve') DIMENSIONLESS.
- _solve_config read/store + stochastic_invest_method_for resolver
(absent -> 'none').
- Regenerated spinedb_schema.json master template + the two managed
canonical DBs (templates_examples, howto_stochastics).
- export_settings.yaml split_params: surface the new solve knob.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Three flag-independent / flag-keyed validation guards that keep the recourse opt-in inert until Slice D lands. - Guard 1 (flag hard-reject): stochastic_invest_method != none raises FlexToolConfigError at the top of _orchestration.run() (authoritative, covers rolling/contained solves via the roll-suffix strip) plus a fail-fast pass in _validate_model_solve over the top-level solves. - Guard 2 (Benders x genuine stochastics): _is_genuinely_stochastic raises in _run_benders_solve (primary) and at the top of solve_benders (defensive). Detector = period_branch_full d!=b b-tokens intersected with period_in_use_set, so the rolling realized-only bookkeeping shape does NOT false-positive. - Guard 3 (handoff synthetic-name assertion): build_handoff_from_solution rejects any realized_invest / committed period that is a synthetic branch fan member; None-degrades when flex_data is unavailable. New shared cycle-free module _stochastic_detect.py holds the single token-extraction idiom (synthetic_branch_tokens) used by Guards 2 and 3 and the PIU-intersected is_genuinely_stochastic used by Guard 2. Both cast d/b to Utf8 before comparing — at runtime period_branch_full is Enum-typed with disjoint d/b category sets, and comparing two enums raises in polars (would otherwise crash Guard 3 on every stochastic handoff); null b (deterministic shape) is dropped. Empirically verified on stoch_two_period: genuine stochastic -> True (the _rlz metadata members are excluded by the PIU intersection, the _low branches survive); deterministic + rolling -> False. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
- test_v70_stochastic_invest_method.py (M1-M3): migration round-trip
(definition present, default none, bound to stochastic_invest_methods,
grouped solve_advanced; value list {none,recourse}); pre-v70 -> v70
reach + idempotent re-run; master-template --verify up to date.
- test_recourse_guards.py (G1, G2a-c, G3): resolver normalisation +
_validate_model_solve reject/pass + rolling-suffix-strip composition
(G1); PIU-intersected detector on genuine/deterministic/self-row/
rolling-realized-only shapes incl. the Enum disjoint-category
regression pin and solve_benders defensive raise (G2); solver-based
handoff clean-path / config-level synthetic raise / None-degrade (G3).
Detector shapes mirror the empirically verified stoch_two_period
runtime frames (Enum-typed d/b, PIU excludes _rlz metadata members).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Emit solve_data/stochastic_invest_method.csv (resolved per-solve mode) next to the realized-invest emit; _recourse_invest_active reader (provider-first, workdir fallback, absent -> False); FlexData.recourse_invest bool stamped in apply_derived_c; _has_branch_invest_axis capability conjunct reusing is_genuinely_stochastic. Dump-registry + accumulator-manifest lines so a dump/reload round-trip preserves the flag. Flag defaults none -> no behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Define LineageFilterError(ValueError) in _solve_state; the two Slice B lineage guards (_assert_lineage_castable, assert_recourse_npv_preconditions) raise it. Narrow the four blanket except-Exception swallows in apply_derived_c / apply_synthetic_invest_sets (edd lookback / edd_invest / edd_divest_active / synthetic edd) to re-raise LineageFilterError so a lineage failure is loud instead of silently reverting to unfiltered first-stage sets. Raise path is unreachable while lineage is None (flag-off byte-parity). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
(A) _expand_invest_branch_periods — invest-side mirror of the dispatch
_expand_branch_periods; wraps the 5 axis-defining _solve_periods(invest_periods)
sites (ed_invest_set, ed_divest_set, ed_invest_forbidden, eager
ed_entity_annual_family, apply_npv) so period_invest gains {p_k, p_k_b}
under recourse. ctx/provider threaded through those readers + their
apply_derived_c / apply_synthetic callers. period_first + PIU-fallback
sites left unwrapped (design §3.1).
(B) _anchor_expand_explicit + module-scoped anchor-pairs holder set at each
Layer-4 boundary (apply_derived_c / apply_npv / apply_synthetic): explicit
per-period rows (invest_cost/discount/lifetime/fixed_cost Maps + per-period
caps) inherit their anchor's value onto branch periods, wired into the three
producers (_resolve_per_period_lf, _per_entity_period_cost,
_ed_explicit_period_param). Holder None flag-off/deterministic -> no-op ->
byte-parity.
Flag-off spot-suites (fanout, npv_cluster, lineage) green; new axis unit
tests cover expansion + anchor inheritance.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Thread provider through period_walk_iterator into _p_years_d_lf (year side) and _inflation_factors_lf -> _years_for_period_lf -> _years_for_period_from_source (factor side). Add a provider-gated canonical arm to _years_for_period_from_source reading the emitted p_years_represented.csv (which byte-copies fan-member year rows via derive_years_represented's branch loop). Under recourse the caller (D4 apply boundaries) forwards provider so branch periods resolve their anchor's year-from-start -> correct annuity windows; provider=None (flag-off) keeps the canonical arms dead -> today's fill_null behaviour (the W6 today-bug pin) -> byte-parity. Plumbing only this phase (no caller forwards provider yet); activation lands with lineage at D4. Flag-off npv_cluster/walk/fanout suites green; new canonical-arm unit tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Compute the scenario-lineage frame (flex_data.dd_same_scenario) at the three Layer-4 apply boundaries, gated on recourse AND a genuine branch invest axis (capability conjunct). Forward lineage=... into the edd builders threaded by Slice B: edd_invest_lookback_set, edd_invest_set_lf, edd_divest_active (#18), the synthetic edd, and apply_npv's four ed_*_from_source walks. Hoist assert_recourse_npv_preconditions above the edd try/except blocks so a violation raises LineageFilterError (loud, D1). alpha-1 walker Provider: a boundary-scoped module-global (_RECOURSE_WALK_PROVIDER) armed alongside the anchor-pairs scope; period_walk_iterator falls back to it so the canonical year/factor arms revive under the flag without threading provider through ~12 NPV/edd caller signatures. None flag-off -> arms dead -> W6 today-bug preserved. Activation is inert flag-off (lineage None) -> byte-parity; 103-test spot-suite (npv_cluster/fanout/lineage/axis/year) green. Flag-on edd cross-leaf integration lands at D5 (Guard 1 lift + fixture). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Objective: multiply ALL EIGHT invest/divest cost Params (process annu/lf/lfd/annd + node annu_n/lf_n/lfd_n/annd_n) by pd_branch_weight[d] via the _wt closure, gated on d.recourse_invest AND pd_branch_weight; replace the .mod "NOT weighted" comment with the recourse rationale. Flag-off skips the multiply (byte-parity). Guard 1 LIFTED for 'recourse': _stochastic_invest_raw_guard validates the RAW authored value (resolver collapses typos to 'none' so the guard must read the raw dict); both sites (run() + _validate_model_solve) switched; guard tests flipped per design §15.4 + NEW raw-typo rejection test. Activation fixes found by the gate (recorded in design implementation notes): (a) precondition checker needs the REAL workdir (Provider-only readers key paths off it — design §8 snippet's None arg made obligations fail spuriously); (b) lineage built ON-DEMAND per boundary via dd_same_scenario_df (flex_data.dd_same_scenario populates only at derived_g); (c) alpha-1 walker holders gain the workdir twin (_RECOURSE_WALK_WORKDIR) so the canonical p_years_d arm goes live; (d) _recourse_usable_ctx sanitizes the snapshot-reload SolveContext (all-null d_anchor artifact) with provider-arm fall-through. Gate fixture stoch_two_period_invest.json (+ builder, + conftest wiring): supply-side branch variation (deterministic demand 100/130 + wind upper_limit profile rlz .4/.4 low .2/.1 via group__unit -> peaker residual 60/90 / 80/120) because node-inflow branch resolution gates on group__node and a stoch-group node would fire NA net-charge pinning. D5 gate: objective 196875 EXACT; coefficients 525/262.5/1575/787.5; per-leaf invest 60/80/30/40; committed output realized-only; edd cross-leaf ABSENT (W6 re-pointed); pd_non_anticipativity empty; STRICT autoscale clean. Flag-off parity: 128 tests green (lineage/walk/fanout/ guards/npv_cluster + recourse unit suites). ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Per-scenario-path total-capacity caps so a total cap applies once per
scenario leaf under recourse, never cross-scenario double-counting
(design sliceD §7 (G) / §14 (L)).
- d_leaf (d, leaf) frame in _derived_branch.py: partitions period_in_use
into scenario leaves (real anchors -> "__realized", synthetic members
-> time-branch id); registered on FlexData, built in apply_branch_cluster.
- model.py _emit_entity_total_cap: the four maxInvest/maxDivest entity
total builders emit the legacy over=("d",) shape + name for a single
leaf (deterministic / flag-off, byte-identical) and …_path indexed by
(entity, leaf) for >1 leaf, masking the inner sum to each leaf.
- _cumulative_invest.py _emit_group_total_divest_cap: the divest-side
group total gains the same per-path treatment (max/<= cap only).
- autoscale registry: maxInvest/maxDivest_entity_total_path(_n) (POWER/
ENERGY) + maxDivestGroup_entity_total_path (group_capacity resolver);
test_registry_coverage dynamic-name pins.
- Fixture invest_max_total tightened 250 -> 150 (§15.1): per-leaf totals
90/120 non-binding, legacy all-d sum 210 would have bound — the
asymmetry proves per-path. Gate test asserts two leaf rows.
Gate: objective 196,875 exact; STRICT autoscale clean; flag-off parity
(derived_branch/period_walk lineage, continuation fanout, recourse
guards, deterministic invest cumulative/total goldens) green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Make the recourse handoff's drop of branch-named (non-realized-scenario) v_invest rows EXPLICIT (design sliceD §10.1 (H)). - build_handoff_from_solution: partition invest_by_ed on synthetic branch-token membership BEFORE the commit loop and debug-log the dropped mass; the commit gate previously dropped these by accident. The Slice C Guard 3 assertion stays as a belt-and-braces invariant check (no synthetic token may survive into committed rows), not the mechanism. Flag-off / deterministic: synthetic set empty → no-op (byte-parity). - Gate test: the pre-filter drops the low leaf's 80+40 = 120 MW, debug- logs it (periods p2035_low/p2040_low), and commits only the realized 60/30; Guard 3 stays inert. Guard 1 raw-value lift was already landed at D5. Flag-off parity (handoff cumulative carriers, continuation fanout, recourse guards) green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Outputs, cost reconciliation, and the cross-scenario-coupling fixture (design sliceD §11 (I) / §15.5 / §15.6). Outputs (§11.2): - read_highs_solution: under recourse AND output_horizon, union the horizon-expanded d_realize_dispatch_or_invest set (branch periods included) into the invest-axis canonical order, so non-realized branch v_invest rows surface. Gated on both flags → byte-parity for deterministic models and for recourse without horizon output. Committed path stays realized-only (§11.1). Costs (§11.3, choice I-a): - calc_costs probability-weights the invest/divest + lifetime-fixed cost rows by pd_branch_weight under recourse, so the cost breakdown reconciles to the LP objective's (expected) invest contribution rather than over-reporting the realized spend by 1/w_realized. read_parameters threads par.pd_branch_weight + par.recourse_invest. Gated → factor 1.0 off-flag (byte-parity; deterministic cost-aggregation goldens green). - test_npv_cluster skips the eager-vs-lazy NPV byte-equality under recourse (the eager reference is not lineage-aware, §11.4). Fixtures/tests: - variant A gains a recourse_horizon scenario (model.output_horizon=yes); gate test asserts branch rows 80@p2035_low / 40@p2040_low surface. - variant B (stoch_two_period_invest_na): extends the proven stochastics.json storage shape with a recourse_na scenario making the stochastic-group storage node investable. Witnesses: non_anticipativity_storage_use fires (>0 rows), branch v_state == the NA-tied realized value, and the node-side annu_n objective coefficient carries pd_branch_weight (the F3 node-side pin). - cost reconciliation e2e: reported realized invest cost 0.039375 M = 0.25·(60·2100 + 30·1050); solver objective 0.196875 M. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
- how_to.md: replace the "only one period investment stochastic models are supported" line with the per-scenario (wait-and-see) recourse recipe — what the objective means (expected value of per-scenario plans, EVPI-style; not a hedged plan), the coupling change vs the default, the α-1 cross-comparability caveat, and the Benders exclusion (design sliceD §16). - reference.md: document solve.stochastic_invest_method (none/recourse), the output_horizon interaction, and the Benders exclusion. - CHANGELOG: Unreleased draft entry (public wording, no internal model names; schema change note). No version bump — release is the owner's decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The end-to-end cost-reconciliation test invokes a real solve; add the ``solver`` marker so it deselects under ``-m "not solver"`` like the other recourse solver tests (it already carried ``slow``). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…nvariant) D0 introduced _recourse_invest_active with a provider-first / workdir-disk fallback read of solve_data/stochastic_invest_method.csv. The pl.read_csv disk arm tripped test_meta_provider_invariants::test_no_disk_csv_reads_in_cascade (Rule 1) — the same seed-funnel bug class fixed for the Slice B obligation checker in 91db38b. Make the reader Provider-only (_provider_has_key -> _provider_read), matching every other reader in this module (_build_recourse_anchor_pairs et al.). The flag CSV reaches the cascade exclusively through the Provider on all production paths: cascade entry points thread an explicit Provider, and the snapshot-reload path (load_flextool off a dumped workdir) seeds the Provider from solve_data/ via seed_provider_from_dir before the cascade runs, so it carries the solve_data/stochastic_invest_method key by construction. The recourse gate is unchanged (returns True under flag='recourse' on both the normal and reload paths). Update test_recourse_plumbing to seed a Provider from the workdir (mirroring the cascade) instead of relying on the removed disk arm; add a no-Provider case asserting the byte-parity-safe default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The v70 migration (_migrate_v70_stochastic_invest_method) attached
solve.stochastic_invest_method to the solve_advanced parameter_group only
"when that group exists", but probed existence with the raising
db.item(db.mapped_table("parameter_group"), name="solve_advanced"). In the
current spinedb-api, db.item RAISES SpineDBAPIError when the item is absent
(it does not return None), so the `is not None` guard never helped — it
crashed before returning.
solve_advanced is created by the v44 migration (recoloured in v45), so real
DBs migrated from < v44 and the shipped template always carry it. But a DB
built from scratch at a version above v44 never ran the v44 step and has no
parameter_groups at all — e.g. the minimal v64 fixtures that
tests/test_v65_migration.py and tests/test_v66_migration.py seed and then
migrate forward through v70. Those two tests crashed on the v70 step
(regression vs main, where they pass).
Fix: use the non-raising db.get_item guard and get-or-create the group —
create solve_advanced (mirroring v44's priority 87 and the post-v45 colour
b56f6f) only when the DB lacks it, then always attach the param. This keeps
the param in its canonical schema home and satisfies the v69 invariant that
every parameter belongs to a group; it is a clean forward add, not
crash-tolerance. The v70 fixture (lh2_three_region) already has the group,
so the migrate-from-v69 path is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The three recourse process-globals (_RECOURSE_ANCHOR_PAIRS, _RECOURSE_WALK_PROVIDER, _RECOURSE_WALK_WORKDIR) were reset only inside the three flag-aware Layer-4 boundaries. Add _reset_recourse_scope() called once at the top of _apply_db_overrides (the single per-solve cascade entry, before pass 1a) so a future per-period producer wired into an earlier cascade step cannot inherit a stale global from a prior flag-on solve in a chained flag-on->flag-off multi-solve. The boundaries still overwrite unconditionally; this is belt-and-suspenders. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…icit The final cast(d_dtype, strict=False) could silently null an out-of-vocab token, contradicting the docstring's claim that the round-trip "cannot silently null a token." Mirror the sibling _derived_branch.d_leaf_lf: on the non-Utf8 (Enum) path, collect and hard-raise on any nulled token instead of dropping it. Cannot fire in valid runs -- anchor tokens arrive already typed as d_dtype (lossless Utf8->Enum round-trip) and branch (br) tokens are covered by the 4.0.4 vocabulary splice. Utf8-d callers (both production call sites) skip the collect entirely, so no hot-path laziness/perf regression. Docstring corrected to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Enable create_stochastic_periods to trigger the branch fan at a period k>1 (mid-horizon reveal) while keeping pre-reveal periods as a single real-named trunk (Option B, non-anticipativity by construction). - _stochastic.py: replace the 'realized row at the solve first step' validation with a well-formedness check (period-boundary reveal + exactly-one-realized per branching period), admitting a boundary mid-horizon reveal byte-identically for fan-at-first-step. - _timeline.py make_step_jump + _derived_params.py dtttdt_from_source: a branch copy with no earlier same-time-branch period now links its first step to the nearest anchor STRICTLY BEFORE the copy's own anchor (shared pre-reveal trunk end-state) instead of self-cycling. First- period fan self-cycles unchanged (empty anchor range) -> byte-parity. Byte-parity gate green (recourse 196875, storage *_na goldens unmoved). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…dedup
Option B correctness for the shared pre-reveal trunk:
- _lineage_leaves now returns (label, members) pairs (F4) so callers
carry the authoritative leaf label instead of a brittle zip-alignment.
- d_leaf: partition -> RELATION. A shared pre-reveal anchor belongs to
EVERY branch leaf's path, so the per-path total cap counts it into
each leaf's total (byte-parity for fan-at-first/deterministic: no
pre-reveal anchors).
- New dd_same_scenario_annuity lineage: the NPV annuity / fixed-cost
window walks dedup branch copies of the same CALENDAR slot by anchor,
so a shared trunk's window counts each future period once (window
{p2035,p2040}=2), not once per branch (=3). Reconciles design
§4.1 (capacity edd keeps the full frame with (p2035,p2040_low)) with
§8.1 (annuity=2). Fixes the latent NPV annuity over-count that the
shared trunk exposes. No-op for every non-shared-trunk solve.
- check_recourse_npv_preconditions obligation (i): re-scope leaf-weight
constancy to the POST-REVEAL segment (exclude shared pre-reveal
anchors, which carry the certain weight 1.0) per the function's own
Slice-E note; the two-stage weighting is correct, not a violation.
Fixture stoch_two_period_hedge (RP mid-horizon 173250 / WS fan-at-first
157500) + conftest wiring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
- test_recourse_invest_hedge.py: T-RP (173250, hedge x=60/r=60/0), T-WS (157500, EVPI=15750>0), T-EEV (192937.5 hand-calc, VSS>0), T-F/T-V (mid-horizon fan + validation, solver-free unit harness), T-L/T-E2 (dd_same_scenario shape A, d_leaf relation, pd_non_anticipativity empty, no invest-NA family), T-C (per-path cap two-leaf), T-S (storage NA fires on (p2040,p2040_low), branch copy links to shared trunk, make_step_jump <-> dtttdt parity). - hedge fixture gains a hedge_storage scenario (stochastic-group storage node resv) for the storage-NA / continuity assertions. - docs/how_to.md + CHANGELOG: mid-horizon hedged two-stage reveal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Completes the E1-E3 annuity-lineage addition (functions defined and consumed by _derived_npv; finish the public export). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…ng fix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The v70 stochastic_invest_method migration (Slice C) regenerated only the 2 generate_canonical recipes; the 7 tests/fixtures and 8 canonical_databases JSONs were left at v69, failing the CI test_fixtures/canonical_databases verify steps. Pure schema-additive regen (new value list + param definition + version bump); no test behavior or golden change (tests migrate JSON on load). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
sync_master_template runs migrate_database, whose sqlite engine handle is still open when TemporaryDirectory cleanup runs -> WinError 32 on Windows. Same check runs on the ubuntu-only template-check CI job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
test_migration_reaches_v70_and_is_idempotent loaded lh2_three_region.json from the test_fixtures corpus and asserted start_version < 70. But that corpus is regenerated to HEAD (>= v70) by `test_fixtures migrate-all`, so after the v70 regen it starts AT v70 and `70 < 70` fails — the two requirements are contradictory for the same file. Seed a minimal v69 DB from scratch (model + solve classes, version pinned to 69) via new `_build_v69_solve_db`, mirroring the v65/v66 migration tests, and migrate with `up_to=70`. This still exercises the v70 block from below and verifies idempotency, while staying green as the schema chain grows past 70. The v70 step already tolerates such minimal DBs (creates solve_advanced if absent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…in file Several test files opened the checked-in `templates/examples.sqlite` directly. Under `-n auto --dist loadfile` those files run on DIFFERENT xdist workers, so multiple worker processes open the SAME sqlite file concurrently. On Windows, sqlite lock contention on a shared file can block indefinitely, and pytest-timeout (900s, `thread` method) cannot interrupt a native lock wait, so the job hangs until GitHub's 6h cap cancels it (observed: test_parameter_group_coverage hung ~3h48m on windows-3.12; passed on the other matrix legs). It also violates CLAUDE.md invariant #3 (tests must not read a checked-in .sqlite). Add session-scoped `examples_db_path` / `examples_db_url` fixtures in tests/conftest.py that copy the materialized template to a per-worker temp path (tmp_path_factory's basetemp is unique per xdist worker), so no two workers ever touch the same file. Session scope copies once per worker and is a valid dependency for the module-scoped consumer fixture. A copy is byte-identical, so every reader's assertions are unchanged. Refactored readers of templates/examples.sqlite: - test_parameter_group_coverage.py (the hang) -> fixture URL, with-close - spinedb_backend/test_memory_budget.py -> fixture URL - spinedb_backend/test_parsed_value_eviction.py -> fixture path in the module-scoped backend fixture - test_handoff_writers.py -> thread the isolated URL into the subprocess test_examples_e2e.py reads a different checked-in DB (projects/examples/input_sources/examples.sqlite); it now reads a per-test tmp_path copy for the same invariant-#3 reason. Already-isolated readers (materialize from JSON into tmp) left unchanged: test_export_to_tabular.py, test_xlsx_workflow.py, engine_polars/test_param_shapes_phase_a.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…st, per-scenario trimmed Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…solation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Add the solve.non_anticipativity_periods parameter (Array of periods, default unset) that decouples the operational non-anticipativity window from realized_periods (Slice G, Axis B). - v70->v71 migration (_migrate_v71_non_anticipativity_periods): Array param with default_value=None/default_type=None (no value list), grouped under solve_advanced. Adds the definition only, never a per-solve value, so every migrated solve stays on the unset -> legacy-window path (byte-identical). - PARAMETER_TYPES entry (DIMENSIONLESS) per autoscale Invariant 1. - Regenerated all four CI artifacts: spinedb_schema.json (master template), test-fixture corpus, canonical_databases JSON, canonical DBs. All four --verify steps pass; diffs are the mechanical v71 bump plus the new definition row. - v71 idempotency + definition-shape test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Wire the resolved non_anticipativity_periods window (Slice G, Axis B) end to end, single source of truth, no new CSV. - _solve_config: tri-state reader non_anticipativity_periods_to_tristate keys off raw find_parameter_values presence (periods_to_tuples' defaultdict cannot tell unset from stored []): None=unset, []=free-ops, [(p,p),...]=curated. Stored in a plain dict so a missing key reads back None. Read after the four period dicts; propagated through duplicate_solve's dup_map_list (F3) for rolling/laddered sub-solves. - _emit_per_solve: emit_per_solve_sets drives dt_non_anticipativity_set.csv from the window (F1) — None=legacy realized u fix union (byte-identical), []=header-only, [periods]=those periods' timesteps intersected with steps_in_use (the storage family's only domain guard, F7). Threaded in from _emit_solve_time.run (runs during preprocessing, before build). - _derived_branch.dt_non_anticipativity_lf repointed to READ the authoritative set CSV; present-but-empty -> empty frame -> model's height==0 early-out skips all four families. Legacy union kept only as a defensive fallback when the CSV is absent. Byte-parity green (unchanged goldens): test_non_anticipativity, test_stochastic_continuation_fanout, test_derived_branch_lineage, non_anticipativity_storage_use_emits, recourse gate/hedge/na — all under FLEXTOOL_AUTOSCALE_STRICT=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Two-period t0-branching fixture stoch_two_period_free_ops.json (mid/up/low branches at t0001, shared investable base with stochastic_invest_method= none, stochastic-group storage node resv with branch-asymmetric wind timing -> unique per-branch storage optimum, F6). Two scenarios differ ONLY in non_anticipativity_periods. test_configurable_nonanticipativity.py (10 tests, all @solver): - free_ops (non_anticipativity_periods=[]): header-only set CSV, zero non_anticipativity_* rows, ONE shared v_invest (no branch fan), and v_state[resv] strictly divergent across branches (up>0.5, low<0.5, mid idle); objective 757_369.4. - pinned_ops (unset): set CSV = legacy 6-step horizon, storage NA fires (12 rows), v_state equal across branches (byte-parity guard); objective 1_920_800. - free_obj < pinned_obj monotonicity. conftest: stoch_two_period_free_ops_db_url session fixture + scenario_ workdir wiring (three-touchpoint pattern). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Document the new solve.non_anticipativity_periods knob in the multi-period stochastics section: unset = legacy pinned-over-horizon window; empty array = free operations from t0 (two-stage capacity expansion with a shared invest); period list = curated window (e.g. mid-horizon reveal). Notes that a curated list replaces the default window and that realized_periods still governs the dispatch output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The v71 solve param was missing from export_settings.yaml, failing test_no_schema_params_missing_from_yaml. Added alongside the sibling period-array params (realized_periods etc.). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…config-load-fan limit Code-review follow-ups (non-blocking): a non-Array value now resolves to the legacy window instead of silently freeing operations (G-2); docstring now states explicit windows do not propagate to config-load-fanned 2D-Map-laddered solves (G-1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…ract) The parameter_value_dtypes override marked solve.stochastic_branches as Utf8 alongside the period-name params, but its map leaf is the branch WEIGHT (a number). The comment says overrides list only genuinely non-numeric leaves; removing it lets the weight use the Float64 default. Dormant today (the live reader path returns floats via find_parameter_ values), but corrects the classification for any future SpineDbReader consumer. No data migration — DB weights are already numeric. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…ons + overrides Adds an upfront subsection stating what a dispatch solve and an investment solve tie across branches by default (storage/online/reserve over the realized window; flows free; single shared investment) and how to override each (stochastic_invest_method, non_anticipativity_periods). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…he main mode The main stochastic-investment mode is a single shared here-and-now investment with operations free per scenario from t0 (none + non_anticipativity_periods=[]), not the mid-horizon hedge. The hedge is recast honestly as the variant that shares NEAR-TERM OPERATIONS too (first periods fully deterministic). Notes the multi-period investment non-anticipativity window (shared-first-period / recourse-later) as a planned addition. Wait-and-see framed as a screening tool. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Add solve.non_anticipativity_invest_periods — the standard two-stage stochastic-investment mode knob (Array of periods over which v_invest / v_divest are tied across stochastic branches, a single shared here-and-now decision). Default unset == [] == NO tie, byte-identical to today's recourse/none. - db_migration: _migrate_v72_non_anticipativity_invest_periods mirrors the v71 non_anticipativity_periods migration (Array, null/null default, no value list, solve_advanced group); FLEXTOOL_DB_VERSION 71 -> 72 + dispatch arm. - PARAMETER_TYPES row (DIMENSIONLESS) so autoscale does not silently degrade to an un-scaled LP. - export_settings.yaml: split_params.solve_period.params AND data_type_overrides.nested_periodic (both locations, F6). - _solve_config: read via plain periods_to_tuples (unset == [] == no tie, so no tri-state needed) + dup_map_list propagation for rolling/laddered sub-solves. - Regenerated all four CI artifacts (spinedb_schema.json master template, canonical databases, test fixtures, generate_canonical); the only diff is the mechanical v72 version bump + the new parameter-definition row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Populate pd_non_anticipativity by window membership and tie invest AND divest across branches over that window (design §3-§5). - _emit_per_solve: emit solve_data/pd_non_anticipativity_invest_periods.csv ONLY when the window is non-empty (absent file -> legacy empty frame -> byte-parity, no new CSV for unset fixtures); threaded from _emit_solve_time.run. - _derived_branch.pd_non_anticipativity_lf: new tie_periods arg + ctx/CSV reader (_read_invest_tie_periods). When resolved, emit (anchor_of[m], m) for every synthetic member whose anchor is in the window (supersedes the never-supplied branch_start reveal-boundary gate, which stays as the byte-parity legacy path -> provably empty today). - model.py: _add_non_anticipativity_invest_constraints emits all four families non_anticipativity_invest_p/n and _divest_p/n (divest is mandatory, F2/§4.4 — tying invest alone leaks net first-stage capacity). Gated on recourse_invest + non-empty frame. The cross-Enum domain guard carries the db_pairs Utf8 dance (F3): pd_non_anticipativity.b is branch-Enum while pd_invest_set.d renamed d->b is period-Enum, so both b keys are cast to Utf8 for the guard join and restored via cast_dim. - autoscale: register all four family names (CstrFamily(None), RHS=0) + pin them in test_registry_coverage dynamic samples (passed as a variable, invisible to the literal grep). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Extend the hedge builder with fan-at-t0 invest-NA scenarios and add the
solver/structural gate (design §9).
- build_stoch_two_period_hedge: new scenarios on the fan-at-t0 (ws)
topology — hedge_na_window (window=[p2035] + ops free), hedge_na_divest
(base invest_retire_total + existing 50 + salvage_value so v_divest
exists), hedge_na_none (stochastic_invest_method=none).
- test_invest_na_window: T-Main obj=173,250 with p2035 v_invest EQUAL
across branches (60) and p2040 per-branch (recourse 60/0); T-WS control
157,500 with RP−WS=15,750; T-Frame {(p2035,p2035_low)} (empty when
unset); T-Weight branch-weight partition 0.25/0.75; T-C invest rows;
T-Divest divest tied + net capacity equal across branches; T-None
graceful no-op; T-Reg four families registered. All under
FLEXTOOL_AUTOSCALE_STRICT=1 (T-Main is the first coverage of the F3
cross-Enum guard join).
- model.py: T-None advisory log — window set on a non-recourse solve is a
graceful no-op (never mutates the problem, §4.5).
- test_v72 migration: idempotency + definition shape (Array, null/null
default, solve_advanced group), mirroring the v71 test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Replace the "planned addition" note in how_to.md with the delivered three-knob recipe (recourse + non_anticipativity_invest_periods=[first] + non_anticipativity_periods=[]) and add the CHANGELOG entry for the standard two-stage stochastic-investment mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Code-review nit: _read_invest_tie_periods had a speculative ctx-first
branch that nothing populates and that would silently iterate
solve-name keys if a future caller wired the {solve: [...]} config.
The emitted CSV is authoritative; removed the branch, documented why
ctx is not a source.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on the Slice G branch (top of the recourse stack). Delivers the standard two-stage stochastic capacity-expansion mode — the one that was previously not achievable: a single shared here-and-now investment over the first period(s), per-scenario recourse investment later, and operations free per scenario from timestep 0.
What it adds
New solve param
non_anticipativity_invest_periods(Array of periods; schema v72). It tiesv_invest/v_divestacross stochastic branches over the chosen periods — a single shared decision where the branches share information.[](default) → no tie → byte-identical to today (recoursestays per-branch,nonestays shared).[period1]underrecourse→ the main mode: investment shared in period 1 (the genuine here-and-now first stage), per-branch recourse investment afterward. Combine with Slice Gnon_anticipativity_periods = []for operations free from t0.This finally decouples the shared first stage from the operational tie: previously you could only get all-shared investment (
none), all-per-branch (recourse), or the mid-horizon hedge (which also shares near-term operations). The investment-NA window consumes thepd_non_anticipativityframe that Slice A built and Slice E deferred, adding thenon_anticipativity_invest/divest_p/nconstraint families.Correctness points (from the adversarial review)
v_investwould leak anticipativity through the net first-stage capacity (existing + invest − divest) for any reinvest/lifetime entity.db_pairspath. This join class had no prior test coverage — the solver-level hand-calc test is its first.Verification
recourse+non_anticipativity_invest_periods=[period1]+non_anticipativity_periods=[]→ period-1v_investequal across branches, period-2 per-branch, operations divergent from t0; objective RP = 173,250 at a unique LP vertex, exactly matching Slice E's hedge (shared-first-stage-by-constraint ≡ shared-trunk-by-construction), withRP − WS = 15,750(EVPI). Divest-tie and node-side tests included.Next (spec'd): construction lead time (Slice H) builds on this — per-technology lead times, where per-technology staging emerges automatically from this window once
v_investis order-indexed, plus the commissioning lag.🤖 Generated with Claude Code
https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ