Skip to content

Investment non-anticipativity window — the standard two-stage stochastic investment mode - #339

Open
jkiviluo wants to merge 49 commits into
mainfrom
feat/investment-na-window
Open

jkiviluo wants to merge 49 commits into
mainfrom
feat/investment-na-window

Conversation

@jkiviluo

Copy link
Copy Markdown
Member

Stacked on the Slice G branch (top of the recourse stack). Delivers the standard two-stage stochastic capacity-expansion mode — the one that was previously not achievable: a single shared here-and-now investment over the first period(s), per-scenario recourse investment later, and operations free per scenario from timestep 0.

What it adds

New solve param non_anticipativity_invest_periods (Array of periods; schema v72). It ties v_invest/v_divest across stochastic branches over the chosen periods — a single shared decision where the branches share information.

  • unset / [] (default) → no tie → byte-identical to today (recourse stays per-branch, none stays shared).
  • [period1] under recourse → the main mode: investment shared in period 1 (the genuine here-and-now first stage), per-branch recourse investment afterward. Combine with Slice G non_anticipativity_periods = [] for operations free from t0.

This finally decouples the shared first stage from the operational tie: previously you could only get all-shared investment (none), all-per-branch (recourse), or the mid-horizon hedge (which also shares near-term operations). The investment-NA window consumes the pd_non_anticipativity frame that Slice A built and Slice E deferred, adding the non_anticipativity_invest/divest_p/n constraint families.

Correctness points (from the adversarial review)

  • All four families are tied — invest and divest, process and node. Tying only v_invest would leak anticipativity through the net first-stage capacity (existing + invest − divest) for any reinvest/lifetime entity.
  • The constraint's domain-guard join is cross-Enum (branch vocabulary vs period vocabulary); it uses the Utf8-cast dance from the working db_pairs path. This join class had no prior test coverage — the solver-level hand-calc test is its first.

Verification

  • Hand-calc gate (solver): fan-at-t0 fixture with recourse + non_anticipativity_invest_periods=[period1] + non_anticipativity_periods=[] → period-1 v_invest equal across branches, period-2 per-branch, operations divergent from t0; objective RP = 173,250 at a unique LP vertex, exactly matching Slice E's hedge (shared-first-stage-by-constraint ≡ shared-trunk-by-construction), with RP − WS = 15,750 (EVPI). Divest-tie and node-side tests included.
  • Full sweep: 3795 passed, 0 failed; all 4 CI schema-verify steps pass; zero golden value changes beyond the mechanical v72 diff.

Next (spec'd): construction lead time (Slice H) builds on this — per-technology lead times, where per-technology staging emerges automatically from this window once v_invest is order-indexed, plus the commissioning lag.

🤖 Generated with Claude Code

https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ

jkiviluo and others added 30 commits September 22, 2026 16:01
…pd_non_anticipativity)

Two always-on derived frames in _derived_branch.py, consumed by
nothing (recourse plan §6b Slice A; design
specs/sliceA_lineage_frames_design.md r3):

- dd_same_scenario (d, d_other): symmetric/reflexive pairs sharing a
  scenario leaf-path; domain = period_in_use only; all-pairs
  degenerate content for deterministic solves; leaf rule with the F6
  authority convention (branch-local member displaces the anchor).
- pd_non_anticipativity (d, b): period-level invest-NA pairs
  (strictly-pre-reveal anchor ties), parameterized on an optional
  branch_start frame (Slice E hook); provably empty for every
  currently-constructible solve under the min-anchor fallback.

Wiring: unconditional assignment in apply_branch_cluster (step 2b),
guarded-ctx input acquisition per the dtttdt_from_source null-guard
precedent, ValueError failure semantics on inconsistent per-solve
artefacts, Utf8-construct -> non-strict cast_dim -> live null
assertion -> deterministic Utf8-key sort.  FlexData registration
(two typed fields, None only pre-cascade), d_other axis synonym.
No consumers, no schema change, no migration, no autoscale entries
(nothing enters the LP), no dump-side change (_copy_meta already
glob-copies the rebuild inputs).

Tests: T1-T12 per design §4/§5 (17 tests) — five exact shape tables,
both mid-horizon shapes incl. the branch_start-driven [(p1, p1_low)]
pair, symmetry property, metadata exclusion, failure semantics,
_drop_master_rows pass-through, integration + enum hygiene on the
stoch_two_period fixture, and the first stochastic dump/reload
round-trip (pins the _copy_meta glob).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Slice B of the recourse-invest plan (specs/sliceB_walker_lineage_design.md).
Add a keyword-only lineage=pl.DataFrame|None=None parameter to the shared
period_walk_iterator, applied PRE-aggregation under the Slice A semi-join
contract (remove (e,d,d_all) iff both ends in period_in_use and (d,d_all)
not in lineage; history-anchored rows pass unconditionally).

- _derived_walks: _apply_lineage_filter (anchor_col/dall_col params,
  fixed-direction lineage-side non-strict cast, mandatory .unique(),
  marker left-join) + _assert_lineage_castable null guard; zero-cost
  None path behind a single if.
- Thread lineage=None through inventory #1-17 (cohort/variant/from_source
  chain in _derived_existing, _derived_npv, _derived_params) and apply
  the shared helper in the one non-walker pair-former #18
  edd_divest_active_from_source (anchor_col="d_divest", dall_col="d").
- _derived_branch: check_recourse_npv_preconditions +
  assert_recourse_npv_preconditions — the three named NPV proof
  obligations (leaf-weight constancy, cohort sum-to-1, byte-equal
  fan-member year rows), shipped as a pure checker with the F4
  soundness-scope docstring; added to __all__.

Inert: every production call passes lineage=None; no flag, no consumer
activation, no schema change, no autoscale entries, no LP change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Slice B test suite (specs/sliceB_walker_lineage_design.md §9).

Capability harness (solver-free, hand-built frames + stub source):
- W1 set-shape (history passes, cross-leaf both-ends-PIU pairs removed)
- W2 factor-side PRE-aggregation proof (sums 7.0/3.0/4.0 vs unfiltered
  7.0 — a post-aggregation filter cannot produce 3.0/4.0)
- W3 lineage=None 12-row unfiltered pin
- W4/W4b empty-frame contract (set: 4 history rows; factor: single
  (h2020,7.0) row, anchors absent not zeroed)
- W5a duplicate-pair .unique() guard; W5b uncastable-token ValueError;
  W5c Enum x Enum missing-token raises / all-present identical

Stochastic fixture pins (solver-required fixture, solver-free asserts):
- W6 edd_invest_set today-bug witness ((wind,p2040,p2040_low) absent =
  erratum E1 first-stage-visibility break)
- W7 edd/divest non-vacuity + repeat-determinism; NPV variant vacuity
  pins + invest-method overlay leg
- P1/P2 obligation checker clean on stoch/deterministic workdirs
- P3a-d hand-seeded obligation violations (i)/(ii)/(iii)
- P4 walker year-source blindness diagnostic (erratum E1 witness pair)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The Slice B NPV obligation checker's year-row reader
(`_read_solve_data_csv`) fell back to a bare `pl.read_csv(...)` on the
workdir CSV when the Provider didn't carry the frame. That disk arm
tripped the post-Step-2 cascade invariant
(test_meta_provider_invariants::test_no_disk_csv_reads_in_cascade,
Rule 1) — the seed-funnel bug class the invariant guards.

Make the reader Provider-only (`_provider_get` -> `provider.get`),
matching every other reader in `_derived_branch.py`. The checker now
does no disk I/O; the caller owns placing the true emitted year CSVs
on the Provider (tests seed via seed_provider_from_dir / _mk_provider;
the Slice D activation site inherits the same solve_data frames the
rest of the cascade already consumes). Obligation (iii) semantics
preserved. SOUNDNESS SCOPE docstring + design (local) sec 6.2 updated
with the deviation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Add the opt-in solve.stochastic_invest_method enum (value list
stochastic_invest_methods: none/recourse, default none) via migration
v70.  'none' is byte-identical to prior behaviour; 'recourse' is
reserved for Slice D and hard-rejected by the Slice C guards.

- FLEXTOOL_DB_VERSION 69 -> 70; _migrate_v70_stochastic_invest_method
  mirrors the v64 solve.scaling exemplar (value list + definition +
  guarded solve_advanced grouping).
- PARAMETER_TYPES row ('stochastic_invest_method','solve') DIMENSIONLESS.
- _solve_config read/store + stochastic_invest_method_for resolver
  (absent -> 'none').
- Regenerated spinedb_schema.json master template + the two managed
  canonical DBs (templates_examples, howto_stochastics).
- export_settings.yaml split_params: surface the new solve knob.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Three flag-independent / flag-keyed validation guards that keep the
recourse opt-in inert until Slice D lands.

- Guard 1 (flag hard-reject): stochastic_invest_method != none raises
  FlexToolConfigError at the top of _orchestration.run() (authoritative,
  covers rolling/contained solves via the roll-suffix strip) plus a
  fail-fast pass in _validate_model_solve over the top-level solves.
- Guard 2 (Benders x genuine stochastics): _is_genuinely_stochastic
  raises in _run_benders_solve (primary) and at the top of solve_benders
  (defensive).  Detector = period_branch_full d!=b b-tokens intersected
  with period_in_use_set, so the rolling realized-only bookkeeping shape
  does NOT false-positive.
- Guard 3 (handoff synthetic-name assertion): build_handoff_from_solution
  rejects any realized_invest / committed period that is a synthetic
  branch fan member; None-degrades when flex_data is unavailable.

New shared cycle-free module _stochastic_detect.py holds the single
token-extraction idiom (synthetic_branch_tokens) used by Guards 2 and 3
and the PIU-intersected is_genuinely_stochastic used by Guard 2.  Both
cast d/b to Utf8 before comparing — at runtime period_branch_full is
Enum-typed with disjoint d/b category sets, and comparing two enums
raises in polars (would otherwise crash Guard 3 on every stochastic
handoff); null b (deterministic shape) is dropped.

Empirically verified on stoch_two_period: genuine stochastic -> True
(the _rlz metadata members are excluded by the PIU intersection, the
_low branches survive); deterministic + rolling -> False.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
- test_v70_stochastic_invest_method.py (M1-M3): migration round-trip
  (definition present, default none, bound to stochastic_invest_methods,
  grouped solve_advanced; value list {none,recourse}); pre-v70 -> v70
  reach + idempotent re-run; master-template --verify up to date.
- test_recourse_guards.py (G1, G2a-c, G3): resolver normalisation +
  _validate_model_solve reject/pass + rolling-suffix-strip composition
  (G1); PIU-intersected detector on genuine/deterministic/self-row/
  rolling-realized-only shapes incl. the Enum disjoint-category
  regression pin and solve_benders defensive raise (G2); solver-based
  handoff clean-path / config-level synthetic raise / None-degrade (G3).

Detector shapes mirror the empirically verified stoch_two_period
runtime frames (Enum-typed d/b, PIU excludes _rlz metadata members).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Emit solve_data/stochastic_invest_method.csv (resolved per-solve mode)
next to the realized-invest emit; _recourse_invest_active reader
(provider-first, workdir fallback, absent -> False); FlexData.recourse_invest
bool stamped in apply_derived_c; _has_branch_invest_axis capability conjunct
reusing is_genuinely_stochastic. Dump-registry + accumulator-manifest lines
so a dump/reload round-trip preserves the flag. Flag defaults none -> no
behavior change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Define LineageFilterError(ValueError) in _solve_state; the two Slice B
lineage guards (_assert_lineage_castable, assert_recourse_npv_preconditions)
raise it. Narrow the four blanket except-Exception swallows in
apply_derived_c / apply_synthetic_invest_sets (edd lookback / edd_invest /
edd_divest_active / synthetic edd) to re-raise LineageFilterError so a
lineage failure is loud instead of silently reverting to unfiltered
first-stage sets. Raise path is unreachable while lineage is None
(flag-off byte-parity).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
(A) _expand_invest_branch_periods — invest-side mirror of the dispatch
_expand_branch_periods; wraps the 5 axis-defining _solve_periods(invest_periods)
sites (ed_invest_set, ed_divest_set, ed_invest_forbidden, eager
ed_entity_annual_family, apply_npv) so period_invest gains {p_k, p_k_b}
under recourse. ctx/provider threaded through those readers + their
apply_derived_c / apply_synthetic callers. period_first + PIU-fallback
sites left unwrapped (design §3.1).

(B) _anchor_expand_explicit + module-scoped anchor-pairs holder set at each
Layer-4 boundary (apply_derived_c / apply_npv / apply_synthetic): explicit
per-period rows (invest_cost/discount/lifetime/fixed_cost Maps + per-period
caps) inherit their anchor's value onto branch periods, wired into the three
producers (_resolve_per_period_lf, _per_entity_period_cost,
_ed_explicit_period_param). Holder None flag-off/deterministic -> no-op ->
byte-parity.

Flag-off spot-suites (fanout, npv_cluster, lineage) green; new axis unit
tests cover expansion + anchor inheritance.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Thread provider through period_walk_iterator into _p_years_d_lf (year side)
and _inflation_factors_lf -> _years_for_period_lf -> _years_for_period_from_source
(factor side). Add a provider-gated canonical arm to _years_for_period_from_source
reading the emitted p_years_represented.csv (which byte-copies fan-member
year rows via derive_years_represented's branch loop). Under recourse the
caller (D4 apply boundaries) forwards provider so branch periods resolve
their anchor's year-from-start -> correct annuity windows; provider=None
(flag-off) keeps the canonical arms dead -> today's fill_null behaviour
(the W6 today-bug pin) -> byte-parity. Plumbing only this phase (no caller
forwards provider yet); activation lands with lineage at D4.

Flag-off npv_cluster/walk/fanout suites green; new canonical-arm unit tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Compute the scenario-lineage frame (flex_data.dd_same_scenario) at the
three Layer-4 apply boundaries, gated on recourse AND a genuine branch
invest axis (capability conjunct). Forward lineage=... into the edd
builders threaded by Slice B: edd_invest_lookback_set, edd_invest_set_lf,
edd_divest_active (#18), the synthetic edd, and apply_npv's four
ed_*_from_source walks. Hoist assert_recourse_npv_preconditions above the
edd try/except blocks so a violation raises LineageFilterError (loud, D1).

alpha-1 walker Provider: a boundary-scoped module-global (_RECOURSE_WALK_PROVIDER)
armed alongside the anchor-pairs scope; period_walk_iterator falls back to
it so the canonical year/factor arms revive under the flag without threading
provider through ~12 NPV/edd caller signatures. None flag-off -> arms dead
-> W6 today-bug preserved.

Activation is inert flag-off (lineage None) -> byte-parity; 103-test
spot-suite (npv_cluster/fanout/lineage/axis/year) green. Flag-on edd
cross-leaf integration lands at D5 (Guard 1 lift + fixture).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Objective: multiply ALL EIGHT invest/divest cost Params (process
annu/lf/lfd/annd + node annu_n/lf_n/lfd_n/annd_n) by pd_branch_weight[d]
via the _wt closure, gated on d.recourse_invest AND pd_branch_weight;
replace the .mod "NOT weighted" comment with the recourse rationale.
Flag-off skips the multiply (byte-parity).

Guard 1 LIFTED for 'recourse': _stochastic_invest_raw_guard validates the
RAW authored value (resolver collapses typos to 'none' so the guard must
read the raw dict); both sites (run() + _validate_model_solve) switched;
guard tests flipped per design §15.4 + NEW raw-typo rejection test.

Activation fixes found by the gate (recorded in design implementation
notes): (a) precondition checker needs the REAL workdir (Provider-only
readers key paths off it — design §8 snippet's None arg made obligations
fail spuriously); (b) lineage built ON-DEMAND per boundary via
dd_same_scenario_df (flex_data.dd_same_scenario populates only at
derived_g); (c) alpha-1 walker holders gain the workdir twin
(_RECOURSE_WALK_WORKDIR) so the canonical p_years_d arm goes live;
(d) _recourse_usable_ctx sanitizes the snapshot-reload SolveContext
(all-null d_anchor artifact) with provider-arm fall-through.

Gate fixture stoch_two_period_invest.json (+ builder, + conftest wiring):
supply-side branch variation (deterministic demand 100/130 + wind
upper_limit profile rlz .4/.4 low .2/.1 via group__unit -> peaker residual
60/90 / 80/120) because node-inflow branch resolution gates on group__node
and a stoch-group node would fire NA net-charge pinning.

D5 gate: objective 196875 EXACT; coefficients 525/262.5/1575/787.5;
per-leaf invest 60/80/30/40; committed output realized-only; edd
cross-leaf ABSENT (W6 re-pointed); pd_non_anticipativity empty; STRICT
autoscale clean. Flag-off parity: 128 tests green (lineage/walk/fanout/
guards/npv_cluster + recourse unit suites). ruff clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Per-scenario-path total-capacity caps so a total cap applies once per
scenario leaf under recourse, never cross-scenario double-counting
(design sliceD §7 (G) / §14 (L)).

- d_leaf (d, leaf) frame in _derived_branch.py: partitions period_in_use
  into scenario leaves (real anchors -> "__realized", synthetic members
  -> time-branch id); registered on FlexData, built in apply_branch_cluster.
- model.py _emit_entity_total_cap: the four maxInvest/maxDivest entity
  total builders emit the legacy over=("d",) shape + name for a single
  leaf (deterministic / flag-off, byte-identical) and …_path indexed by
  (entity, leaf) for >1 leaf, masking the inner sum to each leaf.
- _cumulative_invest.py _emit_group_total_divest_cap: the divest-side
  group total gains the same per-path treatment (max/<= cap only).
- autoscale registry: maxInvest/maxDivest_entity_total_path(_n) (POWER/
  ENERGY) + maxDivestGroup_entity_total_path (group_capacity resolver);
  test_registry_coverage dynamic-name pins.
- Fixture invest_max_total tightened 250 -> 150 (§15.1): per-leaf totals
  90/120 non-binding, legacy all-d sum 210 would have bound — the
  asymmetry proves per-path. Gate test asserts two leaf rows.

Gate: objective 196,875 exact; STRICT autoscale clean; flag-off parity
(derived_branch/period_walk lineage, continuation fanout, recourse
guards, deterministic invest cumulative/total goldens) green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Make the recourse handoff's drop of branch-named (non-realized-scenario)
v_invest rows EXPLICIT (design sliceD §10.1 (H)).

- build_handoff_from_solution: partition invest_by_ed on synthetic
  branch-token membership BEFORE the commit loop and debug-log the
  dropped mass; the commit gate previously dropped these by accident.
  The Slice C Guard 3 assertion stays as a belt-and-braces invariant
  check (no synthetic token may survive into committed rows), not the
  mechanism.  Flag-off / deterministic: synthetic set empty → no-op
  (byte-parity).
- Gate test: the pre-filter drops the low leaf's 80+40 = 120 MW, debug-
  logs it (periods p2035_low/p2040_low), and commits only the realized
  60/30; Guard 3 stays inert.

Guard 1 raw-value lift was already landed at D5.  Flag-off parity
(handoff cumulative carriers, continuation fanout, recourse guards) green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Outputs, cost reconciliation, and the cross-scenario-coupling fixture
(design sliceD §11 (I) / §15.5 / §15.6).

Outputs (§11.2):
- read_highs_solution: under recourse AND output_horizon, union the
  horizon-expanded d_realize_dispatch_or_invest set (branch periods
  included) into the invest-axis canonical order, so non-realized
  branch v_invest rows surface.  Gated on both flags → byte-parity for
  deterministic models and for recourse without horizon output.
  Committed path stays realized-only (§11.1).

Costs (§11.3, choice I-a):
- calc_costs probability-weights the invest/divest + lifetime-fixed cost
  rows by pd_branch_weight under recourse, so the cost breakdown
  reconciles to the LP objective's (expected) invest contribution rather
  than over-reporting the realized spend by 1/w_realized.  read_parameters
  threads par.pd_branch_weight + par.recourse_invest.  Gated → factor 1.0
  off-flag (byte-parity; deterministic cost-aggregation goldens green).
- test_npv_cluster skips the eager-vs-lazy NPV byte-equality under
  recourse (the eager reference is not lineage-aware, §11.4).

Fixtures/tests:
- variant A gains a recourse_horizon scenario (model.output_horizon=yes);
  gate test asserts branch rows 80@p2035_low / 40@p2040_low surface.
- variant B (stoch_two_period_invest_na): extends the proven
  stochastics.json storage shape with a recourse_na scenario making the
  stochastic-group storage node investable.  Witnesses:
  non_anticipativity_storage_use fires (>0 rows), branch v_state == the
  NA-tied realized value, and the node-side annu_n objective coefficient
  carries pd_branch_weight (the F3 node-side pin).
- cost reconciliation e2e: reported realized invest cost 0.039375 M =
  0.25·(60·2100 + 30·1050); solver objective 0.196875 M.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
- how_to.md: replace the "only one period investment stochastic models
  are supported" line with the per-scenario (wait-and-see) recourse
  recipe — what the objective means (expected value of per-scenario
  plans, EVPI-style; not a hedged plan), the coupling change vs the
  default, the α-1 cross-comparability caveat, and the Benders exclusion
  (design sliceD §16).
- reference.md: document solve.stochastic_invest_method (none/recourse),
  the output_horizon interaction, and the Benders exclusion.
- CHANGELOG: Unreleased draft entry (public wording, no internal model
  names; schema change note).  No version bump — release is the owner's
  decision.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The end-to-end cost-reconciliation test invokes a real solve; add the
``solver`` marker so it deselects under ``-m "not solver"`` like the
other recourse solver tests (it already carried ``slow``).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…nvariant)

D0 introduced _recourse_invest_active with a provider-first / workdir-disk
fallback read of solve_data/stochastic_invest_method.csv.  The pl.read_csv
disk arm tripped test_meta_provider_invariants::test_no_disk_csv_reads_in_cascade
(Rule 1) — the same seed-funnel bug class fixed for the Slice B obligation
checker in 91db38b.

Make the reader Provider-only (_provider_has_key -> _provider_read), matching
every other reader in this module (_build_recourse_anchor_pairs et al.).  The
flag CSV reaches the cascade exclusively through the Provider on all
production paths: cascade entry points thread an explicit Provider, and the
snapshot-reload path (load_flextool off a dumped workdir) seeds the Provider
from solve_data/ via seed_provider_from_dir before the cascade runs, so it
carries the solve_data/stochastic_invest_method key by construction.  The
recourse gate is unchanged (returns True under flag='recourse' on both the
normal and reload paths).

Update test_recourse_plumbing to seed a Provider from the workdir (mirroring
the cascade) instead of relying on the removed disk arm; add a no-Provider
case asserting the byte-parity-safe default.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The v70 migration (_migrate_v70_stochastic_invest_method) attached
solve.stochastic_invest_method to the solve_advanced parameter_group only
"when that group exists", but probed existence with the raising
db.item(db.mapped_table("parameter_group"), name="solve_advanced"). In the
current spinedb-api, db.item RAISES SpineDBAPIError when the item is absent
(it does not return None), so the `is not None` guard never helped — it
crashed before returning.

solve_advanced is created by the v44 migration (recoloured in v45), so real
DBs migrated from < v44 and the shipped template always carry it. But a DB
built from scratch at a version above v44 never ran the v44 step and has no
parameter_groups at all — e.g. the minimal v64 fixtures that
tests/test_v65_migration.py and tests/test_v66_migration.py seed and then
migrate forward through v70. Those two tests crashed on the v70 step
(regression vs main, where they pass).

Fix: use the non-raising db.get_item guard and get-or-create the group —
create solve_advanced (mirroring v44's priority 87 and the post-v45 colour
b56f6f) only when the DB lacks it, then always attach the param. This keeps
the param in its canonical schema home and satisfies the v69 invariant that
every parameter belongs to a group; it is a clean forward add, not
crash-tolerance. The v70 fixture (lh2_three_region) already has the group,
so the migrate-from-v69 path is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The three recourse process-globals (_RECOURSE_ANCHOR_PAIRS,
_RECOURSE_WALK_PROVIDER, _RECOURSE_WALK_WORKDIR) were reset only inside
the three flag-aware Layer-4 boundaries. Add _reset_recourse_scope()
called once at the top of _apply_db_overrides (the single per-solve
cascade entry, before pass 1a) so a future per-period producer wired
into an earlier cascade step cannot inherit a stale global from a prior
flag-on solve in a chained flag-on->flag-off multi-solve. The boundaries
still overwrite unconditionally; this is belt-and-suspenders.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…icit

The final cast(d_dtype, strict=False) could silently null an out-of-vocab
token, contradicting the docstring's claim that the round-trip "cannot
silently null a token." Mirror the sibling _derived_branch.d_leaf_lf: on
the non-Utf8 (Enum) path, collect and hard-raise on any nulled token
instead of dropping it. Cannot fire in valid runs -- anchor tokens arrive
already typed as d_dtype (lossless Utf8->Enum round-trip) and branch (br)
tokens are covered by the 4.0.4 vocabulary splice. Utf8-d callers (both
production call sites) skip the collect entirely, so no hot-path
laziness/perf regression. Docstring corrected to match.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Enable create_stochastic_periods to trigger the branch fan at a period
k>1 (mid-horizon reveal) while keeping pre-reveal periods as a single
real-named trunk (Option B, non-anticipativity by construction).

- _stochastic.py: replace the 'realized row at the solve first step'
  validation with a well-formedness check (period-boundary reveal +
  exactly-one-realized per branching period), admitting a boundary
  mid-horizon reveal byte-identically for fan-at-first-step.
- _timeline.py make_step_jump + _derived_params.py dtttdt_from_source:
  a branch copy with no earlier same-time-branch period now links its
  first step to the nearest anchor STRICTLY BEFORE the copy's own anchor
  (shared pre-reveal trunk end-state) instead of self-cycling. First-
  period fan self-cycles unchanged (empty anchor range) -> byte-parity.

Byte-parity gate green (recourse 196875, storage *_na goldens unmoved).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…dedup

Option B correctness for the shared pre-reveal trunk:

- _lineage_leaves now returns (label, members) pairs (F4) so callers
  carry the authoritative leaf label instead of a brittle zip-alignment.
- d_leaf: partition -> RELATION. A shared pre-reveal anchor belongs to
  EVERY branch leaf's path, so the per-path total cap counts it into
  each leaf's total (byte-parity for fan-at-first/deterministic: no
  pre-reveal anchors).
- New dd_same_scenario_annuity lineage: the NPV annuity / fixed-cost
  window walks dedup branch copies of the same CALENDAR slot by anchor,
  so a shared trunk's window counts each future period once (window
  {p2035,p2040}=2), not once per branch (=3). Reconciles design
  §4.1 (capacity edd keeps the full frame with (p2035,p2040_low)) with
  §8.1 (annuity=2). Fixes the latent NPV annuity over-count that the
  shared trunk exposes. No-op for every non-shared-trunk solve.
- check_recourse_npv_preconditions obligation (i): re-scope leaf-weight
  constancy to the POST-REVEAL segment (exclude shared pre-reveal
  anchors, which carry the certain weight 1.0) per the function's own
  Slice-E note; the two-stage weighting is correct, not a violation.

Fixture stoch_two_period_hedge (RP mid-horizon 173250 / WS fan-at-first
157500) + conftest wiring.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
- test_recourse_invest_hedge.py: T-RP (173250, hedge x=60/r=60/0), T-WS
  (157500, EVPI=15750>0), T-EEV (192937.5 hand-calc, VSS>0), T-F/T-V
  (mid-horizon fan + validation, solver-free unit harness), T-L/T-E2
  (dd_same_scenario shape A, d_leaf relation, pd_non_anticipativity empty,
  no invest-NA family), T-C (per-path cap two-leaf), T-S (storage NA fires
  on (p2040,p2040_low), branch copy links to shared trunk, make_step_jump
  <-> dtttdt parity).
- hedge fixture gains a hedge_storage scenario (stochastic-group storage
  node resv) for the storage-NA / continuity assertions.
- docs/how_to.md + CHANGELOG: mid-horizon hedged two-stage reveal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Completes the E1-E3 annuity-lineage addition (functions defined and
consumed by _derived_npv; finish the public export).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The v70 stochastic_invest_method migration (Slice C) regenerated only
the 2 generate_canonical recipes; the 7 tests/fixtures and 8
canonical_databases JSONs were left at v69, failing the CI
test_fixtures/canonical_databases verify steps. Pure schema-additive
regen (new value list + param definition + version bump); no test
behavior or golden change (tests migrate JSON on load).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
sync_master_template runs migrate_database, whose sqlite engine handle
is still open when TemporaryDirectory cleanup runs -> WinError 32 on
Windows. Same check runs on the ubuntu-only template-check CI job.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
jkiviluo and others added 19 commits September 23, 2026 14:24
test_migration_reaches_v70_and_is_idempotent loaded lh2_three_region.json
from the test_fixtures corpus and asserted start_version < 70. But that
corpus is regenerated to HEAD (>= v70) by `test_fixtures migrate-all`, so
after the v70 regen it starts AT v70 and `70 < 70` fails — the two
requirements are contradictory for the same file.

Seed a minimal v69 DB from scratch (model + solve classes, version pinned
to 69) via new `_build_v69_solve_db`, mirroring the v65/v66 migration
tests, and migrate with `up_to=70`. This still exercises the v70 block
from below and verifies idempotency, while staying green as the schema
chain grows past 70. The v70 step already tolerates such minimal DBs
(creates solve_advanced if absent).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…in file

Several test files opened the checked-in `templates/examples.sqlite`
directly. Under `-n auto --dist loadfile` those files run on DIFFERENT
xdist workers, so multiple worker processes open the SAME sqlite file
concurrently. On Windows, sqlite lock contention on a shared file can
block indefinitely, and pytest-timeout (900s, `thread` method) cannot
interrupt a native lock wait, so the job hangs until GitHub's 6h cap
cancels it (observed: test_parameter_group_coverage hung ~3h48m on
windows-3.12; passed on the other matrix legs). It also violates
CLAUDE.md invariant #3 (tests must not read a checked-in .sqlite).

Add session-scoped `examples_db_path` / `examples_db_url` fixtures in
tests/conftest.py that copy the materialized template to a per-worker
temp path (tmp_path_factory's basetemp is unique per xdist worker), so
no two workers ever touch the same file. Session scope copies once per
worker and is a valid dependency for the module-scoped consumer fixture.
A copy is byte-identical, so every reader's assertions are unchanged.

Refactored readers of templates/examples.sqlite:
- test_parameter_group_coverage.py (the hang) -> fixture URL, with-close
- spinedb_backend/test_memory_budget.py -> fixture URL
- spinedb_backend/test_parsed_value_eviction.py -> fixture path in the
  module-scoped backend fixture
- test_handoff_writers.py -> thread the isolated URL into the subprocess

test_examples_e2e.py reads a different checked-in DB
(projects/examples/input_sources/examples.sqlite); it now reads a
per-test tmp_path copy for the same invariant-#3 reason.

Already-isolated readers (materialize from JSON into tmp) left unchanged:
test_export_to_tabular.py, test_xlsx_workflow.py,
engine_polars/test_param_shapes_phase_a.py.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…st, per-scenario trimmed

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Add the solve.non_anticipativity_periods parameter (Array of periods,
default unset) that decouples the operational non-anticipativity window
from realized_periods (Slice G, Axis B).

- v70->v71 migration (_migrate_v71_non_anticipativity_periods): Array
  param with default_value=None/default_type=None (no value list),
  grouped under solve_advanced. Adds the definition only, never a
  per-solve value, so every migrated solve stays on the unset ->
  legacy-window path (byte-identical).
- PARAMETER_TYPES entry (DIMENSIONLESS) per autoscale Invariant 1.
- Regenerated all four CI artifacts: spinedb_schema.json (master
  template), test-fixture corpus, canonical_databases JSON, canonical
  DBs. All four --verify steps pass; diffs are the mechanical v71 bump
  plus the new definition row.
- v71 idempotency + definition-shape test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Wire the resolved non_anticipativity_periods window (Slice G, Axis B)
end to end, single source of truth, no new CSV.

- _solve_config: tri-state reader non_anticipativity_periods_to_tristate
  keys off raw find_parameter_values presence (periods_to_tuples'
  defaultdict cannot tell unset from stored []): None=unset,
  []=free-ops, [(p,p),...]=curated. Stored in a plain dict so a missing
  key reads back None. Read after the four period dicts; propagated
  through duplicate_solve's dup_map_list (F3) for rolling/laddered
  sub-solves.
- _emit_per_solve: emit_per_solve_sets drives dt_non_anticipativity_set.csv
  from the window (F1) — None=legacy realized u fix union (byte-identical),
  []=header-only, [periods]=those periods' timesteps intersected with
  steps_in_use (the storage family's only domain guard, F7). Threaded in
  from _emit_solve_time.run (runs during preprocessing, before build).
- _derived_branch.dt_non_anticipativity_lf repointed to READ the
  authoritative set CSV; present-but-empty -> empty frame -> model's
  height==0 early-out skips all four families. Legacy union kept only as
  a defensive fallback when the CSV is absent.

Byte-parity green (unchanged goldens): test_non_anticipativity,
test_stochastic_continuation_fanout, test_derived_branch_lineage,
non_anticipativity_storage_use_emits, recourse gate/hedge/na — all under
FLEXTOOL_AUTOSCALE_STRICT=1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Two-period t0-branching fixture stoch_two_period_free_ops.json (mid/up/low
branches at t0001, shared investable base with stochastic_invest_method=
none, stochastic-group storage node resv with branch-asymmetric wind
timing -> unique per-branch storage optimum, F6). Two scenarios differ
ONLY in non_anticipativity_periods.

test_configurable_nonanticipativity.py (10 tests, all @solver):
- free_ops (non_anticipativity_periods=[]): header-only set CSV, zero
  non_anticipativity_* rows, ONE shared v_invest (no branch fan), and
  v_state[resv] strictly divergent across branches (up>0.5, low<0.5,
  mid idle); objective 757_369.4.
- pinned_ops (unset): set CSV = legacy 6-step horizon, storage NA fires
  (12 rows), v_state equal across branches (byte-parity guard);
  objective 1_920_800.
- free_obj < pinned_obj monotonicity.

conftest: stoch_two_period_free_ops_db_url session fixture + scenario_
workdir wiring (three-touchpoint pattern).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Document the new solve.non_anticipativity_periods knob in the
multi-period stochastics section: unset = legacy pinned-over-horizon
window; empty array = free operations from t0 (two-stage capacity
expansion with a shared invest); period list = curated window (e.g.
mid-horizon reveal). Notes that a curated list replaces the default
window and that realized_periods still governs the dispatch output.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The v71 solve param was missing from export_settings.yaml, failing
test_no_schema_params_missing_from_yaml. Added alongside the sibling
period-array params (realized_periods etc.).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…config-load-fan limit

Code-review follow-ups (non-blocking): a non-Array value now resolves to
the legacy window instead of silently freeing operations (G-2); docstring
now states explicit windows do not propagate to config-load-fanned
2D-Map-laddered solves (G-1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…ract)

The parameter_value_dtypes override marked solve.stochastic_branches as
Utf8 alongside the period-name params, but its map leaf is the branch
WEIGHT (a number). The comment says overrides list only genuinely
non-numeric leaves; removing it lets the weight use the Float64 default.
Dormant today (the live reader path returns floats via find_parameter_
values), but corrects the classification for any future SpineDbReader
consumer. No data migration — DB weights are already numeric.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…ons + overrides

Adds an upfront subsection stating what a dispatch solve and an
investment solve tie across branches by default (storage/online/reserve
over the realized window; flows free; single shared investment) and how
to override each (stochastic_invest_method, non_anticipativity_periods).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…he main mode

The main stochastic-investment mode is a single shared here-and-now
investment with operations free per scenario from t0 (none +
non_anticipativity_periods=[]), not the mid-horizon hedge. The hedge
is recast honestly as the variant that shares NEAR-TERM OPERATIONS too
(first periods fully deterministic). Notes the multi-period investment
non-anticipativity window (shared-first-period / recourse-later) as a
planned addition. Wait-and-see framed as a screening tool.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Add solve.non_anticipativity_invest_periods — the standard two-stage
stochastic-investment mode knob (Array of periods over which v_invest /
v_divest are tied across stochastic branches, a single shared
here-and-now decision).  Default unset == [] == NO tie, byte-identical
to today's recourse/none.

- db_migration: _migrate_v72_non_anticipativity_invest_periods mirrors
  the v71 non_anticipativity_periods migration (Array, null/null default,
  no value list, solve_advanced group); FLEXTOOL_DB_VERSION 71 -> 72 +
  dispatch arm.
- PARAMETER_TYPES row (DIMENSIONLESS) so autoscale does not silently
  degrade to an un-scaled LP.
- export_settings.yaml: split_params.solve_period.params AND
  data_type_overrides.nested_periodic (both locations, F6).
- _solve_config: read via plain periods_to_tuples (unset == [] == no
  tie, so no tri-state needed) + dup_map_list propagation for
  rolling/laddered sub-solves.
- Regenerated all four CI artifacts (spinedb_schema.json master template,
  canonical databases, test fixtures, generate_canonical); the only diff
  is the mechanical v72 version bump + the new parameter-definition row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Populate pd_non_anticipativity by window membership and tie invest AND
divest across branches over that window (design §3-§5).

- _emit_per_solve: emit solve_data/pd_non_anticipativity_invest_periods.csv
  ONLY when the window is non-empty (absent file -> legacy empty frame ->
  byte-parity, no new CSV for unset fixtures); threaded from
  _emit_solve_time.run.
- _derived_branch.pd_non_anticipativity_lf: new tie_periods arg + ctx/CSV
  reader (_read_invest_tie_periods).  When resolved, emit
  (anchor_of[m], m) for every synthetic member whose anchor is in the
  window (supersedes the never-supplied branch_start reveal-boundary gate,
  which stays as the byte-parity legacy path -> provably empty today).
- model.py: _add_non_anticipativity_invest_constraints emits all four
  families non_anticipativity_invest_p/n and _divest_p/n (divest is
  mandatory, F2/§4.4 — tying invest alone leaks net first-stage capacity).
  Gated on recourse_invest + non-empty frame.  The cross-Enum domain
  guard carries the db_pairs Utf8 dance (F3): pd_non_anticipativity.b is
  branch-Enum while pd_invest_set.d renamed d->b is period-Enum, so both b
  keys are cast to Utf8 for the guard join and restored via cast_dim.
- autoscale: register all four family names (CstrFamily(None), RHS=0) +
  pin them in test_registry_coverage dynamic samples (passed as a
  variable, invisible to the literal grep).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Extend the hedge builder with fan-at-t0 invest-NA scenarios and add the
solver/structural gate (design §9).

- build_stoch_two_period_hedge: new scenarios on the fan-at-t0 (ws)
  topology — hedge_na_window (window=[p2035] + ops free), hedge_na_divest
  (base invest_retire_total + existing 50 + salvage_value so v_divest
  exists), hedge_na_none (stochastic_invest_method=none).
- test_invest_na_window: T-Main obj=173,250 with p2035 v_invest EQUAL
  across branches (60) and p2040 per-branch (recourse 60/0); T-WS control
  157,500 with RP−WS=15,750; T-Frame {(p2035,p2035_low)} (empty when
  unset); T-Weight branch-weight partition 0.25/0.75; T-C invest rows;
  T-Divest divest tied + net capacity equal across branches; T-None
  graceful no-op; T-Reg four families registered.  All under
  FLEXTOOL_AUTOSCALE_STRICT=1 (T-Main is the first coverage of the F3
  cross-Enum guard join).
- model.py: T-None advisory log — window set on a non-recourse solve is a
  graceful no-op (never mutates the problem, §4.5).
- test_v72 migration: idempotency + definition shape (Array, null/null
  default, solve_advanced group), mirroring the v71 test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Replace the "planned addition" note in how_to.md with the delivered
three-knob recipe (recourse + non_anticipativity_invest_periods=[first]
+ non_anticipativity_periods=[]) and add the CHANGELOG entry for the
standard two-stage stochastic-investment mode.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Code-review nit: _read_invest_tie_periods had a speculative ctx-first
branch that nothing populates and that would silently iterate
solve-name keys if a future caller wired the {solve: [...]} config.
The emitted CSV is authoritative; removed the branch, documented why
ctx is not a source.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant