Skip to content

Slice E: hedged two-stage stochastic investment (mid-horizon reveal) - #336

Open
jkiviluo wants to merge 32 commits into
mainfrom
feat/mid-horizon-hedge
Open

Slice E: hedged two-stage stochastic investment (mid-horizon reveal)#336
jkiviluo wants to merge 32 commits into
mainfrom
feat/mid-horizon-hedge

Conversation

@jkiviluo

@jkiviluo jkiviluo commented Sep 23, 2026

Copy link
Copy Markdown
Member

Stacked on #335 (Slice D) → #334 → #333. This is the slice that delivers genuinely non-anticipative (hedged) two-stage stochastic investment — the original goal of the program.

What it adds

Slices A–D shipped per-scenario (wait-and-see) investment: with the fan always starting at the solve's first step, every investment decision belongs to one scenario, so there is no shared first stage. Slice E enables a mid-horizon reveal: pre-reveal periods stay a single shared real-named trunk (the here-and-now first-stage decision), and branches fan out per scenario only after the reveal (recourse). The shared trunk investment is seen as existing capacity by every branch's post-reveal dispatch through the lineage machinery Slices A–D built.

Key design outcome: no new constraint families

Non-anticipativity holds by construction — the shared pre-reveal period is a single variable, so there is nothing to tie. The plan had anticipated new non_anticipativity_invest_* constraint families; a two-round adversarial review confirmed they are unnecessary under this "shared trunk" convention. Slice E is therefore a small, low-risk slice: a fan-out enablement, a per-path-cap d_leaf partition→relation fix (a shared pre-reveal period counts into every scenario's path), a storage-continuity linkage fix, and a separate calendar-deduplicated lineage frame for NPV annuity windows.

The hedge is real (independently verified)

On a mid-horizon fixture the hedged objective 173,250 sits strictly between the wait-and-see lower bound 157,500 and the deterministic here-and-now bound 192,937.5 — so EVPI = 15,750 and VSS = 19,687.5 are both positive. The gate would collapse to 157,500 if the shared first stage were broken, so it is a genuine behavioral proof.

A latent bug fixed along the way

The shared trunk exposed a real NPV annuity over-count: the capacity lineage deliberately contains both (p2035, p2040) and (p2035, p2040_low) so the shared invest reaches the branch's dispatch — but feeding that to the annuity window walk would count the trunk's operational window over both branch copies of the same calendar period (window 3 instead of 2). A separate dd_same_scenario_annuity frame deduplicates branch copies by calendar anchor for the annuity walk. Adversarially verified as necessary, correct (2-period, 3-period, two-branch cases), and a true no-op for every existing (non-shared-trunk) model.

Verification

  • Hedging hand-calc gate hit exactly (173,250), with wait-and-see and deterministic controls.
  • Non-anticipativity-by-construction pinned structurally (empty pd_non_anticipativity, single shared trunk variable).
  • Byte-parity: existing recourse gate (196,875), storage-bearing fixtures, and all fan-at-first-step/deterministic shapes unchanged.
  • Storage-linkage guard corrected during review so it preserves existing storage goldens (indexes off the branch's scenario anchor, not its position).
  • Full sweep: 3761 passed, 0 failed, zero golden changes.

🤖 Generated with Claude Code

https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ

jkiviluo and others added 30 commits September 22, 2026 16:01
@jkiviluo @claude
feat(stochastic): Slice A scenario-lineage frames (dd_same_scenario, …
f9d3dd6 
…pd_non_anticipativity)

Two always-on derived frames in _derived_branch.py, consumed by
nothing (recourse plan §6b Slice A; design
specs/sliceA_lineage_frames_design.md r3):

- dd_same_scenario (d, d_other): symmetric/reflexive pairs sharing a
  scenario leaf-path; domain = period_in_use only; all-pairs
  degenerate content for deterministic solves; leaf rule with the F6
  authority convention (branch-local member displaces the anchor).
- pd_non_anticipativity (d, b): period-level invest-NA pairs
  (strictly-pre-reveal anchor ties), parameterized on an optional
  branch_start frame (Slice E hook); provably empty for every
  currently-constructible solve under the min-anchor fallback.

Wiring: unconditional assignment in apply_branch_cluster (step 2b),
guarded-ctx input acquisition per the dtttdt_from_source null-guard
precedent, ValueError failure semantics on inconsistent per-solve
artefacts, Utf8-construct -> non-strict cast_dim -> live null
assertion -> deterministic Utf8-key sort.  FlexData registration
(two typed fields, None only pre-cascade), d_other axis synonym.
No consumers, no schema change, no migration, no autoscale entries
(nothing enters the LP), no dump-side change (_copy_meta already
glob-copies the rebuild inputs).

Tests: T1-T12 per design §4/§5 (17 tests) — five exact shape tables,
both mid-horizon shapes incl. the branch_start-driven [(p1, p1_low)]
pair, symmetry property, metadata exclusion, failure semantics,
_drop_master_rows pass-through, integration + enum hygiene on the
stoch_two_period fixture, and the first stochastic dump/reload
round-trip (pins the _copy_meta glob).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): walker lineage capability, inert
19407fd 
Slice B of the recourse-invest plan (specs/sliceB_walker_lineage_design.md).
Add a keyword-only lineage=pl.DataFrame|None=None parameter to the shared
period_walk_iterator, applied PRE-aggregation under the Slice A semi-join
contract (remove (e,d,d_all) iff both ends in period_in_use and (d,d_all)
not in lineage; history-anchored rows pass unconditionally).

- _derived_walks: _apply_lineage_filter (anchor_col/dall_col params,
  fixed-direction lineage-side non-strict cast, mandatory .unique(),
  marker left-join) + _assert_lineage_castable null guard; zero-cost
  None path behind a single if.
- Thread lineage=None through inventory #1-17 (cohort/variant/from_source
  chain in _derived_existing, _derived_npv, _derived_params) and apply
  the shared helper in the one non-walker pair-former #18
  edd_divest_active_from_source (anchor_col="d_divest", dall_col="d").
- _derived_branch: check_recourse_npv_preconditions +
  assert_recourse_npv_preconditions — the three named NPV proof
  obligations (leaf-weight constancy, cohort sum-to-1, byte-equal
  fan-member year rows), shipped as a pure checker with the F4
  soundness-scope docstring; added to __all__.

Inert: every production call passes lineage=None; no flag, no consumer
activation, no schema change, no autoscale entries, no LP change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
test(stochastic): walker lineage capability + NPV obligation suite
dd20b5e 
Slice B test suite (specs/sliceB_walker_lineage_design.md §9).

Capability harness (solver-free, hand-built frames + stub source):
- W1 set-shape (history passes, cross-leaf both-ends-PIU pairs removed)
- W2 factor-side PRE-aggregation proof (sums 7.0/3.0/4.0 vs unfiltered
  7.0 — a post-aggregation filter cannot produce 3.0/4.0)
- W3 lineage=None 12-row unfiltered pin
- W4/W4b empty-frame contract (set: 4 history rows; factor: single
  (h2020,7.0) row, anchors absent not zeroed)
- W5a duplicate-pair .unique() guard; W5b uncastable-token ValueError;
  W5c Enum x Enum missing-token raises / all-present identical

Stochastic fixture pins (solver-required fixture, solver-free asserts):
- W6 edd_invest_set today-bug witness ((wind,p2040,p2040_low) absent =
  erratum E1 first-stage-visibility break)
- W7 edd/divest non-vacuity + repeat-determinism; NPV variant vacuity
  pins + invest-method overlay leg
- P1/P2 obligation checker clean on stoch/deterministic workdirs
- P3a-d hand-seeded obligation violations (i)/(ii)/(iii)
- P4 walker year-source blindness diagnostic (erratum E1 witness pair)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
fix(stochastic): keep obligation checker I/O out of the cascade
91db38b 
The Slice B NPV obligation checker's year-row reader
(`_read_solve_data_csv`) fell back to a bare `pl.read_csv(...)` on the
workdir CSV when the Provider didn't carry the frame. That disk arm
tripped the post-Step-2 cascade invariant
(test_meta_provider_invariants::test_no_disk_csv_reads_in_cascade,
Rule 1) — the seed-funnel bug class the invariant guards.

Make the reader Provider-only (`_provider_get` -> `provider.get`),
matching every other reader in `_derived_branch.py`. The checker now
does no disk I/O; the caller owns placing the true emitted year CSVs
on the Provider (tests seed via seed_provider_from_dir / _mk_provider;
the Slice D activation site inherits the same solve_data frames the
rest of the cascade already consumes). Obligation (iii) semantics
preserved. SOUNDNESS SCOPE docstring + design (local) sec 6.2 updated
with the deviation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): schema opt-in stochastic_invest_method (v70)
d15e31e 
Add the opt-in solve.stochastic_invest_method enum (value list
stochastic_invest_methods: none/recourse, default none) via migration
v70.  'none' is byte-identical to prior behaviour; 'recourse' is
reserved for Slice D and hard-rejected by the Slice C guards.

- FLEXTOOL_DB_VERSION 69 -> 70; _migrate_v70_stochastic_invest_method
  mirrors the v64 solve.scaling exemplar (value list + definition +
  guarded solve_advanced grouping).
- PARAMETER_TYPES row ('stochastic_invest_method','solve') DIMENSIONLESS.
- _solve_config read/store + stochastic_invest_method_for resolver
  (absent -> 'none').
- Regenerated spinedb_schema.json master template + the two managed
  canonical DBs (templates_examples, howto_stochastics).
- export_settings.yaml split_params: surface the new solve knob.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): recourse guards
af6ebee 
Three flag-independent / flag-keyed validation guards that keep the
recourse opt-in inert until Slice D lands.

- Guard 1 (flag hard-reject): stochastic_invest_method != none raises
  FlexToolConfigError at the top of _orchestration.run() (authoritative,
  covers rolling/contained solves via the roll-suffix strip) plus a
  fail-fast pass in _validate_model_solve over the top-level solves.
- Guard 2 (Benders x genuine stochastics): _is_genuinely_stochastic
  raises in _run_benders_solve (primary) and at the top of solve_benders
  (defensive).  Detector = period_branch_full d!=b b-tokens intersected
  with period_in_use_set, so the rolling realized-only bookkeeping shape
  does NOT false-positive.
- Guard 3 (handoff synthetic-name assertion): build_handoff_from_solution
  rejects any realized_invest / committed period that is a synthetic
  branch fan member; None-degrades when flex_data is unavailable.

New shared cycle-free module _stochastic_detect.py holds the single
token-extraction idiom (synthetic_branch_tokens) used by Guards 2 and 3
and the PIU-intersected is_genuinely_stochastic used by Guard 2.  Both
cast d/b to Utf8 before comparing — at runtime period_branch_full is
Enum-typed with disjoint d/b category sets, and comparing two enums
raises in polars (would otherwise crash Guard 3 on every stochastic
handoff); null b (deterministic shape) is dropped.

Empirically verified on stoch_two_period: genuine stochastic -> True
(the _rlz metadata members are excluded by the PIU intersection, the
_low branches survive); deterministic + rolling -> False.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
test(stochastic): v70 migration + recourse guard tests
d934ef0 
- test_v70_stochastic_invest_method.py (M1-M3): migration round-trip
  (definition present, default none, bound to stochastic_invest_methods,
  grouped solve_advanced; value list {none,recourse}); pre-v70 -> v70
  reach + idempotent re-run; master-template --verify up to date.
- test_recourse_guards.py (G1, G2a-c, G3): resolver normalisation +
  _validate_model_solve reject/pass + rolling-suffix-strip composition
  (G1); PIU-intersected detector on genuine/deterministic/self-row/
  rolling-realized-only shapes incl. the Enum disjoint-category
  regression pin and solve_benders defensive raise (G2); solver-based
  handoff clean-path / config-level synthetic raise / None-degrade (G3).

Detector shapes mirror the empirically verified stoch_two_period
runtime frames (Enum-typed d/b, PIU excludes _rlz metadata members).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): D0 recourse-active plumbing (inert)
abc188c 
Emit solve_data/stochastic_invest_method.csv (resolved per-solve mode)
next to the realized-invest emit; _recourse_invest_active reader
(provider-first, workdir fallback, absent -> False); FlexData.recourse_invest
bool stamped in apply_derived_c; _has_branch_invest_axis capability conjunct
reusing is_genuinely_stochastic. Dump-registry + accumulator-manifest lines
so a dump/reload round-trip preserves the flag. Flag defaults none -> no
behavior change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): D1 beta narrowing — LineageFilterError
95d1411 
Define LineageFilterError(ValueError) in _solve_state; the two Slice B
lineage guards (_assert_lineage_castable, assert_recourse_npv_preconditions)
raise it. Narrow the four blanket except-Exception swallows in
apply_derived_c / apply_synthetic_invest_sets (edd lookback / edd_invest /
edd_divest_active / synthetic edd) to re-raise LineageFilterError so a
lineage failure is loud instead of silently reverting to unfiltered
first-stage sets. Raise path is unreachable while lineage is None
(flag-off byte-parity).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): D2 invest axis + per-period anchor-mapping
c318bb3 
(A) _expand_invest_branch_periods — invest-side mirror of the dispatch
_expand_branch_periods; wraps the 5 axis-defining _solve_periods(invest_periods)
sites (ed_invest_set, ed_divest_set, ed_invest_forbidden, eager
ed_entity_annual_family, apply_npv) so period_invest gains {p_k, p_k_b}
under recourse. ctx/provider threaded through those readers + their
apply_derived_c / apply_synthetic callers. period_first + PIU-fallback
sites left unwrapped (design §3.1).

(B) _anchor_expand_explicit + module-scoped anchor-pairs holder set at each
Layer-4 boundary (apply_derived_c / apply_npv / apply_synthetic): explicit
per-period rows (invest_cost/discount/lifetime/fixed_cost Maps + per-period
caps) inherit their anchor's value onto branch periods, wired into the three
producers (_resolve_per_period_lf, _per_entity_period_cost,
_ed_explicit_period_param). Holder None flag-off/deterministic -> no-op ->
byte-parity.

Flag-off spot-suites (fanout, npv_cluster, lineage) green; new axis unit
tests cover expansion + anchor inheritance.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): D3 alpha year/factor provider threading
8aef021 
Thread provider through period_walk_iterator into _p_years_d_lf (year side)
and _inflation_factors_lf -> _years_for_period_lf -> _years_for_period_from_source
(factor side). Add a provider-gated canonical arm to _years_for_period_from_source
reading the emitted p_years_represented.csv (which byte-copies fan-member
year rows via derive_years_represented's branch loop). Under recourse the
caller (D4 apply boundaries) forwards provider so branch periods resolve
their anchor's year-from-start -> correct annuity windows; provider=None
(flag-off) keeps the canonical arms dead -> today's fill_null behaviour
(the W6 today-bug pin) -> byte-parity. Plumbing only this phase (no caller
forwards provider yet); activation lands with lineage at D4.

Flag-off npv_cluster/walk/fanout suites green; new canonical-arm unit tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): D4 lineage activation + NPV preconditions
4e39d2c 
Compute the scenario-lineage frame (flex_data.dd_same_scenario) at the
three Layer-4 apply boundaries, gated on recourse AND a genuine branch
invest axis (capability conjunct). Forward lineage=... into the edd
builders threaded by Slice B: edd_invest_lookback_set, edd_invest_set_lf,
edd_divest_active (#18), the synthetic edd, and apply_npv's four
ed_*_from_source walks. Hoist assert_recourse_npv_preconditions above the
edd try/except blocks so a violation raises LineageFilterError (loud, D1).

alpha-1 walker Provider: a boundary-scoped module-global (_RECOURSE_WALK_PROVIDER)
armed alongside the anchor-pairs scope; period_walk_iterator falls back to
it so the canonical year/factor arms revive under the flag without threading
provider through ~12 NPV/edd caller signatures. None flag-off -> arms dead
-> W6 today-bug preserved.

Activation is inert flag-off (lineage None) -> byte-parity; 103-test
spot-suite (npv_cluster/fanout/lineage/axis/year) green. Flag-on edd
cross-leaf integration lands at D5 (Guard 1 lift + fixture).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): D5 objective weighting + Guard-1 lift + hand-calc gate
51bf2d4 
Objective: multiply ALL EIGHT invest/divest cost Params (process
annu/lf/lfd/annd + node annu_n/lf_n/lfd_n/annd_n) by pd_branch_weight[d]
via the _wt closure, gated on d.recourse_invest AND pd_branch_weight;
replace the .mod "NOT weighted" comment with the recourse rationale.
Flag-off skips the multiply (byte-parity).

Guard 1 LIFTED for 'recourse': _stochastic_invest_raw_guard validates the
RAW authored value (resolver collapses typos to 'none' so the guard must
read the raw dict); both sites (run() + _validate_model_solve) switched;
guard tests flipped per design §15.4 + NEW raw-typo rejection test.

Activation fixes found by the gate (recorded in design implementation
notes): (a) precondition checker needs the REAL workdir (Provider-only
readers key paths off it — design §8 snippet's None arg made obligations
fail spuriously); (b) lineage built ON-DEMAND per boundary via
dd_same_scenario_df (flex_data.dd_same_scenario populates only at
derived_g); (c) alpha-1 walker holders gain the workdir twin
(_RECOURSE_WALK_WORKDIR) so the canonical p_years_d arm goes live;
(d) _recourse_usable_ctx sanitizes the snapshot-reload SolveContext
(all-null d_anchor artifact) with provider-arm fall-through.

Gate fixture stoch_two_period_invest.json (+ builder, + conftest wiring):
supply-side branch variation (deterministic demand 100/130 + wind
upper_limit profile rlz .4/.4 low .2/.1 via group__unit -> peaker residual
60/90 / 80/120) because node-inflow branch resolution gates on group__node
and a stoch-group node would fire NA net-charge pinning.

D5 gate: objective 196875 EXACT; coefficients 525/262.5/1575/787.5;
per-leaf invest 60/80/30/40; committed output realized-only; edd
cross-leaf ABSENT (W6 re-pointed); pd_non_anticipativity empty; STRICT
autoscale clean. Flag-off parity: 128 tests green (lineage/walk/fanout/
guards/npv_cluster + recourse unit suites). ruff clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): D6 per-path caps
b2d0039 
Per-scenario-path total-capacity caps so a total cap applies once per
scenario leaf under recourse, never cross-scenario double-counting
(design sliceD §7 (G) / §14 (L)).

- d_leaf (d, leaf) frame in _derived_branch.py: partitions period_in_use
  into scenario leaves (real anchors -> "__realized", synthetic members
  -> time-branch id); registered on FlexData, built in apply_branch_cluster.
- model.py _emit_entity_total_cap: the four maxInvest/maxDivest entity
  total builders emit the legacy over=("d",) shape + name for a single
  leaf (deterministic / flag-off, byte-identical) and …_path indexed by
  (entity, leaf) for >1 leaf, masking the inner sum to each leaf.
- _cumulative_invest.py _emit_group_total_divest_cap: the divest-side
  group total gains the same per-path treatment (max/<= cap only).
- autoscale registry: maxInvest/maxDivest_entity_total_path(_n) (POWER/
  ENERGY) + maxDivestGroup_entity_total_path (group_capacity resolver);
  test_registry_coverage dynamic-name pins.
- Fixture invest_max_total tightened 250 -> 150 (§15.1): per-leaf totals
  90/120 non-binding, legacy all-d sum 210 would have bound — the
  asymmetry proves per-path. Gate test asserts two leaf rows.

Gate: objective 196,875 exact; STRICT autoscale clean; flag-off parity
(derived_branch/period_walk lineage, continuation fanout, recourse
guards, deterministic invest cumulative/total goldens) green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): D7 handoff explicit non-realized drop
03f19c0 
Make the recourse handoff's drop of branch-named (non-realized-scenario)
v_invest rows EXPLICIT (design sliceD §10.1 (H)).

- build_handoff_from_solution: partition invest_by_ed on synthetic
  branch-token membership BEFORE the commit loop and debug-log the
  dropped mass; the commit gate previously dropped these by accident.
  The Slice C Guard 3 assertion stays as a belt-and-braces invariant
  check (no synthetic token may survive into committed rows), not the
  mechanism.  Flag-off / deterministic: synthetic set empty → no-op
  (byte-parity).
- Gate test: the pre-filter drops the low leaf's 80+40 = 120 MW, debug-
  logs it (periods p2035_low/p2040_low), and commits only the realized
  60/30; Guard 3 stays inert.

Guard 1 raw-value lift was already landed at D5.  Flag-off parity
(handoff cumulative carriers, continuation fanout, recourse guards) green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): D8 outputs/costs + fixture variant B
f7b724d 
Outputs, cost reconciliation, and the cross-scenario-coupling fixture
(design sliceD §11 (I) / §15.5 / §15.6).

Outputs (§11.2):
- read_highs_solution: under recourse AND output_horizon, union the
  horizon-expanded d_realize_dispatch_or_invest set (branch periods
  included) into the invest-axis canonical order, so non-realized
  branch v_invest rows surface.  Gated on both flags → byte-parity for
  deterministic models and for recourse without horizon output.
  Committed path stays realized-only (§11.1).

Costs (§11.3, choice I-a):
- calc_costs probability-weights the invest/divest + lifetime-fixed cost
  rows by pd_branch_weight under recourse, so the cost breakdown
  reconciles to the LP objective's (expected) invest contribution rather
  than over-reporting the realized spend by 1/w_realized.  read_parameters
  threads par.pd_branch_weight + par.recourse_invest.  Gated → factor 1.0
  off-flag (byte-parity; deterministic cost-aggregation goldens green).
- test_npv_cluster skips the eager-vs-lazy NPV byte-equality under
  recourse (the eager reference is not lineage-aware, §11.4).

Fixtures/tests:
- variant A gains a recourse_horizon scenario (model.output_horizon=yes);
  gate test asserts branch rows 80@p2035_low / 40@p2040_low surface.
- variant B (stoch_two_period_invest_na): extends the proven
  stochastics.json storage shape with a recourse_na scenario making the
  stochastic-group storage node investable.  Witnesses:
  non_anticipativity_storage_use fires (>0 rows), branch v_state == the
  NA-tied realized value, and the node-side annu_n objective coefficient
  carries pd_branch_weight (the F3 node-side pin).
- cost reconciliation e2e: reported realized invest cost 0.039375 M =
  0.25·(60·2100 + 30·1050); solver objective 0.196875 M.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
docs(stochastic): D9 per-scenario invest recipe + changelog
edb158f 
- how_to.md: replace the "only one period investment stochastic models
  are supported" line with the per-scenario (wait-and-see) recourse
  recipe — what the objective means (expected value of per-scenario
  plans, EVPI-style; not a hedged plan), the coupling change vs the
  default, the α-1 cross-comparability caveat, and the Benders exclusion
  (design sliceD §16).
- reference.md: document solve.stochastic_invest_method (none/recourse),
  the output_horizon interaction, and the Benders exclusion.
- CHANGELOG: Unreleased draft entry (public wording, no internal model
  names; schema change note).  No version bump — release is the owner's
  decision.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
test(stochastic): mark recourse cost reconciliation as solver
28e35b0 
The end-to-end cost-reconciliation test invokes a real solve; add the
``solver`` marker so it deselects under ``-m "not solver"`` like the
other recourse solver tests (it already carried ``slow``).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
fix(stochastic): read recourse flag via provider, not disk (cascade i…
f0ea27d 
…nvariant)

D0 introduced _recourse_invest_active with a provider-first / workdir-disk
fallback read of solve_data/stochastic_invest_method.csv.  The pl.read_csv
disk arm tripped test_meta_provider_invariants::test_no_disk_csv_reads_in_cascade
(Rule 1) — the same seed-funnel bug class fixed for the Slice B obligation
checker in 91db38b.

Make the reader Provider-only (_provider_has_key -> _provider_read), matching
every other reader in this module (_build_recourse_anchor_pairs et al.).  The
flag CSV reaches the cascade exclusively through the Provider on all
production paths: cascade entry points thread an explicit Provider, and the
snapshot-reload path (load_flextool off a dumped workdir) seeds the Provider
from solve_data/ via seed_provider_from_dir before the cascade runs, so it
carries the solve_data/stochastic_invest_method key by construction.  The
recourse gate is unchanged (returns True under flag='recourse' on both the
normal and reload paths).

Update test_recourse_plumbing to seed a Provider from the workdir (mirroring
the cascade) instead of relying on the removed disk arm; add a no-Provider
case asserting the byte-parity-safe default.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
fix(migration): create solve_advanced group if absent in v70 migration
c691612 
The v70 migration (_migrate_v70_stochastic_invest_method) attached
solve.stochastic_invest_method to the solve_advanced parameter_group only
"when that group exists", but probed existence with the raising
db.item(db.mapped_table("parameter_group"), name="solve_advanced"). In the
current spinedb-api, db.item RAISES SpineDBAPIError when the item is absent
(it does not return None), so the `is not None` guard never helped — it
crashed before returning.

solve_advanced is created by the v44 migration (recoloured in v45), so real
DBs migrated from < v44 and the shipped template always carry it. But a DB
built from scratch at a version above v44 never ran the v44 step and has no
parameter_groups at all — e.g. the minimal v64 fixtures that
tests/test_v65_migration.py and tests/test_v66_migration.py seed and then
migrate forward through v70. Those two tests crashed on the v70 step
(regression vs main, where they pass).

Fix: use the non-raising db.get_item guard and get-or-create the group —
create solve_advanced (mirroring v44's priority 87 and the post-v45 colour
b56f6f) only when the DB lacks it, then always attach the param. This keeps
the param in its canonical schema home and satisfies the v69 invariant that
every parameter belongs to a group; it is a clean forward add, not
crash-tolerance. The v70 fixture (lh2_three_region) already has the group,
so the migrate-from-v69 path is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
merge: bring v70 migration fix + B/C into recourse-invest
d2837a1 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
fix(stochastic): F1 defensive per-solve reset of recourse globals
4e86d64 
The three recourse process-globals (_RECOURSE_ANCHOR_PAIRS,
_RECOURSE_WALK_PROVIDER, _RECOURSE_WALK_WORKDIR) were reset only inside
the three flag-aware Layer-4 boundaries. Add _reset_recourse_scope()
called once at the top of _apply_db_overrides (the single per-solve
cascade entry, before pass 1a) so a future per-period producer wired
into an earlier cascade step cannot inherit a stale global from a prior
flag-on solve in a chained flag-on->flag-off multi-solve. The boundaries
still overwrite unconditionally; this is belt-and-suspenders.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
fix(stochastic): F2 hard-raise on nulled token in _anchor_expand_expl…
3455dce 
…icit

The final cast(d_dtype, strict=False) could silently null an out-of-vocab
token, contradicting the docstring's claim that the round-trip "cannot
silently null a token." Mirror the sibling _derived_branch.d_leaf_lf: on
the non-Utf8 (Enum) path, collect and hard-raise on any nulled token
instead of dropping it. Cannot fire in valid runs -- anchor tokens arrive
already typed as d_dtype (lossless Utf8->Enum round-trip) and branch (br)
tokens are covered by the 4.0.4 vocabulary splice. Utf8-d callers (both
production call sites) skip the collect entirely, so no hot-path
laziness/perf regression. Docstring corrected to match.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): E0 mid-horizon fan-out + storage linkage
289c462 
Enable create_stochastic_periods to trigger the branch fan at a period
k>1 (mid-horizon reveal) while keeping pre-reveal periods as a single
real-named trunk (Option B, non-anticipativity by construction).

- _stochastic.py: replace the 'realized row at the solve first step'
  validation with a well-formedness check (period-boundary reveal +
  exactly-one-realized per branching period), admitting a boundary
  mid-horizon reveal byte-identically for fan-at-first-step.
- _timeline.py make_step_jump + _derived_params.py dtttdt_from_source:
  a branch copy with no earlier same-time-branch period now links its
  first step to the nearest anchor STRICTLY BEFORE the copy's own anchor
  (shared pre-reveal trunk end-state) instead of self-cycling. First-
  period fan self-cycles unchanged (empty anchor range) -> byte-parity.

Byte-parity gate green (recourse 196875, storage *_na goldens unmoved).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): E1-E3 shared-trunk lineage, per-path caps, annuity …
daa5d93 
…dedup

Option B correctness for the shared pre-reveal trunk:

- _lineage_leaves now returns (label, members) pairs (F4) so callers
  carry the authoritative leaf label instead of a brittle zip-alignment.
- d_leaf: partition -> RELATION. A shared pre-reveal anchor belongs to
  EVERY branch leaf's path, so the per-path total cap counts it into
  each leaf's total (byte-parity for fan-at-first/deterministic: no
  pre-reveal anchors).
- New dd_same_scenario_annuity lineage: the NPV annuity / fixed-cost
  window walks dedup branch copies of the same CALENDAR slot by anchor,
  so a shared trunk's window counts each future period once (window
  {p2035,p2040}=2), not once per branch (=3). Reconciles design
  §4.1 (capacity edd keeps the full frame with (p2035,p2040_low)) with
  §8.1 (annuity=2). Fixes the latent NPV annuity over-count that the
  shared trunk exposes. No-op for every non-shared-trunk solve.
- check_recourse_npv_preconditions obligation (i): re-scope leaf-weight
  constancy to the POST-REVEAL segment (exclude shared pre-reveal
  anchors, which carry the certain weight 1.0) per the function's own
  Slice-E note; the two-stage weighting is correct, not a violation.

Fixture stoch_two_period_hedge (RP mid-horizon 173250 / WS fan-at-first
157500) + conftest wiring.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
test(stochastic): E4 hedging gate (173250) + storage scenario + docs
1f6f26b 
- test_recourse_invest_hedge.py: T-RP (173250, hedge x=60/r=60/0), T-WS
  (157500, EVPI=15750>0), T-EEV (192937.5 hand-calc, VSS>0), T-F/T-V
  (mid-horizon fan + validation, solver-free unit harness), T-L/T-E2
  (dd_same_scenario shape A, d_leaf relation, pd_non_anticipativity empty,
  no invest-NA family), T-C (per-path cap two-leaf), T-S (storage NA fires
  on (p2040,p2040_low), branch copy links to shared trunk, make_step_jump
  <-> dtttdt parity).
- hedge fixture gains a hedge_storage scenario (stochastic-group storage
  node resv) for the storage-NA / continuity assertions.
- docs/how_to.md + CHANGELOG: mid-horizon hedged two-stage reveal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
feat(stochastic): export dd_same_scenario_annuity in __all__
81ca8cf 
Completes the E1-E3 annuity-lineage addition (functions defined and
consumed by _derived_npv; finish the public export).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
test(stochastic): annuity-lineage dedup frame-content tests + docstri…
55ad9cf 
…ng fix

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
fix(schema): migrate test fixtures + canonical DBs to v70
523617c 
The v70 stochastic_invest_method migration (Slice C) regenerated only
the 2 generate_canonical recipes; the 7 tests/fixtures and 8
canonical_databases JSONs were left at v69, failing the CI
test_fixtures/canonical_databases verify steps. Pure schema-additive
regen (new value list + param definition + version bump); no test
behavior or golden change (tests migrate JSON on load).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
fix(test): skip v70 master-template test on Windows (sqlite handle lock)
9322405 
sync_master_template runs migrate_database, whose sqlite engine handle
is still open when TemporaryDirectory cleanup runs -> WinError 32 on
Windows. Same check runs on the ubuntu-only template-check CI job.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo @claude
fix(test): decouple v70 idempotency test from auto-migrated corpus
fd76f5e 
test_migration_reaches_v70_and_is_idempotent loaded lh2_three_region.json
from the test_fixtures corpus and asserted start_version < 70. But that
corpus is regenerated to HEAD (>= v70) by `test_fixtures migrate-all`, so
after the v70 regen it starts AT v70 and `70 < 70` fails — the two
requirements are contradictory for the same file.

Seed a minimal v69 DB from scratch (model + solve classes, version pinned
to 69) via new `_build_v69_solve_db`, mirroring the v65/v66 migration
tests, and migrate with `up_to=70`. This still exercises the v70 block
from below and verifies idempotency, while staying green as the schema
chain grows past 70. The v70 step already tolerates such minimal DBs
(creates solve_advanced if absent).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
@jkiviluo jkiviluo mentioned this pull request Sep 25, 2026
Open
@jkiviluo @claude
docs(stochastics): reorder how-to — basics/modes first, investment la…
f029ae0 
…st, per-scenario trimmed

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jkiviluo