[Metadata] Computed features, reference objective, shared initial guess

Agents: read the guide for agents in #310 first (order, difficulty, how to claim an issue, working method). The specification is in this description and in the comments below. This description was updated on 2026-11-06, after #360 and #375. Where an older comment contradicts it, the description wins: see the update comment.

Goal

Separate what is computed about a problem from what is declared in its metadata, add the declared fields the tests, the crons and the benchmarks need, and make the metadata the single source of the initial guess shared by every modeling (#151).

Part of #310, step 5. Depends on #283 (continuous problems, Problem) and #303 (statuses). Used by #304 (comparison rule), #305 (crons), #307 (documentation pages).

Work: sub-issues

Part	Issue	State
1. Reference objectives	#375	done (#380)
2. Features computed from the model	#376	done (#392)
3. Shared initial guess in the metadata (with #151)	#377	done (#395)
4. Robustness of the initial guess, known local objectives	#378	done (#393; references regenerated in #394)
5. Declared fields: analytic solution and objective; sources (tags and Corleone dropped)	#379	in review (#403)

How the tests use the metadata (since #360)

Three kinds of tests. Convergence is not a binary criterion any more.

Does not break. Every CI run, short: every problem in every configuration, on a small grid with a few iterations. It fails on an exception or a non-finite objective. Convergence is not checked.
Model equivalence. Every CI run, no solve: the modelings of a problem define the same problem (at the initial guess, and the NLP equivalence of [Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284). It fails when they differ. This is the important comparison between modelings.
Convergence. In the weekly cron, or on demand (argument convergence, label test convergence). Each problem is solved on its default grid. The report gives the solver status and the relative gap of the objective to the reference objective of the problem, for the grid and the scheme of the solve. It never fails: there is no tolerance. The cron flags what got worse since the previous run (issue cron-convergence).

So a metadata field is never a test threshold. References and known solutions say how far a solve is from them; statuses (:valid, :draft) say whether a model is finished and equivalent, not whether it converges.

Current state

ext/MetaData/<p>.jl holds :grid_size, :parameters, :status, optionally :notes. The schema METADATA_SCHEMA (src/OptimalControlProblems.jl: required, optional, generated) is checked at load time and by test/suite/core/test_consistency.jl. An unknown field fails, so a new field must be added there.
:reference_objective is a generated field ([Metadata] Reference objectives per scheme, computed by a script #375). See below.
The documentation browser computes dimensions, cost kind (Mayer / Lagrange / Bolza), fixed or free final time and constraint counts from the ocp (docs/browser.jl, collect_problem_data).
Initial guesses are written in each model file. In ext/OptimalControlModels/ they are mostly constants, some functions of time. JuMP models use constant start= values.

Computed, never written by hand (#376)

A function (e.g. features(:beam)) computes, when an extension is loaded: dimensions, cost kind, fixed or free initial and final times, kinds and counts of constraints (box on state / control / variable, nonlinear path, boundary). From the ocp (OptimalControl modelings), or from the keys stored in the JuMP model for problems available in JuMP only. Move the computation out of docs/browser.jl into it; the documentation (browser, status page) and CTBenchmarks read it.

Fields

Field	Content
`:grid_size`, `:parameters`	unchanged
`:status`, `:notes`	see #303; `:draft` = model not finished or not equivalent, never "does not converge" (#360)
`:reference_objective`	generated, see below (#375)
`:init`	see below (#377)
`:tags`	controlled vocabulary defined in `src` and checked by the tests (e.g. `:hard` from CTDirect#580, `:analytic`, `:large_state`) (#379)
`:references`	bibliographic sources of the problem (literature values of the objective, when known, go here as information) (#379)
`:corleone`	optional, informative: a related problem in SciML's OptimalControlBenchmarks (#308, #379)
`:reference_solution`	optional: analytic `x(t)`, `u(t)`, for convergence orders (#379)
known local objectives	see #378

~~:known_failures~~ is dropped: since #360 a convergence failure is reported, never a test failure, so there is nothing to declare.

Every field is type-checked at load time and by the consistency tests.

`:reference_objective` (done in #375)

One reference per scheme, at the default grid of the problem: the objective of the discretized problem, which every solve of the convergence report is compared with. The trapezoidal rule exists for every problem; the other schemes come with [Dev] JuMP models for Euler, implicit Euler and midpoint #369.
Computed from the convergence tests, never typed. tools/references/update.jl reads their results (cron artifacts or a local run) and keeps the best objective among the solves that converged (every modeling, configuration and platform). It records the spread between them; above 1e-6, agreement = false means that the configurations reached different solutions ([Cron] Robustness of the initial guess and known local objectives #378). Stored in ext/MetaData/references/<p>.toml, one table per scheme, with the grid, the parameters, the run and every solve. Procedure: tools/references/README.md.
OptimalControlProblems.reference_objective(p; grid_size, scheme) returns the value, or nothing.
The consistency tests fail when a reference no longer matches the default grid or parameters of its problem (compute it again), and when a problem :valid in one modeling only has no reference (comparison rule).
For a problem available in one modeling only, a computed reference is a regression check, not an independent validation: the status page ([Doc] Rewrite the documentation: getting started, user guide, problems, developers #307) shows it as such.
An analytic objective of the continuous problem, when it is known, is information with its source ([Metadata] Declared fields: tags, references, Corleone, reference solution #379). It is not the reference of a discretized solve: the two differ by the discretization error.
No tolerance: the gap is reported, never tested.

Local solutions (#378)

Some problems have several local solutions (glider, cart_pendulum, and mountain_car, where Ipopt and MadNLP reach two of them). Next to the reference, a list of known local objectives per scheme and default grid, generated, not typed. The convergence report says which one a solve reached (the best known, a known local one, or a new one). A cron test solves each problem under several solver options and reports the spread: it tells whether an initial guess is robust.

`:init` (#377)

The initial guess of every modeling comes from the metadata, no longer from each model file:

format (state=…, control=…, variable=…); each component a constant, a vector, or a function of t, never data on a grid;
OptimalControl and OptimalControlScalarized: passed as is (Problem.init, [Paper] Port to OptimalControl v2 and new problem interface #283);
JuMP: evaluated at the nodes, with the number of controls of the scheme (N+1 for trapezoid, N for Euler and midpoint, see [Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284);
overridable: beam(JuMPBackend(); init=…), and init= in solve on the OptimalControl side;
maintainer's rule: the guess must make the solvers converge without being too close to the solution, so that the solves still exercise the iterations. Do not tune tolerances or grids first;
the tests check that the OptimalControl guess and the JuMP start values coincide at the nodes (comparison/test_init).

Not now

:expected_structure (smooth, bang-bang, singular, state constraints) is dropped: it depends on the parameters, and doing it properly means a list of arcs and switching times. To revisit with indirect methods.

Done when

features(:p) computed from the model; docs/browser.jl uses it ([Metadata] Compute the features of a problem from its model #376).
Declared fields with type checks; consistency tests cover them ([Metadata] Declared fields: tags, references, Corleone, reference solution #379).
:reference_objective for every problem, computed from the convergence tests, source stated ([Metadata] Reference objectives per scheme, computed by a script #375).
Known local objectives and the robustness report ([Cron] Robustness of the initial guess and known local objectives #378).
Initial guesses read from the metadata by every modeling; no initial guess left in the model files ([Metadata] Shared initial guess in the metadata, read by every modeling #377).
Initial guess documented ([Doc] Intial guess #151), with how to override it ([Metadata] Shared initial guess in the metadata, read by every modeling #377).

New field (decided in #305): :known_failures, per problem, the configurations known to fail for a :valid problem, e.g. [(modeler=:exa, parameter=:gpu, reason="…")]. An undeclared failure fails the cron; a declared one that no longer happens is reported in the cron-capabilities issue. Type-checked like the other fields.

Note (from #308): :corleone becomes optional and informative: the name of a related problem in SciML's OptimalControlBenchmarks catalogue, if any. The Corleone modeling of this package is written here (ext/CorleoneModels/), so no mapping is needed for it.

Correction to the previous note: the modeling of #308 is ModelingToolkit (solved with Corleone); :corleone stays an optional, informative pointer to a related problem in SciML's OptimalControlBenchmarks catalogue.

From #304 (PRs #319, #324):

Metadata schema: METADATA_SCHEMA in src/OptimalControlProblems.jl (required grid_size, parameters, status; optional notes, unique_control). Unknown fields fail at load time, so new fields (:reference_objective, :init, :known_failures, features) must be added there, with their type; test/suite/core/test_consistency.jl checks every metadata file against it.
Comparison rule (test_comparison_rule): a problem :valid in one modeling only is reported ("no reference") until a sourced :reference_objective exists; once it does, make the rule fail.
:unique_control => false (jackson) replaces a problem name written in the comparison code; it is the kind of computed/declared feature this issue generalizes.
mountain_car: the scalarized model is :draft (objective 102.381 vs 102.398, 1.6e-4 on the default grid, note in the metadata): a candidate for :known_failures if the gap is understood.
The rocket metadata had five unused legacy fields (state_components, …, time_grid_names), removed when the schema was introduced.
Initial guess, maintainer's rule (when a problem converges to different minima or fragilely, e.g. cart_pendulum in one CI job): the initial guess may be changed, but it must both make the solvers converge and not be too close to the solution, so that the tests still exercise the iterations (an initial guess at the solution would make the comparisons trivial). Do not tune the tolerances or the grid first. The guess lives in the metadata (:init), shared by every modeling.
Local minima (measured in test: tests by status, consistency tests and JuMP model contract #324: glider at N = 500 reaches −1247.98, or −1066.76 under some Ipopt options; cart_pendulum has dozens of local solutions, and at a coarse grid two of them are reached depending on the platform). Proposal from the maintainer's discussion: in the metadata, a :reference_objective (best known, sourced) and a list of known local objectives, both at the default grid and per scheme ([Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284). Three levels of test: (1) converges (solver status; fast, per modeling), (2) known solution (objective in the list, within tolerance), (3) global (equals the reference). The initial guess then does not have to reach the global minimum; it must converge to a known solution, and the report says which one. An objective outside the list is reported as a new local minimum. Comparing the objectives of two modelings after a solve is not a model comparison (it compares two optimization paths): model equivalence is the NLP equivalence of [Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284 (same objective and constraints at the same point, no solve). A cron robustness test (each problem solved under several Ipopt options, the spread of the objective reported) tells whether an initial guess is robust.

Taken by an agent; draft PR to follow. A plan (split into parts, reference objectives first) is proposed to the maintainer before any code.

Split into sub-issues, agreed with the maintainer:

[Metadata] Reference objectives per scheme, computed by a script #375 part 1: reference objectives per scheme, computed by a script (independent; taken now)
[Metadata] Compute the features of a problem from its model #376 part 2: features computed from the model (independent; the docs switch waits for [Doc] Migrate the documentation to DocumenterVitepress #311)
[Metadata] Shared initial guess in the metadata, read by every modeling #377 part 3: shared initial guess in the metadata (after [Dev] Factorise the trapezoidal JuMP models in schemes.jl #368 and [Dev] JuMP models for Euler, implicit Euler and midpoint #369, which rewrite the JuMP models)
[Cron] Robustness of the initial guess and known local objectives #378 part 4: robustness of the initial guess and known local objectives (after [Metadata] Reference objectives per scheme, computed by a script #375)
[Metadata] Declared fields: tags, references, Corleone, reference solution #379 part 5: declared fields: tags, references, Corleone, reference solution (low priority)

Dropped: :known_failures (decided in #305). Since #360 a convergence failure is reported, never a test failure, so there is nothing left to declare.

🤖 Generated with Claude Code

Update after #360 and #375: convergence is reported, not tested

The description is updated (2026-11-06). The principle, now stated at the top:

Does not break: every CI run, short (small grid, a few iterations). It fails on an exception or a non-finite objective.
Model equivalence: every CI run, no solve (at the initial guess; NLP equivalence of [Paper] JuMP models for Euler, trapeze and midpoint, and NLP equivalence tests #284). It fails when the modelings differ.
Convergence: weekly cron, or on demand (test convergence). It gives the solver status and the relative gap of the objective to the reference objective. It never fails, and there is no tolerance.

Superseded in the description and in the comments above:

Was	Now
"Tolerance: to decide later, from the first cron results"	No tolerance: the gap is reported, never tested (#360)
Reference: analytic first, otherwise computed "on a fine grid" by `scripts/reference_objective.jl`	One reference per scheme at the default grid, computed from the convergence tests by `tools/references/update.jl` (#375); an analytic objective is information (#379)
`:known_failures` (description; comment of 2026-11-04)	Dropped: a convergence failure is no longer a test failure
"Three levels of test: converges / known solution / global" (comment "From #304")	Three levels of report (#378)
`mountain_car` scalarized as a candidate for `:known_failures` (same comment)	`:valid` since #363. The gap comes from the solvers: Ipopt and MadNLP reach two local solutions, in both OptimalControl modelings (#375, #378)
Comparison rule: report a problem without reference, then fail once references exist (same comment)	Done (#375)
`METADATA_INFOS`, `METADATA_TYPES`	`METADATA_SCHEMA` (`required`, `optional`, `generated`)

The work is split into #375 (done), #376, #377, #378 and #379; the description has the table.

🤖 Generated with Claude Code

All five parts are done. Closing.

Part	Issue	PR
1. Reference objectives, generated from the convergence tests	#375	#380
2. Features computed from the model (`features(problem)`)	#376	#392
3. One initial guess per problem, in its metadata, shared by every modeling	#377, #151	#395
4. Robustness of the initial guess, known local solutions	#378	#393, references regenerated in #394
5. Analytic solution of the continuous problem; sources	#379	#403

The result

The metadata declares only what cannot be computed: grid, parameters, statuses, notes, initial guess, and the analytic solution where one is known.
Everything else is computed or generated: the features from the model, the reference objectives and the known solutions from the convergence tests (tools/references/update.jl).
Every declared field is type-checked at load time and by the consistency tests.
In line with [Tests] Report convergence instead of testing it; keep hard checks for breakage and model equivalence #360, no metadata field is a test threshold. The convergence report gives the gap to the reference and says which known solution a solve reached; it never fails. The monthly robustness cron says how robust each initial guess is.

Dropped

:known_failures ([Tests] Report convergence instead of testing it; keep hard checks for breakage and model equivalence #360).
:expected_structure, :tags and :corleone (see [Metadata] Declared fields: tags, references, Corleone, reference solution #379). A descriptive structure, made of characteristics rather than judgments, is for later.

Follow-up. #398: remove the MadNLP barrier bypass of the robustness tests once the test environments use CTSolvers ≥ 0.7.1 (fixed upstream in control-toolbox/CTSolvers.jl#321).

Checked after #406: main CI green at c3ade68.

Navigation Menu

Repository navigation

[Metadata] Computed features, reference objective, shared initial guess #306

Goal

Work: sub-issues

How the tests use the metadata (since #360)

Current state

Computed, never written by hand (#376)

Fields

`:reference_objective` (done in #375)

Local solutions (#378)

`:init` (#377)

Not now

Done when

Update after #360 and #375: convergence is reported, not tested

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Metadata] Computed features, reference objective, shared initial guess #306

Description

Goal

Work: sub-issues

How the tests use the metadata (since #360)

Current state

Computed, never written by hand (#376)

Fields

:reference_objective (done in #375)

Local solutions (#378)

:init (#377)

Not now

Done when

Activity

ocots commented on Nov 4, 2026

ocots commented on Nov 4, 2026

ocots commented on Nov 4, 2026

ocots commented on Nov 5, 2026

ocots commented on Nov 5, 2026

ocots commented on Nov 5, 2026

ocots commented on Nov 6, 2026

Update after #360 and #375: convergence is reported, not tested

ocots commented on Nov 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`:reference_objective` (done in #375)

`:init` (#377)