Skip to content

feat: bench.axes combine and gate for the 19 holdout cells - #4

Open
jasperdew wants to merge 1 commit into
feat/odl-bench-job-960-v2from
feat/axes-gate
Open

jasperdew wants to merge 1 commit into
feat/odl-bench-job-960-v2from
feat/axes-gate

Conversation

@jasperdew

Copy link
Copy Markdown
Member

Adds python -m bench.axes combine|gate (and make axes-combine, make axes-gate).

  • combine: per-set run outputs to axis_means.json, per configuration and split. Reproduces the existing 1.2.0 round file exactly (checked by a test that skips visibly when the run directories are absent).
  • gate: holdout only; exit 1 naming the cell (or the 19-cell mean) on a drop beyond --tolerance (default 0.005); exit 3 plus SKIPPED (not a pass): <reason> when a value or row is missing; a drop wins over a skip.
  • tests/test_axes.py: synthetic fixtures, threshold boundary, a mutation check over all 19 cells, skip behaviour, CLI exit codes. Runs through pytest tests in make repro/CI.

No scores or baselines added. Not merged; base is the odl-bench PR branch.

combine writes axis_means.json from run directories (it reproduces the
1.2.0 round file exactly); gate fails when a holdout cell or the 19-cell
mean drops by more than the tolerance and exits 3 with
SKIPPED (not a pass) when a value is missing. Tests use synthetic numbers
and run in make repro and CI through pytest tests.

Signed-off-by: jasperdew <10383561+jasperdew@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant