Repository navigation
Conversation
…ression gate Recipes (versioned, printed into every result), manifest hashes of the 200 PDFs and ground-truth files, a scoreless tune/holdout split file, per-metric scores per half, a baseline gate (tolerance 0.005 and per-document canaries), a loud skip when the models are absent, and the tests for all of it, run in CI. The full class labels (#969) stay private and are passed with --labels. Refs pdfluent-internal#960 and #969. Signed-off-by: jasperdew <10383561+jasperdew@users.noreply.github.com>
…P-Bench fingerprints (#968) Signed-off-by: jasperdew <10383561+jasperdew@users.noreply.github.com>
This was referenced Oct 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs pdfluent-internal#960, #969, #968. Replaces #2 (closed unmerged).
python -m bench.odlplusodl-bench/: versioned recipes (printed into every result), manifest hashes of the 200 PDFs and ground-truth files at the pinned commit (corpus cloned at run time, never copied), scoreless tune/holdout split (split-v1.json, with its SHA-256), per-metric scores per half, baseline gate with tolerance 0.005 and canaries,SKIPPED (not a pass)(exit 3) when models are absent, tests in CI, plus the contamination matcher (#968).Baselines and the full class labels (#969) carry numbers and stay private: the engine CI passes
--baselineand--labels, and runs the labels checks withODL_LABELS.Known and not from this PR:
tests/test_workflows.pyhas two failures on main since the identity-check workflow landed (not self-hosted);tests/test_round.pyhas four failures that need the docker images.Not done: competitor rerun with paired bootstrap, the 19-axes holdout call, the engine-side CI job.