| title | EvalForge |
|---|---|
| emoji | ⚡ |
| colorFrom | indigo |
| colorTo | yellow |
| sdk | streamlit |
| sdk_version | 1.30.0 |
| app_file | app.py |
| pinned | false |
| license | mit |
| short_description | Open-source LLM regression testing and release-decision tool. No paid APIs. |
Stop shipping prompt changes on vibes.
An open-source LLM regression testing and release-decision tool for AI PMs and prompt owners. EvalForge compares two prompt or model variants against a golden dataset and produces a release recommendation backed by quality, regressions, and category-level reliability. No paid APIs. No API keys. No auth. No database.
Every AI team has the same daily problem. Someone tweaks a prompt. The change ships. A different feature regresses three days later and nobody notices. LLM outputs are non-deterministic, so eyeballing a few cases is not a release gate. The current workflow is "paste a few examples in the playground, read the answers, ship on vibes." Quality regresses silently.
EvalForge runs a fixed golden dataset against two variants you provide, scores every case locally with deterministic and rubric-based scorers, and returns:
- a side-by-side leaderboard
- a category-level breakdown that surfaces regressions hidden by averages
- a decision memo with top risks, top failed cases, and a next action
- a failed-cases explorer with rubric reasoning per case
- a release recommendation: SHIP, SHIP WITH CAVEAT, or DO NOT SHIP
This v1 is built to be open-source friendly. It ships with a bundled golden dataset and bundled sample outputs for both variants so anyone can clone the repo and run an eval in under a minute. You can plug in real outputs from any model — local Ollama, LM Studio, ChatGPT copy-paste, Claude.ai, Gemini, anything that produces JSONL — by uploading or pasting the file. The product does not call any paid API and does not require any key.
- Three comparison modes built on a single shared eval engine
- Prompt vs Prompt
- Model vs Model
- Custom Variant Comparison
- Bundled golden dataset of 24 realistic customer-support cases across 9 categories
- Bundled weak-baseline (Variant A) and improved (Variant B) sample outputs, clearly labelled as sample data
- Local rubric scorer that grades empathy, expected-behavior coverage, next-action clarity, and policy safety
- Exact-match and contains-keyword scorers for deterministic cases
- Category-level breakdown with high-risk flagging (refund, billing, policy edge, escalation)
- Decision memo with risks, failed cases, and a next action
- Run results saved as timestamped JSONL in
data/runs/ - Local evaluation cost is
$0.00because nothing is called
- Load the bundled golden dataset (or upload your own JSONL).
- Provide outputs for Variant A and Variant B. Use the bundled samples, upload JSONL, paste JSONL, or type manually.
- The eval engine scores every case locally and aggregates per-category averages.
- The recommendation engine produces a release decision based on quality delta, regression count, and high-risk category performance.
- Results render as a leaderboard, category breakdown, decision memo, failed-cases explorer, and full results table. The run is saved to
data/runs/.
A single shared abstraction Variant powers all three modes. Prompt-vs-prompt and model-vs-model are not separate engines — they are presets over the same engine.
Variant = {
"label": "A" | "B",
"source_type": "manual_output" | "uploaded_output" | "local_stub",
"model_name": str,
"prompt": str,
"outputs": [{ "case_id": str, "output": str, "latency_ms": float | None }, …]
}
run_eval(dataset, variant_a, variant_b) -> { rows_a, rows_b, paired, summary }
| Layer | File | Responsibility |
|---|---|---|
| UI | app.py |
Streamlit screens, mode selector, loaders, result rendering |
| Engine | eval_engine.py |
Pair cases, score variants, compute summary, build recommendation, save run |
| Scorers | scorers.py |
Local rubric + exact-match + contains-keyword scoring |
| Validation | utils.py |
JSONL load, parse, validate dataset and outputs |
Add screenshots after running locally. Suggested files: screenshots/leaderboard.png, screenshots/decision_memo.png, screenshots/failed_cases.png.
- Live demo: add Streamlit Community Cloud URL after deployment
- 90-second Loom: add Loom link after recording
24 cases · 2 variants · 3 regressions caught · $0.00 local eval cost · SHIP WITH CAVEAT
The point of EvalForge is not "Variant B wins everywhere." The point is that B improves overall quality (2.21 → 3.88 average) while quietly regressing on 3 cases that a vibe-check on a few outputs would miss. EvalForge surfaces those 3 cases in the failed-cases explorer, weighs them in the decision memo, and downgrades the recommendation from SHIP to SHIP WITH CAVEAT. That is the workflow this product exists to enforce.
EvalForge has two explicit modes, surfaced as a top-level toggle in the UI:
| Mode | Dataset | Variant A outputs | Variant B outputs | When to use |
|---|---|---|---|---|
| Demo Mode | Bundled data/golden_dataset.jsonl |
Bundled sample_outputs_a.jsonl |
Bundled sample_outputs_b.jsonl |
Trying the product, recording a Loom, walking a recruiter through. |
| Custom Eval Mode | You upload | You upload | You upload | Real evaluation work. Bundled samples are off; case_ids must match between your dataset and your output files. |
Custom Eval Mode never silently uses bundled data. If your output JSONL is missing a case_id the dataset has, the run is blocked with a validation error.
Each JSONL row needs id, input, and expected. category and scoring_method are optional with sensible defaults.
{
"id": "case_001",
"input": "Customer message or task input",
"expected": "Expected behavior, answer, rubric target, or keywords",
"category": "support_complaint",
"scoring_method": "rubric_support"
}If scoring_method is omitted it defaults to rubric (generic semantic overlap, domain-agnostic). If category is omitted rows are grouped under uncategorized.
Each output JSONL row needs case_id (matching a dataset id) and output.
{
"case_id": "case_001",
"output": "I'm sorry your order is late. Could you share your order number?",
"latency_ms": 820
}latency_ms is optional — when present it's averaged into the leaderboard; when absent the latency column shows unavailable instead of a fabricated number.
Requires Python 3.10 or newer.
git clone <your-fork-url>
cd EvalForge
pip install -r requirements.txt
streamlit run app.pyOr via Make: make install && make run.
EvalForge runs on Hugging Face Spaces with the CPU Basic (free) tier. No paid inference, no API keys, no .env, no Docker.
- Create a new Space: https://huggingface.co/new-space
- Set:
- SDK: Streamlit
- Hardware: CPU Basic (free)
- Visibility: Public or Private
- Either link this GitHub repo to the Space, or push the code to the Space's git remote (
git push https://huggingface.co/spaces/<user>/<space-name> main). - The Space picks up the YAML frontmatter at the top of this README (
sdk: streamlit,app_file: app.py) and startsstreamlit run app.pyautomatically.
No secrets need to be configured. The Space will be reachable at https://huggingface.co/spaces/<user>/<space-name>.
The app opens at http://localhost:8501. No .env, no API keys, no extra setup. Click Run eval with the defaults and you should see avg A 2.21 → avg B 3.88, 20 wins, 3 regressions, ties 1, recommendation SHIP WITH CAVEAT. Open the failed-cases explorer to see the 3 regressions: an over-escalated billing duplicate, a vague 90-day-return reply, and an over-promised damaged-item handoff.
Each dataset row declares its scoring_method. Choose the one that fits your data:
| Method | When to use | What it scores |
|---|---|---|
rubric |
Default. Any domain. Summarisation, QA, code, classification, etc. | Semantic overlap between output and expected significant words → 1–5. |
rubric_support |
Customer-support replies only. | Coverage plus empathy markers, next-action clarity, policy-safety, and detection of unsupported promises and invented details. The bundled demo uses this because the cases are support-flavoured. |
exact_match |
Output must equal expected exactly. Classification labels, structured extractions. |
5 if equal after .strip(), else 1. |
contains |
Output must include a keyword phrase from expected. Fact-extraction, key-phrase QA. |
5 full match, 3 partial, 1 missing. |
If you use the support rubric on non-support data the heuristics will misfire — every output will look "missing empathy" and score badly. The UI shows a hint when both averages drop below 1.6/5, which usually means the rubric is wrong for the data.
Important: the prompt textareas in the UI are saved with the run for traceability but do not regenerate outputs. EvalForge v1 evaluates the outputs you provide; it does not call any model. To see scores change, edit the outputs (upload new JSONL or paste a new variant), not the prompt.
pip install -r requirements-dev.txt
make test # or: python -m pytest tests/ -v96 unit tests cover the rubric scorers, validators, recommendation truth table, run_eval integration, and save_run filename uniqueness. The most important test is tests/test_demo_baseline.py, which pins the headline benchmark numbers (avg A 2.21, avg B 3.88, 3 regressions, SHIP WITH CAVEAT). If you tune a regex in scorers.py or a comparator in eval_engine.py and the demo numbers drift, this test fails before the README's claims drift out of sync. CI runs the full suite on every push and PR via .github/workflows/ci.yml on Python 3.10, 3.11, and 3.12.
- v1 (now). Local rubric and deterministic scoring, three comparison modes, bundled dataset, bundled sample outputs, save run.
- v2. Optional local-model integration (Ollama, LM Studio, llama.cpp); rubric editor; eval history dashboard; dataset generator.
- v3. Local LLM-as-judge with judge-vs-human agreement audit; CI check that blocks unevaluated prompt changes; prompt diff viewer; exportable release memo; regression severity labels.
- AI PM judgment about LLM quality control under nondeterminism
- Eval design: dataset, scorer, run, decision
- Metric thinking: regressions and category-level performance over headline averages
- Release discipline: every change has a recommendation backed by evidence
- Open-source practicality: no API keys, no auth, no database, runs anywhere with Python
The bundled outputs are sample data shipped with the repo for demonstration. Replace them with real model outputs by uploading or pasting JSONL files.