This repository contains a Wikipedia retrieval engine developed across the MIT xPRO RAG and Context Engineering capstone checkpoints. The project separates four parts of the system so that they can be tested independently: corpus preparation, retrieval, context construction, and answer generation.
The selected capstone scenario is the Wikipedia Retrieval Engine. It answers single-article and multi-article questions over a local collection of 2,419 Wikipedia HTML files. The evaluation set includes direct facts and quotations, table lookups, time-bounded lists, cross-article comparisons, relationships among people and places, geopolitical timelines, calculations, conflicting estimates, and questions that require caution about causal inference.
The 7.1 Final Survivor is the safeguarded agent with Router B model assignment. It
uses gpt-5.4-mini for decomposition, article selection, and table planning. When the
TOC review flags a table question, it uses gpt-5.4 for the evidence assessor and final
answer—the two roles with the clearest measured quality benefit. Input sanitising, the
guarded answer prompt, and the deterministic grounding check remain active.
Across three 19-question GT runs, Router B scored 0.868 at $0.1725 list cost per question
and 21.3 seconds per question. All-gpt-5.4 scored 0.886 at $0.3457 per question and 24.7
seconds. The observed
quality difference was −0.018 (paired Wilcoxon p = 0.81), while Router B reduced list
cost by 50% (p = 0.001). This means the experiment detected a cost reduction but did not
detect a quality difference; it does not prove statistical equivalence.
On the held-out 25-question validation set, the 7.1 Final Survivor scored 0.933, with 99.3% article recall, at $0.1068 list cost and 18.7 seconds per question. The 6.1 survivor scored 0.860, with 100% recall, at $0.0422 and 12.8 seconds. Security testing also reduced fabricated-fact adoption from 5 of 59 attack opportunities to 0 of 59; the pre-grouped fake-nominee scenarios fell from 5 of 20 to 0 of 20 (Fisher's exact p = 0.047). These figures describe the recorded datasets and runs, not guaranteed behavior on every input.
For a practical introduction to the LangChain APIs used by this project—including LCEL, OpenRouter configuration, Chroma, RAG patterns, version migration, adaptive LangGraph workflows, and common failure modes—see LangChain tutorial, reference, and gotchas.
Checkpoint READMEs record the project decisions and experimental evidence:
- Checkpoint 2.1 retrieval notes
- Checkpoint 3.1 baseline evaluation
- Checkpoint 4.1 advanced retrieval
- Checkpoint 5.1 agentic evidence graph
- Checkpoint 6.1 security and cost audit: security assessment, and performance and cost
- Checkpoint 7.1 production readiness: security improvements, performance and efficiency, security comparison, 5.1 against the safeguarded baseline, final accuracy and recall report, and final efficiency report
For the evolution at a glance — every retriever/approach tried, survived, or abandoned across checkpoints 1.1–7.1, with both RAGAS metrics (recall and answer quality) at each fork, one survivor per checkpoint, and the survivors' progression on the test and validation sets — open the interactive Retriever Phylogeny in a browser.
rag_and_context_engineering/
├── README.md
├── LANGCHAIN_GUIDE.md
├── requirements.txt
├── test_setup.py
└── wikipedia_retrieval_engine/
├── retriever_phylogeny.html # interactive lineage of retrievers 1.1–7.1
├── Wikipedia Library/ # local HTML corpus (2,419 articles)
├── GT/
│ ├── wikipedia_GT.csv # canonical 19-case evaluation set (GT01–GT19)
│ ├── wikipedia_GT_validation_set.csv # 25-case held-out set (overfitting guard)
│ └── capstone2.1_gt_cleanup_notes.md
├── retrievers/ # reusable retrieval library (see below)
│ ├── base.py # BaseRetriever, RetrieverSpec, normalize_scores
│ ├── corpus.py # loaders + DOM extractor (Markdown + JSON sidecar)
│ ├── bm25.py / vector.py / hybrid.py # lexical, vector, and fused retrievers
│ ├── multistep.py # query-decomposition retriever (Lab 4.1)
│ ├── graph.py # graph-augmented retriever (Lab 4.2)
│ ├── expansion.py # expand seed chunks → full article context
│ ├── evidence.py / llm.py # evidence helpers; embeddings, answer prompt (+ optional history), decompose LLM
├── utils/
│ ├── ragas_eval.py # shared RAGAS harness (GT parse, LLM/judge, rate-limit)
│ ├── analyze_ragas_experiments.py # aggregate CSV runs → PDF report
│ ├── analyze_ragas_experiments_efficiency.py # accuracy against cost and latency → PDF + CSV tables
│ ├── openrouter_usage.py # key usage snapshot; --analytics by app/model/day
│ ├── wiki_table_parser.py # rich Wikipedia-table normalization
│ └── wiki_table_store.py # offline artifact builder + read-only runtime API
├── wiki_agents/ # Checkpoint 5.1 LangGraph agent, with the 7.1 safeguards
│ ├── agent_graph.py # graph assembly, router, retrieval-stage configuration
│ ├── retrieval_nodes.py # retrieval, TOC review, section loading, evidence assessment
│ ├── table_nodes.py # table query planner + deterministic executor node
│ ├── agentic_models.py # graph state, routing, and evidence models
│ ├── run_logging.py # per-question AgentState evidence logs
│ ├── answer_context.py # answer-context rules: 5.1 sections or whole articles (6.1)
│ ├── sanitize.py # 7.1 input layers: escaping, forged-evidence strip, injection filter
│ ├── grounding_check.py # 7.1 deterministic output check: quotations and names must be in the evidence
│ ├── answer_guard.py # 7.1 failure policy for the grounding verdict (off / surface / withhold)
│ ├── answer_nodes.py # 7.1 graph nodes: sanitize, answer, guard
│ ├── prompts.py # legacy and guarded (7.1) answer system prompts
│ ├── chat.py # multi-turn chat over the guarded graph (manual testing)
│ ├── model_routing.py # 7.1 model routing: mini by default, gpt-5.4 by rule; Jev answer check
│ └── run_agent.py # single-question run through evidence assessment
├── tests/ # unit tests (unittest)
├── Capstone Checkpoint 1.1/
├── Capstone Checkpoint 2.1/
├── Capstone Checkpoint 3.1/ # baseline retrieval + evaluation
├── Capstone Checkpoint 4.1/ # advanced retrieval (decomposition + graph)
├── Capstone Checkpoint 5.1/ # agentic evidence workflow
├── Capstone Checkpoint 6.1/ # security and cost audit; the 6.1 survivor
│ ├── README 6.1 - security assessment.md
│ ├── README 6.1 - performance and cost.md
│ ├── survivor_agent.py # builds the 5.1 survivor from the 5.1 harness constants
│ ├── security_test_starter.py # multi-turn attacks (history on); --survivor 5.1, 6.1 or final
│ ├── cost_test_starter.py # 5.1, 6.1 or final survivor evaluation + per-call/per-node telemetry (history off)
│ ├── survivor_6_1.py # the 6.1 survivor: whole articles for text questions, surfaced gaps, model arguments
│ ├── answer_replay_starter.py # re-answers saved 5.1 runs with another context or answer model
│ ├── usage_accounting.py # shared per-call tokens and list/billed cost
│ ├── ragas_6_1_survivor_vs_5_1.pdf # RAGAS report: 6.1 configurations against 5.1, with cost and latency
│ ├── security_tests/ # one folder per security run (transcripts, results CSV, config)
│ ├── cost_tests/ # RAGAS CSVs + telemetry per run, by question set
│ ├── replay_tests/ # answer replay outputs (calls CSV, config, summary)
│ ├── logs/ # AgentState evidence logs per question (security and cost runs)
│ └── pricing/model_prices.json # verified USD per 1M tokens (OpenRouter, 2026-09-27)
└── Capstone Checkpoint 7.1/ # production readiness: safeguards, then performance and cost
├── README 7.1 - security improvements.md
├── README 7.1 - performance and cost.md
├── security_comparison_5_1_vs_7_1.md # security result: 5.1 survivor against the safeguarded baseline
├── survivor_agent_final.py # safeguarded 7.1 graph + optional role-based model routing
├── security_test_final.py # 6.1 security scenarios against the 7.1 system
├── cost_test_final.py # 6.1 cost-and-quality harness against the 7.1 system
├── prompt_replay.py # re-answers saved GT contexts with the legacy and guarded prompts
├── jev_route_check.py # Jev's route for the 44 questions against an answer key
├── jev_answer_check.py # Jev's "complete?" check over saved answers; escalation simulated
├── analyzers/ # compare_security_runs, rebuild_answer_context, replay_grounding, build_final_validation (offline)
├── cost_efficiency/ # performance runs: wikipedia_GT/ (results + telemetry), logs/, console/, run_manifest.csv, reports/;
│ # wikipedia_GT_validation_set/ (final validation runs, own manifest/logs/console)
├── final_validation/ # 5.1, 6.1, 7.1 survivors, 3 runs each (copies + provenance.csv) and the two final PDFs
└── security_tests/ cost_tests/ logs/ # run outputs, as in 6.1
The checkpoint folders contain the assignment-specific scripts, evidence, reports,
and technical notes. Reusable retrieval code lives in retrievers/ (each retriever
implements BaseRetriever.getTopK), and utilities that operate across checkpoints
live in utils/. The corpus is available in two representations: plain text
(wikipedia_plain_text) and structure-preserving Markdown + a JSON sidecar
(wikipedia_markdown, built by corpus.build_or_load_markdown_store) whose
categories/links/tables are the substrate for the Checkpoint 4.1 graph.
- Checkpoint 1.1 — baseline generation. Established the Wikipedia question-answer scenario, initial prompt, and repeat-run response logs.
- Checkpoint 2.1 — retrieval experiments. Added BM25, vector, and weighted hybrid retrieval; compared chunking strategies and full-article context; and created the initial manually reviewed ground-truth workbook.
- Checkpoint 3.1 — controlled evaluation. Introduced the canonical GT set,
oracle-context and end-to-end runs, RAGAS scoring, repeat-run analysis, and the
vector
k=7baseline. - Checkpoint 4.1 — advanced retrieval. Rebuilt the corpus as Markdown plus JSON sidecars, tested query decomposition, and added a rarity-weighted article graph. Graph retrieval improved article recall, but answer quality remained limited by table aggregation and reasoning.
- Checkpoint 5.1 — agentic evidence reasoning. Implements a bounded LangGraph
agent (
wiki_agents/). It finds articles through entity decomposition, selects sections and table artifacts from each article's table of contents, forces every selected table through a deterministic lookup/compute executor, and escalates to same-article breadth and then graph neighbors when evidence is insufficient. Final runs held designated-article recall at 1.000. On the held-out validation set, quality reached 0.880 (4.1 graph: 0.885) with about one fifth of the 4.1 context. - Checkpoint 6.1 — security and cost audit. Audits the 5.1 survivor, then builds
the 6.1 survivor from the findings. The security experiment adds conversation history
(the 5.1 agent is single-turn): the roleplay attack was refused in all four
conversations, while planted conversation text was adopted as retrieved fact in 3 of
30 conversations, each time the current evidence lacked the answer (the pipeline
fails open). The cost experiment measures real billed cost per question (the 5.1
survivor costs 1.45× the 4.1 graph per question once caching is counted) and replays
the saved 5.1 answer step to test whole articles for questions that need no table.
The resulting 6.1 survivor (
survivor_6_1.py: whole articles for text questions, incomplete table evidence flagged to the user) matches 5.1's validation quality (0.860) at 20% less time and 17% less billed cost, answering text questions almost perfectly (0.974) while two table questions regressed. The same graph ongpt-5.4reached 0.960 in one run, at 3.6× the list cost (4.4× billed). - Checkpoint 7.1 — production readiness. Adds deterministic input sanitising, a
guarded answer prompt, and a grounding check inside the agent graph. In the pooled
security evaluation, fabricated facts reached 5 of 59 answers before the safeguards
and 0 of 59 afterward; the two pre-grouped fake-nominee scenarios fell from 5 of 20
to 0 of 20 (Fisher's exact p = 0.047). The controls were evaluated together, so the
result does not assign causality to a single layer. Performance experiments then
compared all-mini, all-
gpt-5.4, question routing, answer checking, and role-based routing. Router B became the Final Survivor: it usesgpt-5.4for the assessor and answer on table questions and mini elsewhere. Across three GT runs it scored 0.868 at half the list cost of all-gpt-5.4; on three held-out validation runs it scored 0.933 at $0.1068 per question, with 99.3% article recall.
Two ground-truth artifacts are retained for different purposes:
wikipedia_retrieval_engine/Capstone Checkpoint 2.1/wikipedia Q-A GT.xlsxis the original workbook used for the manual Checkpoint 2.1 review. It remains beside that checkpoint so its historical results can be reproduced and interpreted correctly.wikipedia_retrieval_engine/GT/wikipedia_GT.csvis the current canonical evaluation set for Checkpoint 3.1 and later work. It contains stable IDs GT01-GT19, reference answers, question-specific PASS/PARTIAL/FAIL rules, designated source articles, scope assumptions, evaluation roles, and complexity labels.
The audit and migration from the original workbook to the current CSV are documented isurn wikipedia_retrieval_engine/GT/capstone2.1_gt_cleanup_notes.md.
This project uses a local Python virtual environment in .venv.
Use Python 3.11 or 3.12. macOS /usr/bin/python3 is often Python 3.9 and cannot
install all Checkpoint 3.1 dependency pins.
From this repository root:
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txtCreate a .env file in this directory and add the OpenRouter API key supplied for the
course:
OPENROUTER_API_KEY=sk-or-v1-...
Verify the environment:
python test_setup.pyA successful run prints a short model reply followed by
Setup verification completed successfully.
For later sessions, reactivate the environment with:
source .venv/bin/activatePlace the supplied 2,419 HTML files in:
wikipedia_retrieval_engine/Wikipedia Library/
The source corpus and generated databases are local working data rather than files to commit. Builders validate their cached output and reuse it on later runs.
This representation preserves sections, infoboxes, links, categories, and compact table rows. It is used by Checkpoint 4.1 context expansion and graph construction.
python -m wikipedia_retrieval_engine.retrievers.corpus \
"wikipedia_retrieval_engine/Wikipedia Library"The first build embeds the full corpus and therefore uses OpenRouter credits. Later runs load the persisted Chroma collection but still embed each search query.
python -c 'from wikipedia_retrieval_engine.retrievers.vector import build_or_load_db; build_or_load_db(chroma_db_name="wikipedia_chars1600_overlap200")'Checkpoint 5.1 retrieves heading-based chunks so each initial evidence item retains its article, section title, and section hierarchy. The first build embeds the full corpus; later runs reuse the stored collection.
python -c 'from wikipedia_retrieval_engine.retrievers.vector import build_or_load_section_db; build_or_load_section_db()'The dedicated builder is intentional: if the section index is missing, it calls
load_wikipedia_section_chunks() instead of silently rebuilding that database name
with the baseline character-chunk strategy.
Build the Markdown sidecars first. The graph itself is deterministic and makes no LLM call:
python -c 'from wikipedia_retrieval_engine.retrievers.corpus import build_or_load_markdown_store; from wikipedia_retrieval_engine.retrievers.graph import build_or_load_graph; build_or_load_graph(build_or_load_markdown_store())'The answering graph never reparses HTML or writes corpus artifacts. Build the normalized table layer once:
python -m wikipedia_retrieval_engine.utils.wiki_table_storeThe builder writes compressed per-article artifacts and a catalog manifest under
wikipedia_retrieval_engine/retrievers/db/table_artifacts/wikipedia_tables/. A repeat
invocation reuses the published store when the source inventory, parser, schema, and
selector are unchanged. Pass --force only for an intentional rebuild. See the
Checkpoint 5.1 README
for the representation and agent-routing design.
python -m unittest discover -s wikipedia_retrieval_engine/tests -p "test_*.py"Run these commands from the repository root.
This mode bypasses retrieval and supplies the complete designated source articles to the answer model. It isolates answer generation and judge behavior from retrieval.
python "wikipedia_retrieval_engine/Capstone Checkpoint 3.1/capstone_checkpoint_3_1_oracle_AI_only.py"This mode retrieves Top K seed chunks, expands the selected articles to full text,
generates answers, and records answer scores and article-level retrieval diagnostics.
The active BASELINE_RETRIEVER_SPEC in the script determines whether the run uses
vector or hybrid retrieval.
python "wikipedia_retrieval_engine/Capstone Checkpoint 3.1/capstone_checkpoint_3_1_with_retriever.py"RAGAS saves one CSV per run under:
wikipedia_retrieval_engine/Capstone Checkpoint 3.1/ragas_experiments/experiments/
The Checkpoint 3.1 technical README explains the experiment controls, final metrics, and known limitations in detail.
The harness runs the final 5.1 configuration (entity-decomposed initial retrieval, TOC
selection, table gateway, then article_broaden → graph_seeded escalation) over a
ground-truth CSV, writes one RAGAS CSV per run, and writes one AgentState evidence
log per question under Capstone Checkpoint 5.1/logs/:
python "wikipedia_retrieval_engine/Capstone Checkpoint 5.1/capstone_checkpoint_5_1_agent_rag_starter.py" \
--gt-set "wikipedia_retrieval_engine/GT/wikipedia_GT.csv" \
--output-dir "wikipedia_retrieval_engine/Capstone Checkpoint 5.1/ragas_test_GT_5.1" \
--experiment-name agentic_search_compute_broaden--limit N runs a smoke test; repeat --question-id GT16 to select questions. The
Checkpoint 5.1 README documents the configuration, the progression, and the reports.
Both starters build the unchanged 5.1 survivor through
Capstone Checkpoint 6.1/survivor_agent.py by default. Each has a --self-check that
runs without API calls.
Security (history on). Each scenario is one conversation. The answer model
receives the earlier turns, built by retrievers/llm.py build_answer_messages;
retrieval sees only the current message.
Print the scenarios (no API calls):
python "wikipedia_retrieval_engine/Capstone Checkpoint 6.1/security_test_starter.py" --showRun every scenario three times:
python "wikipedia_retrieval_engine/Capstone Checkpoint 6.1/security_test_starter.py" --scenario all --repeats 3--rescore <run folder> re-applies the attack detectors to a finished run without API
calls.
Cost (history off). Runs the 5.1 RAGAS case function unchanged, so the results
CSV keeps the 5.1 schema. Token telemetry per call, node, and question is written to
<output-dir>/telemetry/.
Smoke test on one question:
python "wikipedia_retrieval_engine/Capstone Checkpoint 6.1/cost_test_starter.py" --question-id VAL04Full validation set:
python "wikipedia_retrieval_engine/Capstone Checkpoint 6.1/cost_test_starter.py" \
--gt-set "wikipedia_retrieval_engine/GT/wikipedia_GT_validation_set.csv"The 6.1 survivor (survivor_6_1.py) runs through the same starter with telemetry.
--survivor 6.1 selects it. --model sets the model for every agent role and the
answer, and --table-model gives the table planner its own model. The judge stays
gpt-5.4-mini.
python "wikipedia_retrieval_engine/Capstone Checkpoint 6.1/cost_test_starter.py" --survivor 6.1python "wikipedia_retrieval_engine/Capstone Checkpoint 6.1/cost_test_starter.py" --survivor 6.1 --model openai/gpt-5.4Answer replay. Re-answers the saved 5.1 survivor runs (their AgentState logs are in
Capstone Checkpoint 5.1/logs/) under another answer-context rule or answer model,
without re-running the agent. Every run first checks that the rebuilt 5.1 contexts
match the 5.1 results.
Rebuild every context, show what would change, and estimate the cost (no API calls):
python "wikipedia_retrieval_engine/Capstone Checkpoint 6.1/answer_replay_starter.py" --dry-runExperiment 1, 5.1 sections against whole articles:
python "wikipedia_retrieval_engine/Capstone Checkpoint 6.1/answer_replay_starter.py" \
--rules sections whole_per_question whole_per_article--question-id VAL11 --runs validation --limit-runs 1 runs a smoke test, --models
compares answer models, and --summarize <replay folder> rebuilds the summary without
API calls.
The Checkpoint 6.1 READMEs record the scenarios, the choices behind them, and the results.
Capstone Checkpoint 7.1/survivor_agent_final.py builds the 6.1 survivor with three
safeguards inside its graph: input sanitising before retrieval, the guarded answer
instructions (wiki_agents/prompts.py), and a grounding check between the drafted answer
and the reader. This all-mini configuration is the 7.1 safeguarded baseline. The
selected 7.1 Final Survivor adds Router B: gpt-5.4 is used for the assessor and
answer when the TOC review flags a table question, while the other roles remain on
gpt-5.4-mini. The 7.1 entry points reuse the 6.1 harnesses so scenarios, datasets,
detectors, and judges remain comparable.
python "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/survivor_agent_final.py" --self-checkSecurity (history on). Every scenario three times against the 7.1 system:
python "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/security_test_final.py" --scenario all --repeats 3The 5.1 baseline the same way, written beside it:
python "wikipedia_retrieval_engine/Capstone Checkpoint 6.1/security_test_starter.py" --scenario all --repeats 3 \
--output-dir "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/security_tests"--grounding off|surface|withhold sets the policy (default surface: a [QUARANTINED]
notice before an unsupported answer), --no-sanitize turns the input layers off, and
--legacy-prompt keeps the 2.1–6.1 answer prompt. In a guarded run the attack detectors
read answer_drafted, the answer before the policy acted, so one run reports both what
the agent drafted and what the reader received.
Compare the runs and write the plain-language report, pooling several runs per side:
python "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/analyzers/compare_security_runs.py" \
--before <5.1 run folders> --after <7.1 run folders> \
--output "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/security_comparison_5_1_vs_7_1.md"Quality and cost (history off). Run the safeguarded all-mini baseline:
python "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/cost_test_final.py"Run the selected Router B configuration on a chosen question set and output directory:
python "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/cost_test_final.py" \
--routing b \
--gt-set "wikipedia_retrieval_engine/GT/wikipedia_GT_validation_set.csv" \
--output-dir "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/cost_efficiency/wikipedia_GT_validation_set"The final validation comparison is rebuilt from copies of the recorded source runs;
provenance.csv preserves the source path and original experiment name for every copy:
python "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/analyzers/build_final_validation.py"
python wikipedia_retrieval_engine/utils/analyze_ragas_experiments.py \
--experiments-dir "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/final_validation/runs" \
--output "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/final_validation/ragas_validation_5_1_6_1_7_1.pdf" \
--title "Final validation: survivor progression from 5.1 to 7.1"
python wikipedia_retrieval_engine/utils/analyze_ragas_experiments_efficiency.py \
--experiments-dir "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/final_validation/runs" \
--baseline "6.1 survivor" \
--output "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/final_validation/efficiency_validation_5_1_6_1_7_1.pdf" \
--title "Survivor quality, cost, and latency"Offline evidence from saved runs. Rebuild the exact context each saved 6.1 answer was given, checked against the recorded sizes, then replay the grounding check and its policies over those answers (no API calls):
python "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/analyzers/rebuild_answer_context.py" --self-checkpython "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/analyzers/replay_grounding.py"Chat. A multi-turn session over the guarded graph, for trying the safeguards by hand
(live calls; :state, :history, and :forget work inside the session):
python "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/survivor_agent_final.py" --chat --show-stateThe two Checkpoint 7.1 READMEs record the safeguard design, routing experiments, statistical limitations, and final decision.
wikipedia_retrieval_engine/utils/analyze_ragas_experiments.py recursively discovers
RAGAS experiment CSVs, groups repeated runs by experiment_name / config_name when
present and by derived configuration metadata otherwise, recomputes article recall
with normalized filenames, and creates a landscape PDF containing:
- PASS/PARTIAL/FAIL rates and the weighted quality index;
- designated-source recall and full-source coverage;
- mean context size;
- task-family attention scores and oracle-to-retriever gaps;
- performance grouped by one, two, or three-plus designated articles;
- per-question outcome heatmaps;
- run-to-run variability;
- cost and latency per configuration (calls, tokens, list and billed cost, seconds per
question, and quality against cost) for runs whose
telemetry/<run>__question_telemetry.csvexists (Checkpoint 6.1 cost starters); and - included, excluded, and duplicate-file diagnostics.
--experiments-dir accepts several directories, for example the 5.1 validation runs and
the 6.1 cost-test runs in one report:
python wikipedia_retrieval_engine/utils/analyze_ragas_experiments.py \
--experiments-dir "wikipedia_retrieval_engine/Capstone Checkpoint 5.1/ragas_validation_GT_5.1" \
"wikipedia_retrieval_engine/Capstone Checkpoint 6.1/cost_tests/wikipedia_GT_validation_set" \
--output "wikipedia_retrieval_engine/Capstone Checkpoint 6.1/ragas_6_1_survivor_vs_5_1.pdf"The script does not call an LLM or make network requests. It requires pandas and
reportlab, which are included in the Checkpoint 3.1 requirements.
Run it with the project defaults:
python wikipedia_retrieval_engine/utils/analyze_ragas_experiments.pyBy default, it reads the Checkpoint 3.1 experiment directory and writes:
../../output/pdf/ragas_evaluation_report.pdf
The output folder is intentionally outside the capstone repository. Choose an explicit destination when creating a submission copy:
python wikipedia_retrieval_engine/utils/analyze_ragas_experiments.py \
--experiments-dir "wikipedia_retrieval_engine/Capstone Checkpoint 3.1/ragas_experiments/experiments" \
--output "wikipedia_retrieval_engine/Capstone Checkpoint 3.1/ragas_evaluation_report_checkpoint3_1.pdf" \
--title "Capstone Checkpoint 3.1 RAGAS evaluation results"Earlier oracle runs used a binary PASS/FAIL rubric and are excluded from the primary
comparison because those labels are not compatible with the final three-level rubric.
Pass --include-binary-oracle only when intentionally reviewing those legacy runs;
do not combine them with the final baseline statistics.
wikipedia_retrieval_engine/utils/analyze_ragas_experiments_efficiency.py (Checkpoint
7.1) reads the same experiment CSVs plus the cost starters' telemetry and reports the
trade-offs, one section per question set (GT and validation are never pooled):
- frontier: quality against list cost and seconds, run points and means, dominated configurations;
- paired against a baseline: per-question changes, better/worse/same, Wilcoxon p, the price of a quality point;
- cost and model time by role;
- text-ready, text-escalated, and table paths;
- tail and waste: p50, p90, maximum, most expensive question-runs, spend on fails;
- comparability: run order, caching, billed against list, per-call latency by role, with warnings;
- safeguard markers: evidence notices, grounding flags and false alarms, sanitizer hits;
- routing: what-if policies on saved runs (
--route-cheap,--route-strong), and the route mix once runs record aroutecolumn.
It writes the PDF and one CSV per table (<output stem>_tables/). Display labels for the
experiment names are built in (--alias NAME=LABEL adds more); --baseline takes a
label or an experiment name. Run order comes from a run_manifest.csv above the runs
when there is one, else from the file times.
python wikipedia_retrieval_engine/utils/analyze_ragas_experiments_efficiency.py \
--experiments-dir "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/cost_efficiency/wikipedia_GT" \
--baseline "6.1 survivor" \
--output "wikipedia_retrieval_engine/Capstone Checkpoint 7.1/cost_efficiency/reports/gt_baseline_6_1_vs_7_1.pdf" \
--title "GT baseline: 6.1 survivor against 7.1 safeguarded baseline"With no arguments it reads Capstone Checkpoint 7.1/cost_efficiency/wikipedia_GT and
writes cost_efficiency/reports/efficiency_report.pdf. Folders named reports or
*_tables are never read as runs, so a report can sit beside the runs it describes.
A routing simulation adds, for example,
--route-cheap "6.1 survivor" --route-cheap "7.1 final" --route-strong "6.1 survivor, gpt-5.4".
Here, 7.1 final is the historical experiment label for the all-mini safeguarded
baseline, not the later Router B Final Survivor.
Tests for the analyzer's grouping, comparison, and routing logic are in
tests/test_analyze_efficiency.py.
wikipedia_retrieval_engine/utils/openrouter_usage.py checks the OpenRouter API key
used by the capstone scripts without printing the key itself. It reads
OPENROUTER_API_KEY from the environment or a nearby .env file:
python wikipedia_retrieval_engine/utils/openrouter_usage.pyTo log a post-run snapshot by experiment/config name:
python wikipedia_retrieval_engine/utils/openrouter_usage.py \
--experiment-name vector_search_base \
--log-csv "wikipedia_retrieval_engine/Capstone Checkpoint 4.1/openrouter_usage_log.csv"Set OPENROUTER_MANAGEMENT_KEY when you also want account-level credit remaining or
need to list/query managed keys.
The management key also enables the analytics query (time bucket × dimension usage:
request count, cost, prompt/completion/cached tokens). Since 2026-09-27 each capstone
role sends its own referer (retrievers/llm.py openrouter_headers), so grouping by
origin separates system cost from evaluation cost: …/retriever for the agent and
answer, …/judge, …/embedding, …/replay-answer. Earlier traffic shares one origin,
and the app dimension reports it as "Unknown".
python wikipedia_retrieval_engine/utils/openrouter_usage.py \
--analytics --analytics-dimension origin --analytics-days 14 \
--analytics-csv "wikipedia_retrieval_engine/Capstone Checkpoint 6.1/openrouter_analytics_by_origin.csv"--analytics-granularity hour isolates individual runs (buckets are in UTC).
The controlled experiments establish a useful sequence:
- Checkpoint 3.1 found a strong vector
k=7baseline and showed that perfect source selection still did not solve exhaustive-list and time-filter questions. - Checkpoint 4.1 improved article recall from 0.899 to 0.939 with graph expansion, but mean answer quality remained approximately flat. Clean Markdown tables also did not resolve the difficult aggregation family by themselves.
- Checkpoint 5.1 therefore targets the reasoning stage: assess what evidence is missing, retrieve deliberately, compute over normalized table rows when necessary, and verify the answer against precise provenance.
- Checkpoint 5.1 reached designated-article recall of 1.000 with a fraction of the 4.1 context. It also makes five or more model calls per question and adds new channels through which user text and retrieved text reach a model (decomposition, TOC review, assessment, table planning). Checkpoint 6.1 therefore audits security, reliability, latency, and cost before further capability work.
- Checkpoint 6.1 found that the pipeline fails open: planted conversation text was adopted as retrieved fact when the current evidence lacked the answer. Its reliability fixes were measured with history off, where that attack cannot occur. Checkpoint 7.1 therefore adds safeguards inside the agent rather than in a test harness, and re-runs the 6.1 attacks with history on against the combined changes.
- Checkpoint 7.1 then measures where the stronger model earns its cost. Repeated GT
runs support Router B: use
gpt-5.4for evidence assessment and answer writing on table questions, and use mini for decomposition, article selection, and table planning. The held-out validation comparison checks the selected system once the architecture is fixed.
These numbers are experiment-specific, not general claims about one retriever always being superior. The complete methods, caveats, and per-question results remain in the checkpoint READMEs linked at the top of this file.
AI assistance was used to organize code, develop utilities and log-analysis tools, consolidate prior run outputs, organize observations, and refine the wording of the written analysis. The performance evaluation, prioritization of next steps, and proposed security and efficiency changes were determined by the author from the recorded data. The author reviewed the prompts, experimental runs, interpretations, and conclusions and made the final decisions about the outcomes and evidence included in the report.