Skip to content

Add per-jurisdiction event logs and a workflow map - #606

Merged
ppinchuk merged 28 commits into
mainfrom
add-event-logging
Nov 17, 2026
Merged

ppinchuk merged 28 commits into
mainfrom
add-event-logging

Conversation

@rajeee

@rajeee rajeee commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Each jurisdiction run now writes <jurisdiction>_events.jsonl next to its .log file, recording the searches, downloads, filter decisions, LLM calls and extraction steps it went through. It works for compass process and for the split search, collect, and extract commands, which add to the same file. On by default; keep_event_logs: false turns it off. compass/data/workflow.json5 maps every step and branch the pipeline can take, and tests keep that map in step with the code.

Also records every HTTP response behind a document download (http_response, or http_error when none came).

Depends on ELM

Requires nlr-elm>=0.0.53 (released), which includes NatLabRockies/elm#96: the hooks this PR records through. They are on_search_hook (each search engine attempt), on_response_hook and on_error_hook (each HTTP response, or why none came), and on_fetch_hook (what each document fetch produced). pyproject.toml raises the minimum version from 0.0.52 to 0.0.53.

Errors and retries are recorded so a slow or failed run can be explained from its log:

  • Errors keep their whole cause chain (for example ELM's "Maximum number of retries" down to the socket's ConnectionResetError), and each exception lists the file, line and function it passed through, including the code that led to where it was caught.
  • Retries that the OpenAI client and ELM's retry wrapper make on their own are recorded as llm_retry events, with the wait and the error behind each, tagged with the LLM call they belong to; retries of downloads by the same wrapper are download_retry events. A call that succeeds after retrying otherwise looks only slow.
  • To tag retries with their call, compass/services now runs each queued job in its caller's context (asyncio.create_task(..., context=...)); the provider already named each job's task after its caller.

Live demo

https://pages.github.nlr.gov/radhikar/htmls/COMPASS/pipeline_replay.html (lab network only)

A browser viewer that replays event logs against the workflow map. It opens with five example data center runs: three that found their ordinance (Atlanta GA, Tucson AZ, Chandler AZ) and two that did not (Provo UT, DeKalb County GA), which show where a search stops. Many LLM calls in these runs were retried, and some failed, because the LLM proxy dropped connections; the viewer shows each retry and error chain, with links to the source lines. Use "Open log folder" to load the logs from your own run. The viewer itself is not part of this PR, and the example logs come from runs that used the priority crawler in #607, so they include crawler events this PR alone does not emit.

First of three stacked PRs: event logging (this one) → priority crawler (#607) → data center plugin (#608).


Stack created with GitHub Stacks CLI • Give Feedback 💬

Each jurisdiction run writes an events.jsonl recording the searches,
downloads, filter decisions, LLM calls and extraction steps it went
through. compass/data/workflow.json5 maps every step and branch the
pipeline can take, and tests keep that map in step with the code.

Also adds elapsed-time logging for long-running operations and records
the HTTP exchange behind each document download.
@rajeee
rajeee added this pull request to stack #609 November 9, 2026 19:37
@rajeee rajeee changed the title add event logging Add per-jurisdiction event logs and a workflow map Nov 9, 2026
@codecov-commenter

codecov-commenter commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.60839% with 60 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.94%. Comparing base (0a4668d) to head (7c426cd).
⚠️ Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
compass/scripts/download.py 72.72% 14 Missing and 1 partial ⚠️
compass/extraction/apply.py 12.50% 14 Missing ⚠️
compass/utilities/workflow.py 84.12% 5 Missing and 5 partials ⚠️
compass/common/tree.py 82.50% 5 Missing and 2 partials ⚠️
compass/utilities/event_log.py 97.65% 3 Missing and 2 partials ⚠️
compass/llm/calling.py 92.10% 2 Missing and 1 partial ⚠️
compass/validation/content.py 93.93% 1 Missing and 1 partial ⚠️
compass/web/search.py 86.66% 1 Missing and 1 partial ⚠️
compass/pipeline/jurisdiction.py 96.87% 1 Missing ⚠️
compass/plugin/ordinance.py 95.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #606      +/-   ##
==========================================
+ Coverage   72.21%   74.94%   +2.72%     
==========================================
  Files          82       84       +2     
  Lines        8549     9190     +641     
  Branches      935     1005      +70     
==========================================
+ Hits         6174     6887     +713     
+ Misses       2195     2115      -80     
- Partials      180      188       +8     
Flag Coverage Δ
unittests 74.94% <91.60%> (+2.72%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rajeee rajeee left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer guide: one note per changed source file, saying what changed and why the existing structure couldn't do it. Files that only wrap existing calls in logging context say so. Tests and docs are not annotated.

Comment thread compass/utilities/event_log.py Resolved
Comment thread compass/utilities/workflow.py Resolved
Comment thread compass/data/workflow.json5 Resolved
Comment thread compass/utilities/timing.py Resolved
Comment thread compass/llm/calling.py Resolved
Comment thread compass/plugin/ordinance.py Resolved
Comment thread compass/pipeline/extraction/base.py Resolved
Comment thread compass/web/website_crawl.py Resolved
Comment thread compass/extraction/apply.py Resolved
Comment thread compass/pipeline/collection/base.py Resolved
Comment thread compass/pipeline/data_classes.py Resolved
Comment thread compass/pipeline/runtime.py Resolved
Comment thread compass/services/base.py Resolved
Comment thread compass/services/provider.py Resolved
…osts; keep HTTP recording in the event log module

@ppinchuk ppinchuk left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks!

@ppinchuk
ppinchuk merged commit 75c018a into main Nov 17, 2026
35 checks passed
@ppinchuk
ppinchuk deleted the add-event-logging branch November 17, 2026 17:16

This branch was previously deployed

2 inactive deployments
codecov — 7c426cd0 Deployed Nov 17, 2026 by rajeee via Codecov #696
pypi — 7c426cd0 Deployed Nov 17, 2026 by rajeee via Build, check, and optionally upload wheels #528
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants