Repository navigation
Add per-jurisdiction event logs and a workflow map - #606
Merged
Merged
Conversation
Each jurisdiction run writes an events.jsonl recording the searches, downloads, filter decisions, LLM calls and extraction steps it went through. compass/data/workflow.json5 maps every step and branch the pipeline can take, and tests keep that map in step with the code. Also adds elapsed-time logging for long-running operations and records the HTTP exchange behind each document download.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #606 +/- ##
==========================================
+ Coverage 72.21% 74.94% +2.72%
==========================================
Files 82 84 +2
Lines 8549 9190 +641
Branches 935 1005 +70
==========================================
+ Hits 6174 6887 +713
+ Misses 2195 2115 -80
- Partials 180 188 +8
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
rajeee
commented
Nov 10, 2026
rajeee
left a comment
Collaborator
Author
There was a problem hiding this comment.
Reviewer guide: one note per changed source file, saying what changed and why the existing structure couldn't do it. Files that only wrap existing calls in logging context say so. Tests and docs are not annotated.
rajeee
commented
Nov 16, 2026
rajeee
commented
Nov 16, 2026
rajeee
commented
Nov 16, 2026
rajeee
commented
Nov 16, 2026
…osts; keep HTTP recording in the event log module
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each jurisdiction run now writes
<jurisdiction>_events.jsonlnext to its.logfile, recording the searches, downloads, filter decisions, LLM calls and extraction steps it went through. It works forcompass processand for the splitsearch,collect, andextractcommands, which add to the same file. On by default;keep_event_logs: falseturns it off.compass/data/workflow.json5maps every step and branch the pipeline can take, and tests keep that map in step with the code.Also records every HTTP response behind a document download (
http_response, orhttp_errorwhen none came).Depends on ELM
Requires
nlr-elm>=0.0.53(released), which includes NatLabRockies/elm#96: the hooks this PR records through. They areon_search_hook(each search engine attempt),on_response_hookandon_error_hook(each HTTP response, or why none came), andon_fetch_hook(what each document fetch produced).pyproject.tomlraises the minimum version from 0.0.52 to 0.0.53.Errors and retries are recorded so a slow or failed run can be explained from its log:
ConnectionResetError), and each exception lists the file, line and function it passed through, including the code that led to where it was caught.llm_retryevents, with the wait and the error behind each, tagged with the LLM call they belong to; retries of downloads by the same wrapper aredownload_retryevents. A call that succeeds after retrying otherwise looks only slow.compass/servicesnow runs each queued job in its caller's context (asyncio.create_task(..., context=...)); the provider already named each job's task after its caller.Live demo
https://pages.github.nlr.gov/radhikar/htmls/COMPASS/pipeline_replay.html (lab network only)
A browser viewer that replays event logs against the workflow map. It opens with five example data center runs: three that found their ordinance (Atlanta GA, Tucson AZ, Chandler AZ) and two that did not (Provo UT, DeKalb County GA), which show where a search stops. Many LLM calls in these runs were retried, and some failed, because the LLM proxy dropped connections; the viewer shows each retry and error chain, with links to the source lines. Use "Open log folder" to load the logs from your own run. The viewer itself is not part of this PR, and the example logs come from runs that used the priority crawler in #607, so they include crawler events this PR alone does not emit.
First of three stacked PRs: event logging (this one) → priority crawler (#607) → data center plugin (#608).
Stack created with GitHub Stacks CLI • Give Feedback 💬