Skip to content

Add opt-in LLM priority search crawler - #607

Open
rajeee wants to merge 21 commits into
mainfrom
add-priority-crawler
Open

rajeee wants to merge 21 commits into
mainfrom
add-priority-crawler

Conversation

@rajeee

@rajeee rajeee commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Setting priority_search replaces the standard search-engine step with
an LLM-ranked crawl: search results are ranked, pages are visited in
priority order, each page is assessed as an ordinance document, and
its best links are queued. Documents the crawler confirms as adopted
legal text skip the legal-text check during filtering.

Off by default; omitting priority_search keeps the existing behavior.

Also fixes Service.process_using_futures raising InvalidStateError when a caller stopped waiting (for example, a crawler page that timed out): the late result is now dropped.


Stack created with GitHub Stacks CLI • Give Feedback 💬

@codecov-commenter

codecov-commenter commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.22720% with 52 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.39%. Comparing base (75c018a) to head (7efa9aa).

Files with missing lines Patch % Lines
compass/web/priority_crawl.py 94.43% 21 Missing and 10 partials ⚠️
compass/scripts/download.py 53.84% 18 Missing ⚠️
compass/pipeline/collection/steps.py 81.25% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #607      +/-   ##
==========================================
+ Coverage   74.94%   76.39%   +1.45%     
==========================================
  Files          84       85       +1     
  Lines        9190     9844     +654     
  Branches     1005     1080      +75     
==========================================
+ Hits         6887     7520     +633     
- Misses       2115     2120       +5     
- Partials      188      204      +16     
Flag Coverage Δ
unittests 76.39% <92.22%> (+1.45%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread compass/services/base.py Resolved
Comment thread compass/web/file_loader.py Resolved

@ppinchuk ppinchuk left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few questions/comments. Hopefully nothing outrageous or too painful, but LMK

Comment thread compass/pipeline/collection/steps.py Resolved
Comment thread compass/utilities/url.py Resolved
Comment thread compass/web/file_loader.py Resolved
Comment thread compass/web/file_loader.py Resolved
Comment thread compass/web/file_loader.py Resolved
Comment thread compass/web/priority_crawl.py Resolved
Comment thread compass/web/priority_crawl.py Resolved
Comment thread compass/web/priority_crawl.py Resolved
Comment thread compass/web/search.py Outdated
Comment on lines +31 to +82
async def search_ordinance_candidates(
queries,
results_per_engine=10,
url_ignore_substrings=None,
url_keep_substrings=None,
**se_kwargs,
):
"""Merge each engine's top results across queries with provenance

Each engine's per-query results are interleaved by rank (every
query's first result, then every query's second, ...) and the first
`results_per_engine` unique URLs that pass the URL filters are kept.
The configured ``search_engines`` are used, or Google and
DuckDuckGo through SerpAPI when none are configured.
"""
if not se_kwargs.get("search_engines"):
se_kwargs["search_engines"] = list(_PRIORITY_SEARCH_ENGINES)
for query in queries:
logger.info("Search query: %s", query)
results = await search_all_se(
queries,
num_urls=10,
on_search_hook=record_search_attempt,
**se_kwargs,
)
url_filter = URLPartFilter(url_ignore_substrings, url_keep_substrings)
candidates = {}
for engine_results in results:
for result in _top_engine_results(
engine_results, results_per_engine, url_filter
):
url = canonical_url(result["url"])
entry = candidates.setdefault(url, {"url": url, "sources": []})
entry["sources"].append(result)
return list(candidates.values())


def _top_engine_results(engine_results, limit, url_filter):
"""Interleave one engine's query results by rank and keep the top"""
selected, seen = [], set()
for same_rank in zip_longest(*engine_results):
for result in filter(None, same_rank):
url = canonical_url(result["url"])
if url in seen or url_filter.blacklist_match(url) is not None:
continue
if len(selected) == limit:
return selected
seen.add(url)
selected.append(result)
return selected


Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the benefit of this over the holistic search? I think the holistic search does this and more

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mostly wanted to keep them separate as I iterate on the implementation, but, yeah, I should probably integrate what I need into existing framework. Let me see.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea if we can unify and de-duplicate, that would be great, but also if there is a solid reason for keeping them separate, I am open to hearing it

Comment thread docs/source/dev/README.rst Resolved

@ppinchuk ppinchuk left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, I think just bump to elm v0.0.54 (+related changes to that) and we are good to go!

This branch was previously deployed

2 inactive deployments
codecov — 7efa9aac Deployed Nov 18, 2026 by rajeee via Codecov #712
pypi — 7efa9aac Deployed Nov 18, 2026 by rajeee via Build, check, and optionally upload wheels #544
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants