Repository navigation
Conversation
ad2a959 to
0715481
Compare
0715481 to
cd90bb5
Compare
cd90bb5 to
dea07a2
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #607 +/- ##
==========================================
+ Coverage 74.94% 76.39% +1.45%
==========================================
Files 84 85 +1
Lines 9190 9844 +654
Branches 1005 1080 +75
==========================================
+ Hits 6887 7520 +633
- Misses 2115 2120 +5
- Partials 188 204 +16
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
dea07a2 to
f04bd9d
Compare
f04bd9d to
bb8f76e
Compare
3d045f8 to
0f60752
Compare
…e-block documents by topic
…P description from the event log module)
ppinchuk
left a comment
There was a problem hiding this comment.
A few questions/comments. Hopefully nothing outrageous or too painful, but LMK
| async def search_ordinance_candidates( | ||
| queries, | ||
| results_per_engine=10, | ||
| url_ignore_substrings=None, | ||
| url_keep_substrings=None, | ||
| **se_kwargs, | ||
| ): | ||
| """Merge each engine's top results across queries with provenance | ||
|
|
||
| Each engine's per-query results are interleaved by rank (every | ||
| query's first result, then every query's second, ...) and the first | ||
| `results_per_engine` unique URLs that pass the URL filters are kept. | ||
| The configured ``search_engines`` are used, or Google and | ||
| DuckDuckGo through SerpAPI when none are configured. | ||
| """ | ||
| if not se_kwargs.get("search_engines"): | ||
| se_kwargs["search_engines"] = list(_PRIORITY_SEARCH_ENGINES) | ||
| for query in queries: | ||
| logger.info("Search query: %s", query) | ||
| results = await search_all_se( | ||
| queries, | ||
| num_urls=10, | ||
| on_search_hook=record_search_attempt, | ||
| **se_kwargs, | ||
| ) | ||
| url_filter = URLPartFilter(url_ignore_substrings, url_keep_substrings) | ||
| candidates = {} | ||
| for engine_results in results: | ||
| for result in _top_engine_results( | ||
| engine_results, results_per_engine, url_filter | ||
| ): | ||
| url = canonical_url(result["url"]) | ||
| entry = candidates.setdefault(url, {"url": url, "sources": []}) | ||
| entry["sources"].append(result) | ||
| return list(candidates.values()) | ||
|
|
||
|
|
||
| def _top_engine_results(engine_results, limit, url_filter): | ||
| """Interleave one engine's query results by rank and keep the top""" | ||
| selected, seen = [], set() | ||
| for same_rank in zip_longest(*engine_results): | ||
| for result in filter(None, same_rank): | ||
| url = canonical_url(result["url"]) | ||
| if url in seen or url_filter.blacklist_match(url) is not None: | ||
| continue | ||
| if len(selected) == limit: | ||
| return selected | ||
| seen.add(url) | ||
| selected.append(result) | ||
| return selected | ||
|
|
||
|
|
There was a problem hiding this comment.
What is the benefit of this over the holistic search? I think the holistic search does this and more
There was a problem hiding this comment.
I mostly wanted to keep them separate as I iterate on the implementation, but, yeah, I should probably integrate what I need into existing framework. Let me see.
There was a problem hiding this comment.
yea if we can unify and de-duplicate, that would be great, but also if there is a solid reason for keeping them separate, I am open to hearing it
ppinchuk
left a comment
There was a problem hiding this comment.
Awesome, I think just bump to elm v0.0.54 (+related changes to that) and we are good to go!
Setting priority_search replaces the standard search-engine step with
an LLM-ranked crawl: search results are ranked, pages are visited in
priority order, each page is assessed as an ordinance document, and
its best links are queued. Documents the crawler confirms as adopted
legal text skip the legal-text check during filtering.
Off by default; omitting priority_search keeps the existing behavior.
Also fixes
Service.process_using_futuresraisingInvalidStateErrorwhen a caller stopped waiting (for example, a crawler page that timed out): the late result is now dropped.Stack created with GitHub Stacks CLI • Give Feedback 💬