Skip to content

TEST: freeze query clocks in analytics identity checks - #3221

Merged
Richard Lundeen (richlundeen) merged 2 commits into
microsoft:mainfrom
richlundeen:merge-queue-fix/identity-query-clock-20261015
Oct 17, 2026
Merged

Richard Lundeen (richlundeen) merged 2 commits into
microsoft:mainfrom
richlundeen:merge-queue-fix/identity-query-clock-20261015

Conversation

@richlundeen

Copy link
Copy Markdown
Contributor

Description

Freeze the query-control clock in the analytics identity unit tests. Keep the event-loop clock, database operations, 20-second query deadline values, and identity assertions unchanged.

The tests check grouping, filters, facets, pagination, and identifier replacement. They do not check query latency. Their _control() helper uses a real monotonic deadline. CI can consume that budget during correct SQLite work, so QueryControl.check() raises AnalyticsTimeoutException before the identity assertions run.

The fixture replaces only the time reference in pyrit.memory.query_control. It restores the reference after each test. It does not replace the process-wide clock or the remaining and check implementations. A regression test advances the runner clock past the deadline and checks that the identity-test budget stays unchanged.

Category: test-defect. Confidence: high, from the CI traceback, code, and a deterministic local clock-jump reproduction. The defect is present on origin/main.

Five queue entries for five PRs were affected during the window starting at 2026-10-14T00:13:34Z. Example: https://github.com/microsoft/PyRIT/actions/runs/38622454726/attempts/1, attempt 1, build_and_test, main-job (macos-latest, 3.11, pyrit, dev_all), step Run unit tests.

Fingerprints:

  • tests/unit/memory/test_attack_analytics_eval_identity.py::test_evaluation_groups_filters_facets_and_pages_share_result_identity
  • tests/unit/memory/test_attack_analytics_eval_identity.py::test_replacing_atomic_identifier_updates_evaluation_groups_and_graph

Error: AnalyticsTimeoutException: Status Code: 504, Message: Analytics query timed out. Narrow the filters or retry.

Duplicate search: searched both test functions, the test filename, both full fingerprints, and the timeout error; checked open PR file lists, merged PRs from the past 14 days, open issues, and all-state PRs with the automation fingerprint line. No existing PR freezes this test clock. PR #3171 changes target identity rules and adds tests, but leaves the real-time control unchanged. PR #3220 only changes a message-send test. No related open issue was found.

Tests and Documentation

  • Baseline: uv run --no-sync pytest -n 4 --dist=loadfile tests\unit\backend\test_message_send_service.py tests\unit\memory\test_attack_analytics_eval_identity.py -q: one run, 177 passed, zero failed runs. The CI load-dependent failure did not reproduce in this run.
  • Regression before the fixture: uv run --no-sync pytest -n 4 --dist=loadfile tests\unit\memory\test_attack_analytics_eval_identity.py::test_identity_query_deadline_does_not_follow_runner_clock -q: one run, one expected failure with the same AnalyticsTimeoutException.
  • After the fix: uv run --no-sync pytest -n 4 --dist=loadfile tests\unit\memory\test_attack_analytics_eval_identity.py -q: 20 runs of the final file, six passed per run, zero failed runs. An earlier repeat series had seven complete passing runs, zero observed test failures, and one interrupted run. It was stopped to correct a type-only import; the full 20-run series was then started again.
  • uv run --no-sync ruff check tests\unit\memory\test_attack_analytics_eval_identity.py and uv run --no-sync ruff format --check tests\unit\memory\test_attack_analytics_eval_identity.py: passed after moving the type-only Iterator import into the existing type-checking block.
  • uv run --no-sync pre-commit run --all-files: could not complete. Hook environment installation fails when Windows creates a long Jedi stub path for nbQA. Standard, short-drive, and extended-path attempts during this run did not remove the hook setup blocker. No hook was disabled.
  • Validation used Windows and Python 3.12.12. The failing macOS matrix was not reproduced.
  • Added a clock-isolation regression test. No product behavior or documentation changes are required.

This PR is a draft because the full pre-commit run is blocked by the local Windows hook environment.

merge-queue-fix fingerprint: tests/unit/memory/test_attack_analytics_eval_identity.py::test_evaluation_groups_filters_facets_and_pages_share_result_identity

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 18f4b401-48f9-4728-995f-db1431157452
@richlundeen Richard Lundeen (richlundeen) changed the title [DRAFT] TEST: freeze query clocks in analytics identity checks TEST: freeze query clocks in analytics identity checks Oct 15, 2026
@richlundeen
Richard Lundeen (richlundeen) marked this pull request as ready for review October 15, 2026 16:00
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Oct 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 16, 2026
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Oct 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 16, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: edeb4a16-e95f-4da7-8893-9032522d7ada
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Oct 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 17, 2026
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Oct 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 17, 2026
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Oct 17, 2026
Merged via the queue into microsoft:main with commit 11d6f83 Oct 17, 2026
60 checks passed
@richlundeen
Richard Lundeen (richlundeen) deleted the merge-queue-fix/identity-query-clock-20261015 branch October 17, 2026 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants