Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 5 additions & 15 deletions .github/workflows/mcp-parity.yml
Original file line number Diff line number Diff line change
@@ -1,29 +1,19 @@
name: MCP Transport Parity
name: MCP Transport Parity Audit

# Diffs the remote MCP bridge against the stdio package against one live
# AutoMem. See docs/MCP_TRANSPORT_PARITY.md for the contract.
#
# No scheduled run yet, deliberately. While the harness is intentionally red
# against the known gaps, `continue-on-error` leaves the workflow successful
# either way, so a cron job could not distinguish "same known gaps" from "a
# newly published mcp-automem added drift" — it would be an alarm that cannot
# alarm. The weekly drift check lands with the change that turns the harness
# green, where a failure is a real signal.
# Manual migration audit: known gaps in the other five tools keep the full
# comparison red. Required per-PR contract coverage stays in CI's node-test
# job, including the deterministic recall parity tests. Restore automatic
# runs only once the full comparison has a useful passing baseline.

on:
pull_request:
paths:
- 'mcp-sse-server/**'
- 'automem/api/**'
- 'app.py'
workflow_dispatch:

jobs:
parity:
runs-on: ubuntu-latest
# Phase 1 only. The harness is legitimately red until the bridge is moved
# onto the shared MCP surface; this is removed in the same change.
continue-on-error: true
steps:
- uses: actions/checkout@v4

Expand Down
43 changes: 32 additions & 11 deletions docs/MCP_TRANSPORT_PARITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,8 +164,18 @@ rendered `tools/call` text — must be identical.

## How parity is enforced

A differential harness drives both transports against a single
docker-compose AutoMem and diffs them:
### Required PR contract checks

CI's required `node-test` job runs `npm test` in `mcp-sse-server` on every PR.
This keeps the bridge's normal tests and the deterministic
`test/recall-parity.test.js` contract guard: tool definition, API request
mapping, ID fetches, pagination, validation, rendered output and response
budgets. These tests need no live AutoMem stack and remain required.

### Manual live parity audit

The broader differential harness drives stdio and Streamable HTTP against a
single docker-compose AutoMem and diffs them:

```bash
make test-parity
Expand Down Expand Up @@ -198,18 +208,29 @@ It lives in `mcp-sse-server/parity/` with its entry point at
score.

The harness is gated on `AUTOMEM_RUN_PARITY_TESTS=1` because it needs a live
service. Without the gate it skips cleanly, which is what CI's `node-test` job
sees. `.github/workflows/mcp-parity.yml` runs it with the stack up on PRs that
touch `mcp-sse-server/**`, `automem/api/**`, or `app.py`.
service. Without the gate only the live audit skips; the deterministic contract
tests still run in CI's `node-test` job. The full audit is intentionally red
while the other five tools retain known differences, so it does not run on PRs.

To run it in GitHub Actions, manually dispatch **MCP Transport Parity Audit**
(`.github/workflows/mcp-parity.yml`) on the branch to audit. For example, once
this workflow change is on `develop`:

```bash
gh workflow run mcp-parity.yml --ref develop
```

The manual workflow starts its own stack and reports test failures as failures;
it no longer masks them with `continue-on-error`. A red audit currently means
the full migration is incomplete, so inspect its test output to distinguish
known gaps from a new regression. It is not a PR readiness gate.

It brings its own stack up on port **8011**, not the usual 8001. The harness
writes fixtures and bulk-deletes by tag, and a developer running a local
AutoMem install (`~/.automem/server`) already holds 8001 — pointing the harness
at that instance would seed test data into a real memory store.

**There is no scheduled drift run yet, deliberately.** While the harness is
intentionally red against the known gaps, `continue-on-error` leaves the
workflow successful either way, so a cron job could not distinguish "same known
gaps" from "a newly published `mcp-automem` added drift". That would be an alarm
that cannot alarm. The weekly check lands with the change that turns the harness
green, where a failure is a real signal.
**There is no scheduled drift run.** Restore automatic execution only after the
full comparison has a useful passing baseline, so a failure can signal new
drift. Until then, use the manual audit to inspect migration progress and the
required contract tests to catch regressions in the supported recall surface.
Loading