OpenHuman-shaped simulation, hardening & benchmark suite for tinymemory on local CortexDB #251

Why

Today memory_eval runs 12 hand-written scenarios, about 53 probes, one after another, with substring scoring. It drives the library's idea of the lifecycle, not OpenHuman's. OpenHuman calls a different surface: pre_turn_dated/pre_turn_resumed under a 1500 ms timeout, logged_reply tool lines, its own memory tool (recall|fetch|learn|forget) instead of MemoryTools, a scrub guard before every store, Brain::ingest_with(Accepted), collect_items → store_many, and the v1 import loop with its own retries. Much of that surface has never run end-to-end against a live engine. We want a suite that:

1. replays realistic OpenHuman workloads through that exact surface, against a local CortexDB on real OpenRouter models;
2. scores recall, accuracy, latency, cost and leakage with noise-aware comparisons;
3. turns gaps into tracked findings and regression tests.

CortexDB only for now. Section H documents how any other engine is benchmarked the same way.

What already exists (build on it, don't duplicate)

• crates/tinymemory-integrations/examples/memory_eval/:
  • 12 scenarios (scenarios.rs); KPIs (kpi.rs) covering pack hit, MRR, extractive and LLM answers, captured, leaks, cost via v1/admin/usage, and p50/p95;
  • compare with noise bands, and JSON and markdown output.
• scripts/memory-eval.sh runs the eval with MODELS=openrouter on a throwaway compose stack.
• scripts/memory-flag-sweep.sh runs 17 flag profiles and checks the key's budget first.
• integration/cortexdb/ holds the compose file, cortex.toml, mock_inference.py (hash embeddings, no extraction) and flags/*.env.
• tests/live_cortexdb.rs and tests/live_cortex_lifecycle.rs are the live contract and lifecycle tests.
• src/cortex/testing/ is the in-process HTTP double, with fault knobs.
• crates/tinymemory-api/tests/conformance_reference.rs injects faults into the conformance suite.
• Last recorded real-model baseline (docs/evals/cortex-flags.md): pack hit 96%, model answer 87%, planted conflicts flagged 0%, fresh-first 57%, synthesis gain +0 pp, $0.43/run, pre_turn p50/p95 1.1/1.5 s, which is already at OpenHuman's 1500 ms timeout.

A. OpenHuman host simulator (--host openhuman profile in memory_eval)

The dependency direction is one-way. OpenHuman imports tinymemory (vendored at vendor/tinymemory, pinned at v1.27.1 against the current v1.27.2). tinymemory must never depend on openhuman-core, not even as a dev-dependency or from an example. A cycle would break both builds and the vendoring.

• The simulator therefore mirrors OpenHuman's memory shim inside the tinymemory eval. It re-implements the few hooks below, with OpenHuman's defaults as named constants and a parity table in the docs that points at each OpenHuman source line.
• Scenarios and the turn script are data files (JSON) under the eval. OpenHuman may later load them in its own tests, which keeps the arrow pointing OpenHuman → tinymemory and lets it check parity from its side.
• Runtime loop guard. The <memory-context> pack injected into a prompt must never be written back as memory. That would be a recall → store → recall echo that amplifies itself turn after turn. A probe asserts that stored turns and ingested items never contain pack text, and that pack size and duplicate rate stay flat over a 500-turn run.

A mocked OpenHuman turn is the unit of simulation:

user message → pre_turn_* (with the 1500 ms timeout) → model step → zero or more tool calls, including the memory tool and a fake tool with canned results → reply → post_turn(logged_reply) → enqueue the returned jobs → BackgroundRunner.

The model step is pluggable:

• mock (default, CI): deterministic and scripted. It chooses tool calls and replies from the scenario script and answers extractively from the pack.
• live: an OpenRouter chat model given OpenHuman-style system and memory-context prompts. It records tokens and cost and lets the model decide when to call memory.

The same scenario runs in both modes. Comparing them separates retrieval quality from model behaviour.

• Policy: AgentMemory::with_policy with OpenHuman's defaults: budget 1200, learnings 8, brain 6, history 6, team 0, beliefs every 10 turns, build_delay_secs 300.
• Turn hooks: pre_turn_dated/pre_turn_resumed wrapped in tokio::time::timeout(1500ms) (a timeout counts as an empty pack and is recorded).
  • post_turn uses logged_reply (a "tool → result" line appended to the reply).
  • Turn indices are 2n/2n+1.
  • The pack goes inside <memory-context>.
  • recall_for_compaction runs under an 8000 ms timeout.
• Writes: every store goes through a safety::scrub_item_with guard, as in OpenHuman's guard.rs.
• Model tool: a memory tool with recall/fetch/learn/forget/erase/list, calling the engine directly as ops.rs does.
• Jobs: BackgroundRunner with a persisted queue, plus hand-built BuildBeliefs after a source sync.
• Ingestion:
  • Brain::ingest_with(WriteOptions::accepted());
  • reader_for_request + collect_items → store_many;
  • LegacyWorkspace::open(..).skip_connector_syncs(true) + items_from with a retry loop.
• Both wires: direct cortexdb and hosted tinyhumans when its env is set. OpenHuman defaults to hosted, and the lifecycle has never run live on it.

B. Higher-complexity OpenHuman scenarios

Each scenario gets probes tagged lexical/paraphrase/temporal/multi-hop, with expect/accept/stale/forbidden. A generator (seeded, deterministic) can scale any scenario from S to L (tens → thousands of events).

1. Coding session. A multi-hour session in a repo: file reads, failing test output, stack traces, diffs and decisions inside tool results.
   • Probes: "why did we revert X", "which file holds the retry logic", "what was the flaky test".
   • Covers tool-result retention (known gap: ToolCallRef keeps only name/id).
2. Long-running task (task_manager/goals). A goal worked over a simulated 14 days with checkpoints, blockers, a changed plan, and resumes through pre_turn_resumed.
   • Probes: current status, why the plan changed, what is still blocked.
   • Tests "newest state wins" and multi-day aging.
3. Recall after compaction. Threads of 200, 500 and 1000 turns, compacted repeatedly with a sliding in_prompt_from.
   • Facts planted only in the compacted region must come back.
   • Invariant: nothing at or after in_prompt_from, and never the logged turn.
4. Orchestrator → subagents. The planner, critic and summarizer fan out.
   • Workflow and service-sandbox writes must stay out of chat packs.
   • The team section must not echo the agent's own turns (known follow-up).
5. Multi-channel inbox. Slack, Telegram and email threads with several senders and observed_actor/subject attribution.
   • Probes: "what did Alice ask for", and cross-thread disambiguation.
6. Morning briefing / cron triggers. Dated recall such as "what happened yesterday" and "last Tuesday's meeting" via refers_to, plus trigger triage over noisy events.
7. Document-heavy brain.
   • Real PDF, DOCX, PPTX, XLSX, Markdown and code fixtures through ConverterChain/OfficeConverter and the folder reader.
   • Web and RSS readers against a local fixture HTTP server.
   • Score extraction fidelity: tables, multi-page, long docs split into chunks.
8. Legacy v1 import → recall. Import a synthetic v1 workspace, then ask questions only the imported data answers.
9. Preference drift and corrections. The user changes preferences and corrects the agent; the latest must win, and the stale value must never come first.
10. Multi-hop and temporal "as of". "Who owned billing when the outage happened?" Add a temporal KPI.
11. Right to be forgotten. A user asks the agent to forget X through the tool, or the host erases a source. X must be absent from the pack, recall answer, beliefs, facts, export and context.md, immediately and again after enrichment and belief builds.
12. Needle in a haystack, at scale. Today's needle_in_noise is one rule among 24 threads. Extend it with:

• a single needle in 100, 1k and 10k events, swept by depth (early, middle, late) and age (same day vs 30 days old);
• multi-needle (3–5 facts that must be combined);
• needles hidden in a document page, a tool result, an imported v1 item and a compacted thread region;
• near-duplicate decoys and paraphrased needles.

KPIs per (size × depth): pack hit, rank/MRR, answer accuracy and pre_turn latency. Report them as a heatmap table so recall decay with scale is visible.
13. Multi-tenant org. Org root, several users and shared agents. Cross-user and cross-org probes must never leak.

C. KPIs and harness upgrades

• Latency:
  • p50/p95/p99 per call (pre_turn, post_turn, compaction, ingest, recall tool);
  • cold vs warm;
  • timeout rate against OpenHuman's 1500 ms and 8000 ms budgets;
  • write→visible lag and write→extracted lag per item.
• Cost:
  • USD and tokens per turn, per write, per ingested MB and per correct answer, split by role (enrichment, extraction, answer, verifier);
  • an in-eval budget guard (--max-usd) that reads OpenRouter limit_remaining before and during a run.
• Accuracy:
  • keep substring scoring;
  • add an optional LLM-judge (--judge) with recorded rationale;
  • a paraphrase/lexical/temporal/multi-hop breakdown;
  • planted conflicts across more than the single "refund" case.
• Pack invariants: budget adherence (with a real tokenizer as well as 4 chars/token), dedupe, no logged turn, nothing at or after in_prompt_from.
• Leakage: leak rate per channel (pack, answer text, beliefs, facts, team section, context.md), with forbidden strings checked across all of them.
• Deletion: residual rate after forget or erase, measured immediately and after enrichment.
• Concurrency: N parallel agents and tenants, with throughput and tail latency while enrichment runs; pre_turn racing post_turn on the same thread.
• Repeats: a real-model REPEAT ≥ 3 to measure the actual noise band (never measured so far).

D. Leakage and isolation audit — simulate and confirm

Code reading surfaced these. Each needs a live simulation, then a regression test, then a fix issue if it is confirmed.

1. Interior root repeat. Hosted parse_below locates the root with rposition (cortex/envelope/layout.rs:378-381). A namespace such as ws:main/user:42 under root user:42 can parse as the root.
   • Risk: root-inherited reads/beliefs, and erase-at-root reaching that scope (engine/scopes.rs:156,184).
2. Recall answer text ignores metadata filters. Only citations are filtered; the answer is grounded on a whole pack plus facts/beliefs (engine/recall.rs:16-28,52,71).
3. Belief hits are not re-checked against reach (engine/beliefs.rs:79,145-160).
4. Derived layers survive forget/erase. cascade: "redact_events" is trusted but unverified against real CortexDB, especially for beliefs that cite several events (log/forget.rs:28).
5. Fail-open defaults:
   • memory_forget with no reach deletes ids globally, service sandboxes included (tools/write/mod.rs:102);
   • ToolScope::default() and ContextSpec.reach: None are unscoped;
   • Facet::narrow(Namespace) replaces a confined reach (explore/mod.rs:158).
6. Unscrubbed writes and prompt injection.
   • The safety scrub runs only in OpenHuman's guard, never in tinymemory itself.
   • It skips tags, tool_calls, thread/agent ids, repo and source.id.
   • Recalled prose is rendered raw and can forge ## headings or --- frontmatter in the pack (recall/render.rs:159).
7. Partial forget/erase on transport failure returns an error with no report of what was already deleted (engine/forget.rs:71-80, engine/erase.rs:47-52).
8. Backend error bodies (up to 300 chars) reach logs. If a 4xx echoes content, user text is logged (transport/failure.rs:161,192-215, logged at store.rs:212 and recall/gather.rs:112).
9. Sticky downgrades.
   • One InvalidRequest on a dated recall turns date hints off for the life of the process (engine/fetch.rs:185-188).
   • Attribution is turned off after one refusal.
10. Silent truncation beyond 1000 scopes on lenient reads (log/read.rs:141-150).
11. Segment::sanitized collisions (a valid id spelled like a sanitized one; ids over 119 chars rely on a 32-bit FNV), with PII left in scope paths (namespace/mod.rs:126-145). Follows up Namespace addressing: non-injective sanitization, PII-redacted section prefixes, and unspecified recall(None) #116.
12. Path digests are unsalted SHA-256, so they can be reversed by dictionary (envelope/labels.rs:33).

The conformance suite should gain recall-answer and beliefs isolation checks across sibling namespaces, with new fault variants in conformance_reference.rs.

E. Fuzzing and property tests

There are none today. Add proptest (dev-dependency) for:

• Segment::sanitized injectivity;
• layout render/parse round-trip, including an interior root repeat;
• MetaFilter/Reach::admits;
• tool-args parsing (host_fixed_key recursion has no depth bound);
• TimeHint (no span limit).

Add cargo-fuzz targets for envelope decode/rebuild and tool-call JSON. Add chaos runs that use the HTTP double's fault knobs (429/503, apply-then-fail, hidden listings), and on live, a proxy that drops or delays requests.

F. Mock fidelity (so CI measures more than wiring)

• mock_inference.py: deterministic semantic-ish embeddings (for example a hashed bag of words or char n-grams) and a minimal rule-based extractor, so facts and beliefs appear offline.
• The Rust double gains v1/facts, v1/conflicts, v1/admin/usage and /admin/ready, so the inspector runs against it.
• A CI smoke job runs memory_eval --host openhuman on the mocks. Real-model runs stay manual or nightly behind OPENROUTER_API_KEY.

G. Running it locally

• MODELS=openrouter OPENROUTER_API_KEY=… scripts/memory-eval.sh brings up cortexdb:v0.10.4 on :3145 with real models.
• Mind the shared key's daily cap: check limit_remaining first (the sweep already does).
• Results go to target/memory-eval/<label>.{md,json}, and memory_eval compare gives the deltas.

H. Docs: benchmarking another engine's integration

Add docs/evals/engine-integration.md:

1. Pass tinymemory_api::conformance::run(engine), including the new isolation checks.
2. Register the engine in registry so memory_eval --engine <id> and --host openhuman can drive it.
3. Run the scenario suite with REPEAT ≥ 3 and compare against the CortexDB baseline.
4. Meet the release gates.
5. Run the leakage, deletion and fuzz suites.

Also refresh docs/evals/agent-memory.md (it predates the 12-scenario harness) and add the missing integration/cortexdb/flags/README.md.

Proposed gates (initial, to be tuned after the first real-model baseline)

Suggested order (sub-issues)

1. A: the host simulator and the mocked OpenHuman turn (mock model first), plus the C latency/timeout/cost upgrades and the loop guard.
2. D: confirmation runs (fix issues per confirmed risk).
3. B scenarios 1–3: coding, long-running, compaction.
4. B scenarios 4–13, plus the generator for scale.
5. C concurrency and the LLM judge.
6. E: fuzz and proptest.
7. F: mock fidelity and the CI smoke job.
8. H: docs.

Out of scope

• Implementing other engines.
• Changes in the OpenHuman repo, beyond noting the vendored-pin gap.