Repository navigation
Firestore Document Reads persistently above alert threshold — day 25, 10 read-reduction PRs merged but still firing, no assignee #11282
Description
Problem
GCP alert "Firestore Instance - Document Reads > 10k/s" (project based-hardware, metric firestore.googleapis.com/document/read_count, type QUERY) has been breaching since Aug 7, 2026. The trend is worsening — daily average reads climbing from ~10k/s (Aug 8) to ~14k/s (Aug 21).
This issue has no assignee (day 28).
Breach data (GCP Monitoring API, updated Aug 22)
233 breach hours out of 609 total since Aug 7 (38% of all hours above threshold).
| Day | Breach hours | Peak | Floor | Avg |
|---|---|---|---|---|
| Aug 7 | 3/24 | — | — | — |
| Aug 8 | 24/24 | — | — | ~10,000/s |
| … | ||||
| Aug 17 | 23/24 (96%) | 15,753/s | 9,930/s | 12,607/s |
| Aug 18 | 20/24 (83%) | 18,438/s | 8,750/s | 12,972/s |
| Aug 19 | 22/24 (92%) | 13,768/s | 9,546/s | 11,847/s |
| Aug 20 | 19/24 (79%) | 18,599/s | 4,545/s | 13,438/s |
| Aug 21 | 23/24 (96%) | 19,061/s | 8,353/s | 14,431/s |
| Aug 22 | 8/9 so far (89%) | 14,374/s | 4,515/s | 11,878/s |
Read type breakdown: QUERY dominates (max 18.3k/s). NOT_FOUND up to 1.1k/s, LOOKUP up to 400/s.
Trend
- Daily average is climbing: ~10k/s (Aug 8) → ~13k/s (Aug 17) → ~14.4k/s (Aug 21)
- Breach coverage is persistent: 79–96% of hours per day above threshold (Aug 17–22)
- New worst peak: 19,061/s on Aug 21 (~91% above threshold)
Investigation findings (@Git-on-my-level, Aug 13)
Pulled prod Cloud Monitoring data for Aug 7 18:00–20:00 window:
- QUERY type = 97% of all document reads (~9,458/s of 9,762/s)
- ~27 docs per RunQuery (68M reads / 2.5M RunQuery calls)
- GCP metrics have no collection-level attribution —
module/versionlabels are__unknown__ - Firestore data-access audit logs are not enabled
Impact (verified Aug 25)
Reliability — observed degradation
Firestore is absorbing the load (99.99%+ OK rate), but tail latency degraded significantly since the breach started:
| Metric | Pre-breach (Aug 6) | During breach (Aug 7–25) | Change |
|---|---|---|---|
| Firestore read latency p50 | 6.8ms | 6.0–6.9ms | flat |
| Firestore read latency p95 | 15.2ms | 32.5–38.1ms | 2x |
| Firestore read latency p99 | 20.5ms | 52.0–68.3ms | 3x |
| Backend service p99 | 3,100ms | 4,300–6,200ms | 2x |
| Backend-sync p95 | 26s | 93s | 3.6x |
| Backend 5xx (worst day, Aug 14) | baseline | 1.74% error rate | correlated |
Daily read volume: pre-breach ~750M/day → during breach 830M–1.27B/day (peak Aug 20: 1.27B). Reads up 40–70% from baseline.
Aug 13–15 5xx spike: Backend 5xx jumped to 1.74% (Aug 14), correlating with a Firestore FAILED_PRECONDITION spike (65K–365K/day). The spike resolved but reads never dropped back to baseline.
Scaling risk
Firestore auto-scales via range sharding, but has no circuit breaker — no gradual degradation between "working" and "overwhelmed." DoiT's production testing documented: violating the 500/50/5 ramp-up rule caused 95% failure rate with latency spikes to 60s. Google's own status page documents a nam5 incident where one project's traffic spike caused elevated latency on up to 50% of all nam5 projects for 73 minutes.
Current reads are 97% QUERY type (~27 docs per RunQuery), concentrating on hot key ranges — the hardest pattern for Firestore to auto-scale.
Cost
An enterprise contract covers Firestore reads at a fixed rate regardless of volume — no incremental cost at current volumes. However, at on-demand rates the current sustained ~14k reads/s would cost approximately 6x the contracted rate, and at the projected 4-week rate (~25k/s) would reach ~11x. The contract absorbs this waste today, but the underlying read volume represents significant inefficiency.
Read-reduction PRs merged
Phase 1: Jul 28–Aug 7 (6 PRs)
| PR | What it fixed | Merged |
|---|---|---|
| #10750 | Bound Firestore list reads (stopped GET 504s) | Jul 28 |
| #10778 | action-items due-range Firestore index | Jul 28 |
| #10781 | action-items composite index | Jul 29 |
| #10902 | Tighten knowledge graph read budget | Jul 30 |
| #11062 | Deduplicate all-time usage reads | Aug 4 |
| #11197 | Bound chat-quota month read to current month | Aug 7 |
Phase 2: Aug 18–21 (10 PRs)
| PR | What it fixed | Merged |
|---|---|---|
| #11731 | Declare opposite-direction Firestore list-scan indexes | Aug 18 |
| #11800 | Bound the memory list page's suppressed-row walk | Aug 18 |
| #11816 | Stop the legacy memory offset read costing O(rounds × prefix) | Aug 18 |
| #11819 | Read the historical memory prefix in one query, not an offset walk | Aug 18 |
| #11830 | Stop listing GETs decrypting/overscanning past the 30s GET timeout | Aug 18 |
| #11849 | Push the completed filter server-side in the action-item ID scan | Aug 19 |
| #11851 | Serve app search from the shared public-app cache | Aug 19 |
| #11858 | Prevent large list GETs from exhausting the 30s request budget | Aug 21 |
| #11949 | Window the chat-first materialization health scan | Aug 20 |
| #11324 | Implement caching for memory get_prompt_data | Aug 20 |
16 read-reduction PRs merged total. Alert is still firing — the Aug 21–22 data shows the trend has not reversed.
Related issues
| Issue | Status |
|---|---|
| #9889 | Open, P1, no assignee — Firestore read amplification audit |
| #10817 | Closed Aug 20 — consolidated into this issue |
| #10950 | Closed Aug 20 — consolidated into this issue |
Deploy correlations noted (timing only — not confirmed causation)
| PR | Merged | Change |
|---|---|---|
| #9581 | Aug 12 | Enforce free-tier chat quota at every managed-AI boundary |
| #11584 | Aug 14 | Default MEMORY_MODE=write — enables memory intake for all users |
| #11590 | Aug 14 | One MEMORY_ENABLED flag for create+list |
| #11630 | Aug 15 | Chat quota accounting before billable turns |
Activity
[-]Firestore Document Reads sustained above alert threshold — 3 alerts in 11 days, escalating despite 6 merged fixes[/-][+]Firestore Document Reads sustained above alert threshold — 43 incidents in 30 days, currently continuous for 32+ hours[/+][-]Firestore Document Reads sustained above alert threshold — 43 incidents in 30 days, currently continuous for 32+ hours[/-][+]Firestore Document Reads sustained above alert threshold — 58 incidents since Jul 28, still breaching daily, no assignee[/+]beastoin commentedon Aug 12, 2026
Status update — Aug 12
Still firing. Latest breach: Aug 12 03:04 UTC (~4% above threshold).
Updated numbers (from GCP Monitoring API)
Progress since issue was filed (Aug 9)
What this issue needs
An owner who will:
This has been breaching for 16 days. The hot-path patching approach (6 PRs) did not work. The issue needs a systematic, data-driven investigation — and someone assigned to do it.
Git-on-my-level commentedon Aug 13, 2026
Prod
read_countfor Aug 7 18:00–20:00 UTCPulled live from Cloud Monitoring on project
based-hardware. No Firestore document contents were read.Collection breakdown
Not available from GCP.
firestore.googleapis.com/document/read_countonly has labelstype,module,version. In this windowmodule/versionare both__unknown__. There is no collection label.data_access/firestore.googleapis.comactivity is empty), so collection cannot be reconstructed from logs either.Granting
roles/monitoring.viewerwould not change that: the metric itself has no collection dimension.What the alert metric does show (document reads)
firestore.googleapis.com/document/read_count,ALIGN_SUMover 60s, then summed across the 2h window:typeQUERYLOOKUPNOT_FOUNDQUERY is already at/above the 10k/s alert for most of this window. The 35h incident start (19:39Z) lines up with the 60s QUERY peak in this slice (12,668 /s at 19:38Z).
API request counts (not the same as document reads)
firestore.googleapis.com/api/request_counton resourcefirestore.googleapis.com/Database(thedatastore_requestresource duplicates the same series; do not double-count):api_methodRunQueryBatchGetDocumentsCommitBeginTransactionRunAggregationQueryListDocumentsListCollectionIdsGetDocument/UpdateDocumentImplied: ~27 QUERY document reads per
RunQueryrequest (68,095,833 / 2,493,451). That is compatible with a resumed query reusing its originallimit(), but it does not by itself prove that path is the whole 10k/s — Monitoring cannot attribute those QUERY reads to a collection or call site.How to get collection attribution later
Needs something the current project does not have, e.g. Firestore data-access audit logs (expensive), an app-emitted counter labeled by collection, or a one-off maintainer export. Until then, collection share cannot be an acceptance-criteria input from prod GCP metrics.
beastoin commentedon Aug 17, 2026
Status update — Aug 17
Still not fixed. Alert fired again Aug 12 03:08 UTC (~0.2% above threshold). This is now day 21 since the first sustained breach on Jul 28.
Progress
One investigation comment from @Git-on-my-level (Aug 13) — the first real prod data on this issue:
module/versionlabels are__unknown__Key finding: The problem is in RunQuery fan-out. Each query returns ~27 documents on average. The fix must target the queries themselves — which collections/routes are issuing the most RunQuery calls.
What has NOT happened
What is needed to unblock
Since GCP metrics lack collection attribution, the next step requires one of:
Without collection attribution, no data-driven fix is possible. This needs an owner.
(Updated incident numbers from GCP pending — will post when available.)
[-]Firestore Document Reads sustained above alert threshold — 58 incidents since Jul 28, still breaching daily, no assignee[/-][+]Firestore Document Reads persistently above alert threshold — day 21, 24/24h breaching, floor rising, no assignee[/+]beastoin commentedon Aug 17, 2026
Status update — Aug 17 (day 21)
Worsening. The situation has changed from intermittent spikes to persistent breaching.
Updated numbers (GCP Monitoring API)
New: deploy correlation
The timing of the Aug 13–14 peak spike correlates with backend deploys that activated new Firestore read paths:
MEMORY_MODEfromofftowrite, enabling memory intake for every UID. This was previously disabled for all of production as a Gate 3 dark-deploy leftover.Still blocked
This needs an owner who will investigate the deploy correlation and identify the top read contributors.
fix(backend): stop the legacy memory offset read costing O(rounds x p…
fix(backend): read the historical memory prefix in one query, not an …
beastoin commentedon Aug 22, 2026
Status update — Aug 22 (day 25)
Alert still firing. Latest incident: Aug 21, 2026 02:22 UTC at 10,485/s (threshold: 10,000/s).
Changes since last update (Aug 17)
Consolidated tracking:
Read-reduction PRs merged Aug 18–21:
What has not changed:
[-]Firestore Document Reads persistently above alert threshold — day 21, 24/24h breaching, floor rising, no assignee[/-][+]Firestore Document Reads persistently above alert threshold — day 25, 10 read-reduction PRs merged but still firing, no assignee[/+]beastoin commentedon Aug 25, 2026
Impact assessment added (day 28)
Reliability impact (verified from GCP Monitoring API + Cloud Run logs, Aug 7–25)
Firestore itself is not returning errors — 99.99%+ OK rate, zero meaningful RESOURCE_EXHAUSTED or DEADLINE_EXCEEDED. But tail latency degraded significantly:
The latency increase cascades: each Firestore query taking 2–3x longer compounds across the dozens of queries per request, doubling backend p99 and making sync 3.6x slower.
Aug 13–15 5xx spike: Backend 5xx jumped to 1.74% (Aug 14), correlating with a Firestore FAILED_PRECONDITION spike (65K–365K/day). The spike resolved but reads never dropped back to baseline.
Scaling risk
Firestore auto-scales via range sharding, but has no circuit breaker. DoiT's production testing documented: violating the 500/50/5 ramp-up rule caused 95% failure rate with latency spikes to 60s. Google's own status page documents a nam5 incident where one project's spike caused elevated latency on up to 50% of all nam5 projects for 73 minutes.
Current reads are 97% QUERY type (~27 docs per RunQuery). These concentrate on hot key ranges, which are the hardest for Firestore to split.
Cost
An enterprise contract covers Firestore reads at a fixed rate regardless of volume — no incremental cost at current volumes. However, at on-demand rates the current sustained ~14k reads/s would cost approximately 6x the contracted rate, and at the projected 4-week rate (~25k/s) would reach ~11x. The contract absorbs this waste today, but the underlying read volume represents significant inefficiency.
Daily read volume
Pre-breach baseline: ~750M reads/day. During breach: 830M–1.27B reads/day (peak Aug 20: 1.27B). Reads up 40–70% from baseline.
SCA-458: cut Firestore memory unbounded queries and NOT_FOUND misses (#…