Admin: evidence-based DocumentDb detection, and soft delete it can act on

readme.md

@@ -2516,9 +2516,47 @@ place.
 | **Blobs** | Appears when the table has a `{table}_blobs` sidecar. Lists payloads without ever selecting the blob column, previews images and text, and downloads over a plain HTTP endpoint rather than the Blazor circuit. Only allow-listed raster images render inline; everything else is served as an attachment under `nosniff` + `default-src 'none'`. |
 | **Outbox** | Appears when the database holds [transactional outbox](#transactional-outbox) messages. A database-level screen rather than a type tab: how deep the queue is, whether it is draining (**oldest pending**, which is what separates "busy" from "the processor died"), what is dead-lettered grouped by message type and error, and requeue / purge behind a confirm. **It cannot dispatch** and says so on the page — delivery is your application's `IOutboxDispatcher`, over a transport the tool cannot reach; requeue only makes a message eligible again. |
 | **Import / Export** | Stream a type out as JSON, NDJSON, envelope JSON (round-trippable) or CSV; import JSON or NDJSON back with fail / replace / skip handling for duplicate ids. Also **generates** documents that look like the ones already there — numbers inside the observed range, dates inside the observed span, categorical fields drawn from the set actually used, ids continuing whatever scheme is in place — with a seeded preview that commits byte-identically. |
-| **Assistant** | Appears when a connection has AI configured. A chat that answers questions about the data by composing the same reads the rest of the tool performs, scoped to the type you're looking at or the whole connection. Configured **per connection** — OpenAI, Azure OpenAI, Anthropic, or any OpenAI-compatible endpoint (OpenRouter, Groq, LM Studio, vLLM, Ollama) — so development can use a hosted model while production uses a local one, or none. **It cannot write:** the model is given ten tools and every one is a read, so read-only is a property of the tool surface rather than an instruction in a prompt. |
+| **Assistant** | Appears when a connection has AI configured. A chat that answers questions about the data by composing the same reads the rest of the tool performs, scoped to the type you're looking at or the whole connection. Configured **per connection** — OpenAI, Azure OpenAI, Anthropic, or any OpenAI-compatible endpoint (OpenRouter, Groq, LM Studio, vLLM, Ollama) — so development can use a hosted model while production uses a local one, or none. **It cannot write:** the model is given eleven tools and every one is a read, so read-only is a property of the tool surface rather than an instruction in a prompt. |
 | **Query** | Two modes. **Filter grammar** takes DocumentDb's own string query syntax (`status == 'Shipped' and total:number > 100`) with `Where` / `OrderBy` / `Project` boxes — run through the library's own parser via `store.Collection(typeName)`, not a reimplementation, so the answer matches what your code would get. The compiled SQL is shown beside the results, **Explain** runs the provider's query plan over it (planned, not executed — safe on a read-only connection), unindexed fields in the query are offered as one-click index creates that then re-run and re-explain, and one button drops the SQL into the other mode. **Raw SQL** is whatever you type, in the target database's dialect, with `@name` parameters bound from a JSON box so the types stay honest. |
 
+#### Is this database even a DocumentDb store?
+
+The overview answers that as a verdict rather than leaving it to be inferred from a filtered table
+list — *"DocumentDb store · 3 document tables · 12 types · temporal, vectors, outbox"*, or *"Not a
+DocumentDb database — 42 tables, none carry the envelope"* — with the reasons behind it, the tables it
+created (each showing which documents table it belongs to) and the tables it did not, counted and
+collapsed.
+
+It is evidence, not guesswork. Two catalog reads (`IDatabaseProvider.BuildListTablesSql` and the new
+`BuildListColumnsSql`) describe the whole database in one go, so pointing the tool at a 300-table shared
+schema no longer issues 300 deliberately-failing probes. A table is a *candidate* when it carries all
+five envelope columns and **confirmed** only when something DocumentDb leaves behind corroborates it —
+a JSON-shaped `Data` column, the `(Id, TypeName)` key, `idx_{table}_typename`, its JSON property
+indexes, or (opt-in) a sampled row that really is a document. Envelope-and-nothing-else is reported as
+**probable**, browsable and badged, rather than as a certainty. Sidecars are then found by *computing
+the names DocumentDb would have created* — `IDatabaseProvider.OwnedTableNames(table, type)`, which each
+provider extends with its own engine shadows (SQLite's R\*Tree and FTS5 tables, sqlite-vec's `vec0`
+shadows, DuckDB's full-text source table) — and looking them up. Anything else is foreign, **whatever
+it is called**: a business table named `audit_history`, `customer_blobs` or `geo_spatial_index` is no
+longer reported as one of ours, and a documents table named `orders_history` is no longer hidden as a
+sidecar.
+
+#### Soft delete has to be declared
+
+[Soft delete](#soft-delete) writes no column, table or index — it is an interceptor plus a named query
+filter inside your application — so **the tool cannot discover it**, and until it is told, its delete
+button is a real `DELETE` against a type your application would only ever have flagged. Declare the
+types on the connection (in the connection's settings, or one click from a candidate the Structure tab
+recognises; a host-provided connection declares them in configuration as
+`Shiny:DocumentDb:{name}:SoftDelete:{Type}` = the flag's JSON path, optionally with `PropertyPath` /
+`FlagKind` spelled out) and the tool acts on it: Browse gains a live / deleted / all filter and badges
+flagged rows, delete writes the flag with **delete permanently** as a separate deliberate choice,
+restore clears it to `false` or `null` per the declared kind, and the verdict lists `soft delete`.
+Undeclared types behave exactly as before. The Structure tab only ever *suggests* a flag — a field
+qualifies when its name is in the deleted family **and** its sampled values match one of the two shapes
+`AddSoftDelete` accepts — and a suggestion changes no role, no browsability and no button until it is
+confirmed.
+
 Mark a connection **read-only** and every write path is blocked, including non-SELECT statements in
 the SQL console. The SQL console is otherwise exactly what it looks like — an open prompt against the
 database, with no statement allow-list — so give the tool a database account with the privileges you

skills/shiny-documentdb/SKILL.md

@@ -126,6 +126,9 @@ triggers:
   - ToLower query
   - Math.Abs query
   - IDatabaseProvider
+  - BuildListColumnsSql
+  - OwnedTableNames
+  - SpatialTableName
   - json document
   - JsonNode
   - late-bound
@@ -3401,6 +3404,19 @@ opts.ConfigureDocument<Order>(cfg => cfg.OnBeforeWrite(async (ctx, ct) =>
 
 The nine document providers share one `IDocumentQuery<T>` implementation. A provider derives from public **`DocumentQueryBase<T>`** and implements four members — `Clone()`, `ExecuteAsync(QueryPlan<T>)`, `DeleteMatchingAsync`, `SetPropertyMatchingAsync` — and inherits builder state/immutability, query-filter resolution, all client-side terminals, grouping, cursor paging, string projection, and the set-based-write interceptor plumbing. `ExecuteAsync` returns `QueryExecution<T>.Candidates` (nothing applied server-side), `.Filtered`, `.Complete`, or `.Partial(...)`; the base applies only what the engine did not. **Report push-down honestly** — claiming ordering/paging you did not apply silently returns wrong results. Optional hooks (`SetPropertiesMatchingAsync` for multi-property `ExecuteUpdate`, `ObserveChanges`, `FullTextSearchCore`, `NearestVectorsCore`, `ToQueryString`, `ToCursorPage`) and the aggregate hooks (`CountCore`/`MaxCore`/…) have safe defaults; override only what the engine can do itself. On the options side implement **`IDocumentStoreOptions`** — `Mappings`, `TypeNameResolution`, `Capabilities`, the interceptor pair, and `SerializerOptions`/`EnsureSerializerOptions` — and `ConfigureDocument<T>` plus every cross-cutting feature (soft delete, `cfg.MapJsonSchema`, field encryption) lights up for free. All per-type mapping state lives in the shared `DocumentMappingRegistry`; `Capabilities` is what the configuration validation pass reads to reject mappings your backend cannot honor.
 
+**Two members exist so a tool can identify a database without guessing, and both have working defaults.**
+`BuildListColumnsSql()` returns `(table_name, column_name, data_type)` for the current schema — the
+counterpart to `BuildListTablesSql()`, so one read describes every table instead of one probe per table.
+The default is ANSI `information_schema.columns`; override it where that view is absent or scoped wrong
+(SQLite joins `sqlite_master` to `pragma_table_info`, Oracle reads `user_tab_columns`, MySQL scopes to
+`DATABASE()`). `OwnedTableNames(table, typeName)` yields every table name **this provider would have
+created around** `table` — the default covers `HistoryTableName` / `BlobTableName` / `SpatialTableName` /
+`VectorTableName` (call `IDatabaseProvider.DefaultOwnedTableNames(this, table, typeName)` from an override
+rather than restating them), and a provider adds whatever its own DDL drags in: full-text tables, and any
+engine shadows behind a virtual table. Names only — the tables need not exist. This is what lets a tool
+decide a table is yours by **computing the name and finding it** rather than matching a substring, so
+leaving a full-text or shadow table out of it means someone's admin UI reports it as a foreign table.
+
 Store writes bracket persistence with the shared pipeline on `DocumentProviderBase` (implement `Mappings`, `IdCache`, `ResolveTypeInfo`, `ResolveDocumentTypeName`): `BeginWriteAsync(op, doc, id, typeInfo, ct)` → check `write.Proceed` (false = an interceptor replaced the write; `Remove` returns `write.CancelResult`), take `write.Doc`, get the id with `ResolveInsertId`/`ResolveInsertIdAsync` (insert) or `RequireDocumentId` (update), persist, then `CompleteWriteAsync(write, id, version, changeType, doc, ct)` which runs `AfterWrite` and publishes the change (buffered until commit inside a unit of work). Never re-implement `PublishChange` — the base owns the broadcaster.
 
 ## Soft Delete (`AddSoftDelete<T>`)

src/Shiny.DocumentDb.DuckDb/DuckDbDatabaseProvider.cs

@@ -607,6 +607,20 @@ public object FormatVectorParameter(ReadOnlyMemory<float> vector, VectorMapping
     static string FtsSourceTable(string tableName, string typeName)
         => $"{tableName}_ftssrc_{IDatabaseProvider.SanitizeTypeSuffix(typeName)}";
 
+    /// <summary>
+    /// Adds the per-type full-text source table. DuckDB's fts index is built by
+    /// <c>PRAGMA create_fts_index</c> over a snapshot table this provider materialises, so unlike every
+    /// other name here it is a real table in the main schema and would otherwise read as foreign.
+    /// </summary>
+    public IEnumerable<string> OwnedTableNames(string tableName, string? typeName)
+    {
+        foreach (var name in IDatabaseProvider.DefaultOwnedTableNames(this, tableName, typeName))
+            yield return name;
+
+        if (!string.IsNullOrEmpty(typeName))
+            yield return FtsSourceTable(tableName, typeName);
+    }
+
     public string BuildFullTextProbeSql(string tableName, string typeName)
         => $"SELECT 1 FROM duckdb_tables() WHERE table_name = '{FtsSourceTable(tableName, typeName)}';";
 

src/Shiny.DocumentDb.MySql/MySqlDatabaseProvider.cs

@@ -287,6 +287,11 @@ public string BuildFullTextProbeSql(string tableName, string typeName)
     public string BuildListTablesSql()
         => "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_SCHEMA = DATABASE();";
 
+    // Scoped to the current database for the same reason as BuildListTablesSql - MySQL's
+    // information_schema spans the whole server.
+    public string BuildListColumnsSql()
+        => "SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = DATABASE();";
+
     public string JsonExtract(string column, string jsonPath)
         => $"NULLIF(JSON_UNQUOTE(JSON_EXTRACT({column}, '$.{jsonPath}')), 'null')";
 

src/Shiny.DocumentDb.Oracle/OracleDatabaseProvider.cs

@@ -414,6 +414,11 @@ public IReadOnlyList<string> BuildExplainSql(string sql)
     public string BuildListTablesSql()
         => "SELECT table_name FROM user_tables";
 
+    // user_tab_columns is the current schema's columns - Oracle's information_schema equivalent, and the
+    // only one a connection is guaranteed to be able to read without extra grants.
+    public string BuildListColumnsSql()
+        => "SELECT table_name, column_name, data_type FROM user_tab_columns";
+
     public string JsonExtract(string column, string jsonPath)
         => $"JSON_VALUE({column}, '$.{jsonPath}')";
 

src/Shiny.DocumentDb.Sqlite/SqliteDatabaseProvider.cs

@@ -171,6 +171,46 @@ public string BuildListAllIndexesSql(string tableName)
     public string BuildListTablesSql()
         => "SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%';";
 
+    // sqlite_master carries no column information, so the names come from pragma_table_info - joined as a
+    // table-valued function so the whole database is one read rather than one PRAGMA per table.
+    public string BuildListColumnsSql()
+        => "SELECT m.name, p.name, p.type FROM sqlite_master m JOIN pragma_table_info(m.name) p " +
+           "WHERE m.type IN ('table', 'view') AND m.name NOT LIKE 'sqlite_%';";
+
+    /// <summary>
+    /// Adds what SQLite creates alongside the portable sidecars: the R*Tree spatial index and its three
+    /// shadow tables (with the rowid map that gives a document id a rowid), sqlite-vec's <c>vec0</c> shadow
+    /// tables and its own id map, and the FTS5 virtual table with its five shadows. None of these are
+    /// written by this library directly - the engine materialises them behind a virtual table - but they
+    /// are ours, and a tool that did not know their names would report them as foreign tables.
+    /// </summary>
+    public IEnumerable<string> OwnedTableNames(string tableName, string? typeName)
+    {
+        foreach (var name in IDatabaseProvider.DefaultOwnedTableNames(this, tableName, typeName))
+            yield return name;
+
+        var spatial = ((IDatabaseProvider)this).SpatialTableName(tableName);
+        yield return spatial + "_map";
+        yield return spatial + "_node";
+        yield return spatial + "_rowid";
+        yield return spatial + "_parent";
+
+        // One FTS5 table per documents table (shared by every mapped type), plus its shadows.
+        var fts = FtsTableName(tableName);
+        yield return fts;
+        foreach (var shadow in new[] { "_data", "_idx", "_content", "_docsize", "_config" })
+            yield return fts + shadow;
+
+        if (string.IsNullOrEmpty(typeName))
+            yield break;
+
+        yield return VecMapTableName(tableName, typeName);
+
+        var vec = VecTableName(tableName, typeName);
+        foreach (var shadow in new[] { "_chunks", "_rowids", "_info", "_vector_chunks00" })
+            yield return vec + shadow;
+    }
+
     public string JsonExtract(string column, string jsonPath)
         => $"json_extract({column}, '$.{jsonPath}')";
 

src/Shiny.DocumentDb/IDatabaseProvider.cs

@@ -251,6 +251,57 @@ string BuildListTablesSql()
         => "SELECT table_name FROM information_schema.tables WHERE table_type = 'BASE TABLE' " +
            "AND table_schema NOT IN ('pg_catalog', 'information_schema');";
 
+    /// <summary>
+    /// Lists every column of every user table in the current schema as three columns in order:
+    /// <c>table_name</c>, <c>column_name</c>, <c>data_type</c>. The counterpart to
+    /// <see cref="BuildListTablesSql"/>, and the reason a tool can tell a DocumentDb table from a foreign
+    /// one <b>without</b> issuing a probe per table — one read answers for the whole database, and it
+    /// carries the column types rather than only the names.
+    /// </summary>
+    /// <remarks>
+    /// The default queries the ANSI <c>information_schema.columns</c> with the same system-schema
+    /// exclusion as <see cref="BuildListTablesSql"/>. SQLite and Oracle override with their catalog
+    /// views; MySQL overrides to scope to the current database.
+    /// </remarks>
+    string BuildListColumnsSql()
+        => "SELECT table_name, column_name, data_type FROM information_schema.columns " +
+           "WHERE table_schema NOT IN ('pg_catalog', 'information_schema');";
+
+    /// <summary>
+    /// Every table name this library would have created <b>around</b> <paramref name="tableName"/> — the
+    /// history, blob, spatial, vector and full-text sidecars, plus whatever engine-side shadow tables the
+    /// provider's own DDL drags in with them. Names only: the tables need not exist.
+    /// </summary>
+    /// <param name="typeName">
+    /// The stored type to compute the per-type sidecars for (vector, full text). Null or empty yields only
+    /// the names that do not depend on a type.
+    /// </param>
+    /// <remarks>
+    /// The counterpart to <see cref="HistoryTableName"/> / <see cref="BlobTableName"/> /
+    /// <see cref="SpatialTableName"/> / <see cref="VectorTableName"/> for the names that are <i>not</i> on
+    /// the contract — full-text tables and shadows are interpolated inside
+    /// <see cref="BuildCreateFullTextSql"/>, and SQLite's R*Tree and vec0 shadows are created by the engine
+    /// rather than by us. The provider already owns that DDL, so it is the honest place for the knowledge:
+    /// it lets a tool decide that a table is DocumentDb's by <b>computing the name and finding it</b>,
+    /// instead of guessing from a substring and mislabelling someone's <c>audit_history</c>.
+    /// </remarks>
+    IEnumerable<string> OwnedTableNames(string tableName, string? typeName)
+        => DefaultOwnedTableNames(this, tableName, typeName);
+
+    /// <summary>
+    /// The provider-independent half of <see cref="OwnedTableNames"/>, exposed so an override can add its
+    /// engine-specific shadows without restating the four naming conventions.
+    /// </summary>
+    static IEnumerable<string> DefaultOwnedTableNames(IDatabaseProvider provider, string tableName, string? typeName)
+    {
+        yield return provider.HistoryTableName(tableName);
+        yield return provider.BlobTableName(tableName);
+        yield return provider.SpatialTableName(tableName);
+
+        if (!string.IsNullOrEmpty(typeName))
+            yield return provider.VectorTableName(tableName, typeName);
+    }
+
     // RFC 7396 JSON Merge Patch support.
     // Providers that lack a native deep-merge function (PostgreSQL, SQL Server) return false;
     // DocumentStore then performs a read-merge-write fallback inside a row-locked transaction.
@@ -509,6 +560,15 @@ string BuildHistoryPruneByCountSql(string tableName)
 
     // Spatial (optional — only SQLite implements these)
     bool SupportsSpatial => false;
+
+    /// <summary>
+    /// The bare (unquoted) name of the per-table spatial sidecar. Every provider follows the same
+    /// <c>{table}_spatial</c> convention and differs only in quoting and in what it hangs off it (SQLite
+    /// adds a rowid map plus the R*Tree shadows), so the convention is defined once here — the
+    /// counterpart to <see cref="HistoryTableName"/>, <see cref="BlobTableName"/> and
+    /// <see cref="VectorTableName"/>.
+    /// </summary>
+    string SpatialTableName(string tableName) => tableName + "_spatial";
     string? BuildCreateSpatialTablesSql(string tableName) => null;
 
     /// <summary>