diff --git a/readme.md b/readme.md index 8d39881..4e962eb 100644 --- a/readme.md +++ b/readme.md @@ -2516,9 +2516,47 @@ place. | **Blobs** | Appears when the table has a `{table}_blobs` sidecar. Lists payloads without ever selecting the blob column, previews images and text, and downloads over a plain HTTP endpoint rather than the Blazor circuit. Only allow-listed raster images render inline; everything else is served as an attachment under `nosniff` + `default-src 'none'`. | | **Outbox** | Appears when the database holds [transactional outbox](#transactional-outbox) messages. A database-level screen rather than a type tab: how deep the queue is, whether it is draining (**oldest pending**, which is what separates "busy" from "the processor died"), what is dead-lettered grouped by message type and error, and requeue / purge behind a confirm. **It cannot dispatch** and says so on the page — delivery is your application's `IOutboxDispatcher`, over a transport the tool cannot reach; requeue only makes a message eligible again. | | **Import / Export** | Stream a type out as JSON, NDJSON, envelope JSON (round-trippable) or CSV; import JSON or NDJSON back with fail / replace / skip handling for duplicate ids. Also **generates** documents that look like the ones already there — numbers inside the observed range, dates inside the observed span, categorical fields drawn from the set actually used, ids continuing whatever scheme is in place — with a seeded preview that commits byte-identically. | -| **Assistant** | Appears when a connection has AI configured. A chat that answers questions about the data by composing the same reads the rest of the tool performs, scoped to the type you're looking at or the whole connection. Configured **per connection** — OpenAI, Azure OpenAI, Anthropic, or any OpenAI-compatible endpoint (OpenRouter, Groq, LM Studio, vLLM, Ollama) — so development can use a hosted model while production uses a local one, or none. **It cannot write:** the model is given ten tools and every one is a read, so read-only is a property of the tool surface rather than an instruction in a prompt. | +| **Assistant** | Appears when a connection has AI configured. A chat that answers questions about the data by composing the same reads the rest of the tool performs, scoped to the type you're looking at or the whole connection. Configured **per connection** — OpenAI, Azure OpenAI, Anthropic, or any OpenAI-compatible endpoint (OpenRouter, Groq, LM Studio, vLLM, Ollama) — so development can use a hosted model while production uses a local one, or none. **It cannot write:** the model is given eleven tools and every one is a read, so read-only is a property of the tool surface rather than an instruction in a prompt. | | **Query** | Two modes. **Filter grammar** takes DocumentDb's own string query syntax (`status == 'Shipped' and total:number > 100`) with `Where` / `OrderBy` / `Project` boxes — run through the library's own parser via `store.Collection(typeName)`, not a reimplementation, so the answer matches what your code would get. The compiled SQL is shown beside the results, **Explain** runs the provider's query plan over it (planned, not executed — safe on a read-only connection), unindexed fields in the query are offered as one-click index creates that then re-run and re-explain, and one button drops the SQL into the other mode. **Raw SQL** is whatever you type, in the target database's dialect, with `@name` parameters bound from a JSON box so the types stay honest. | +#### Is this database even a DocumentDb store? + +The overview answers that as a verdict rather than leaving it to be inferred from a filtered table +list — *"DocumentDb store · 3 document tables · 12 types · temporal, vectors, outbox"*, or *"Not a +DocumentDb database — 42 tables, none carry the envelope"* — with the reasons behind it, the tables it +created (each showing which documents table it belongs to) and the tables it did not, counted and +collapsed. + +It is evidence, not guesswork. Two catalog reads (`IDatabaseProvider.BuildListTablesSql` and the new +`BuildListColumnsSql`) describe the whole database in one go, so pointing the tool at a 300-table shared +schema no longer issues 300 deliberately-failing probes. A table is a *candidate* when it carries all +five envelope columns and **confirmed** only when something DocumentDb leaves behind corroborates it — +a JSON-shaped `Data` column, the `(Id, TypeName)` key, `idx_{table}_typename`, its JSON property +indexes, or (opt-in) a sampled row that really is a document. Envelope-and-nothing-else is reported as +**probable**, browsable and badged, rather than as a certainty. Sidecars are then found by *computing +the names DocumentDb would have created* — `IDatabaseProvider.OwnedTableNames(table, type)`, which each +provider extends with its own engine shadows (SQLite's R\*Tree and FTS5 tables, sqlite-vec's `vec0` +shadows, DuckDB's full-text source table) — and looking them up. Anything else is foreign, **whatever +it is called**: a business table named `audit_history`, `customer_blobs` or `geo_spatial_index` is no +longer reported as one of ours, and a documents table named `orders_history` is no longer hidden as a +sidecar. + +#### Soft delete has to be declared + +[Soft delete](#soft-delete) writes no column, table or index — it is an interceptor plus a named query +filter inside your application — so **the tool cannot discover it**, and until it is told, its delete +button is a real `DELETE` against a type your application would only ever have flagged. Declare the +types on the connection (in the connection's settings, or one click from a candidate the Structure tab +recognises; a host-provided connection declares them in configuration as +`Shiny:DocumentDb:{name}:SoftDelete:{Type}` = the flag's JSON path, optionally with `PropertyPath` / +`FlagKind` spelled out) and the tool acts on it: Browse gains a live / deleted / all filter and badges +flagged rows, delete writes the flag with **delete permanently** as a separate deliberate choice, +restore clears it to `false` or `null` per the declared kind, and the verdict lists `soft delete`. +Undeclared types behave exactly as before. The Structure tab only ever *suggests* a flag — a field +qualifies when its name is in the deleted family **and** its sampled values match one of the two shapes +`AddSoftDelete` accepts — and a suggestion changes no role, no browsability and no button until it is +confirmed. + Mark a connection **read-only** and every write path is blocked, including non-SELECT statements in the SQL console. The SQL console is otherwise exactly what it looks like — an open prompt against the database, with no statement allow-list — so give the tool a database account with the privileges you diff --git a/skills/shiny-documentdb/SKILL.md b/skills/shiny-documentdb/SKILL.md index e4f676f..2881ed5 100644 --- a/skills/shiny-documentdb/SKILL.md +++ b/skills/shiny-documentdb/SKILL.md @@ -126,6 +126,9 @@ triggers: - ToLower query - Math.Abs query - IDatabaseProvider + - BuildListColumnsSql + - OwnedTableNames + - SpatialTableName - json document - JsonNode - late-bound @@ -3401,6 +3404,19 @@ opts.ConfigureDocument(cfg => cfg.OnBeforeWrite(async (ctx, ct) => The nine document providers share one `IDocumentQuery` implementation. A provider derives from public **`DocumentQueryBase`** and implements four members — `Clone()`, `ExecuteAsync(QueryPlan)`, `DeleteMatchingAsync`, `SetPropertyMatchingAsync` — and inherits builder state/immutability, query-filter resolution, all client-side terminals, grouping, cursor paging, string projection, and the set-based-write interceptor plumbing. `ExecuteAsync` returns `QueryExecution.Candidates` (nothing applied server-side), `.Filtered`, `.Complete`, or `.Partial(...)`; the base applies only what the engine did not. **Report push-down honestly** — claiming ordering/paging you did not apply silently returns wrong results. Optional hooks (`SetPropertiesMatchingAsync` for multi-property `ExecuteUpdate`, `ObserveChanges`, `FullTextSearchCore`, `NearestVectorsCore`, `ToQueryString`, `ToCursorPage`) and the aggregate hooks (`CountCore`/`MaxCore`/…) have safe defaults; override only what the engine can do itself. On the options side implement **`IDocumentStoreOptions`** — `Mappings`, `TypeNameResolution`, `Capabilities`, the interceptor pair, and `SerializerOptions`/`EnsureSerializerOptions` — and `ConfigureDocument` plus every cross-cutting feature (soft delete, `cfg.MapJsonSchema`, field encryption) lights up for free. All per-type mapping state lives in the shared `DocumentMappingRegistry`; `Capabilities` is what the configuration validation pass reads to reject mappings your backend cannot honor. +**Two members exist so a tool can identify a database without guessing, and both have working defaults.** +`BuildListColumnsSql()` returns `(table_name, column_name, data_type)` for the current schema — the +counterpart to `BuildListTablesSql()`, so one read describes every table instead of one probe per table. +The default is ANSI `information_schema.columns`; override it where that view is absent or scoped wrong +(SQLite joins `sqlite_master` to `pragma_table_info`, Oracle reads `user_tab_columns`, MySQL scopes to +`DATABASE()`). `OwnedTableNames(table, typeName)` yields every table name **this provider would have +created around** `table` — the default covers `HistoryTableName` / `BlobTableName` / `SpatialTableName` / +`VectorTableName` (call `IDatabaseProvider.DefaultOwnedTableNames(this, table, typeName)` from an override +rather than restating them), and a provider adds whatever its own DDL drags in: full-text tables, and any +engine shadows behind a virtual table. Names only — the tables need not exist. This is what lets a tool +decide a table is yours by **computing the name and finding it** rather than matching a substring, so +leaving a full-text or shadow table out of it means someone's admin UI reports it as a foreign table. + Store writes bracket persistence with the shared pipeline on `DocumentProviderBase` (implement `Mappings`, `IdCache`, `ResolveTypeInfo`, `ResolveDocumentTypeName`): `BeginWriteAsync(op, doc, id, typeInfo, ct)` → check `write.Proceed` (false = an interceptor replaced the write; `Remove` returns `write.CancelResult`), take `write.Doc`, get the id with `ResolveInsertId`/`ResolveInsertIdAsync` (insert) or `RequireDocumentId` (update), persist, then `CompleteWriteAsync(write, id, version, changeType, doc, ct)` which runs `AfterWrite` and publishes the change (buffered until commit inside a unit of work). Never re-implement `PublishChange` — the base owns the broadcaster. ## Soft Delete (`AddSoftDelete`) diff --git a/src/Shiny.DocumentDb.DuckDb/DuckDbDatabaseProvider.cs b/src/Shiny.DocumentDb.DuckDb/DuckDbDatabaseProvider.cs index f7224d4..8b91ddc 100644 --- a/src/Shiny.DocumentDb.DuckDb/DuckDbDatabaseProvider.cs +++ b/src/Shiny.DocumentDb.DuckDb/DuckDbDatabaseProvider.cs @@ -607,6 +607,20 @@ public object FormatVectorParameter(ReadOnlyMemory vector, VectorMapping static string FtsSourceTable(string tableName, string typeName) => $"{tableName}_ftssrc_{IDatabaseProvider.SanitizeTypeSuffix(typeName)}"; + /// + /// Adds the per-type full-text source table. DuckDB's fts index is built by + /// PRAGMA create_fts_index over a snapshot table this provider materialises, so unlike every + /// other name here it is a real table in the main schema and would otherwise read as foreign. + /// + public IEnumerable OwnedTableNames(string tableName, string? typeName) + { + foreach (var name in IDatabaseProvider.DefaultOwnedTableNames(this, tableName, typeName)) + yield return name; + + if (!string.IsNullOrEmpty(typeName)) + yield return FtsSourceTable(tableName, typeName); + } + public string BuildFullTextProbeSql(string tableName, string typeName) => $"SELECT 1 FROM duckdb_tables() WHERE table_name = '{FtsSourceTable(tableName, typeName)}';"; diff --git a/src/Shiny.DocumentDb.MySql/MySqlDatabaseProvider.cs b/src/Shiny.DocumentDb.MySql/MySqlDatabaseProvider.cs index ac4b5af..792fc6c 100644 --- a/src/Shiny.DocumentDb.MySql/MySqlDatabaseProvider.cs +++ b/src/Shiny.DocumentDb.MySql/MySqlDatabaseProvider.cs @@ -287,6 +287,11 @@ public string BuildFullTextProbeSql(string tableName, string typeName) public string BuildListTablesSql() => "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_SCHEMA = DATABASE();"; + // Scoped to the current database for the same reason as BuildListTablesSql - MySQL's + // information_schema spans the whole server. + public string BuildListColumnsSql() + => "SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = DATABASE();"; + public string JsonExtract(string column, string jsonPath) => $"NULLIF(JSON_UNQUOTE(JSON_EXTRACT({column}, '$.{jsonPath}')), 'null')"; diff --git a/src/Shiny.DocumentDb.Oracle/OracleDatabaseProvider.cs b/src/Shiny.DocumentDb.Oracle/OracleDatabaseProvider.cs index 1e2d9dd..d64956f 100644 --- a/src/Shiny.DocumentDb.Oracle/OracleDatabaseProvider.cs +++ b/src/Shiny.DocumentDb.Oracle/OracleDatabaseProvider.cs @@ -414,6 +414,11 @@ public IReadOnlyList BuildExplainSql(string sql) public string BuildListTablesSql() => "SELECT table_name FROM user_tables"; + // user_tab_columns is the current schema's columns - Oracle's information_schema equivalent, and the + // only one a connection is guaranteed to be able to read without extra grants. + public string BuildListColumnsSql() + => "SELECT table_name, column_name, data_type FROM user_tab_columns"; + public string JsonExtract(string column, string jsonPath) => $"JSON_VALUE({column}, '$.{jsonPath}')"; diff --git a/src/Shiny.DocumentDb.Sqlite/SqliteDatabaseProvider.cs b/src/Shiny.DocumentDb.Sqlite/SqliteDatabaseProvider.cs index cd8d106..231594f 100644 --- a/src/Shiny.DocumentDb.Sqlite/SqliteDatabaseProvider.cs +++ b/src/Shiny.DocumentDb.Sqlite/SqliteDatabaseProvider.cs @@ -171,6 +171,46 @@ public string BuildListAllIndexesSql(string tableName) public string BuildListTablesSql() => "SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%';"; + // sqlite_master carries no column information, so the names come from pragma_table_info - joined as a + // table-valued function so the whole database is one read rather than one PRAGMA per table. + public string BuildListColumnsSql() + => "SELECT m.name, p.name, p.type FROM sqlite_master m JOIN pragma_table_info(m.name) p " + + "WHERE m.type IN ('table', 'view') AND m.name NOT LIKE 'sqlite_%';"; + + /// + /// Adds what SQLite creates alongside the portable sidecars: the R*Tree spatial index and its three + /// shadow tables (with the rowid map that gives a document id a rowid), sqlite-vec's vec0 shadow + /// tables and its own id map, and the FTS5 virtual table with its five shadows. None of these are + /// written by this library directly - the engine materialises them behind a virtual table - but they + /// are ours, and a tool that did not know their names would report them as foreign tables. + /// + public IEnumerable OwnedTableNames(string tableName, string? typeName) + { + foreach (var name in IDatabaseProvider.DefaultOwnedTableNames(this, tableName, typeName)) + yield return name; + + var spatial = ((IDatabaseProvider)this).SpatialTableName(tableName); + yield return spatial + "_map"; + yield return spatial + "_node"; + yield return spatial + "_rowid"; + yield return spatial + "_parent"; + + // One FTS5 table per documents table (shared by every mapped type), plus its shadows. + var fts = FtsTableName(tableName); + yield return fts; + foreach (var shadow in new[] { "_data", "_idx", "_content", "_docsize", "_config" }) + yield return fts + shadow; + + if (string.IsNullOrEmpty(typeName)) + yield break; + + yield return VecMapTableName(tableName, typeName); + + var vec = VecTableName(tableName, typeName); + foreach (var shadow in new[] { "_chunks", "_rowids", "_info", "_vector_chunks00" }) + yield return vec + shadow; + } + public string JsonExtract(string column, string jsonPath) => $"json_extract({column}, '$.{jsonPath}')"; diff --git a/src/Shiny.DocumentDb/IDatabaseProvider.cs b/src/Shiny.DocumentDb/IDatabaseProvider.cs index 29603d1..856e4bc 100644 --- a/src/Shiny.DocumentDb/IDatabaseProvider.cs +++ b/src/Shiny.DocumentDb/IDatabaseProvider.cs @@ -251,6 +251,57 @@ string BuildListTablesSql() => "SELECT table_name FROM information_schema.tables WHERE table_type = 'BASE TABLE' " + "AND table_schema NOT IN ('pg_catalog', 'information_schema');"; + /// + /// Lists every column of every user table in the current schema as three columns in order: + /// table_name, column_name, data_type. The counterpart to + /// , and the reason a tool can tell a DocumentDb table from a foreign + /// one without issuing a probe per table — one read answers for the whole database, and it + /// carries the column types rather than only the names. + /// + /// + /// The default queries the ANSI information_schema.columns with the same system-schema + /// exclusion as . SQLite and Oracle override with their catalog + /// views; MySQL overrides to scope to the current database. + /// + string BuildListColumnsSql() + => "SELECT table_name, column_name, data_type FROM information_schema.columns " + + "WHERE table_schema NOT IN ('pg_catalog', 'information_schema');"; + + /// + /// Every table name this library would have created around — the + /// history, blob, spatial, vector and full-text sidecars, plus whatever engine-side shadow tables the + /// provider's own DDL drags in with them. Names only: the tables need not exist. + /// + /// + /// The stored type to compute the per-type sidecars for (vector, full text). Null or empty yields only + /// the names that do not depend on a type. + /// + /// + /// The counterpart to / / + /// / for the names that are not on + /// the contract — full-text tables and shadows are interpolated inside + /// , and SQLite's R*Tree and vec0 shadows are created by the engine + /// rather than by us. The provider already owns that DDL, so it is the honest place for the knowledge: + /// it lets a tool decide that a table is DocumentDb's by computing the name and finding it, + /// instead of guessing from a substring and mislabelling someone's audit_history. + /// + IEnumerable OwnedTableNames(string tableName, string? typeName) + => DefaultOwnedTableNames(this, tableName, typeName); + + /// + /// The provider-independent half of , exposed so an override can add its + /// engine-specific shadows without restating the four naming conventions. + /// + static IEnumerable DefaultOwnedTableNames(IDatabaseProvider provider, string tableName, string? typeName) + { + yield return provider.HistoryTableName(tableName); + yield return provider.BlobTableName(tableName); + yield return provider.SpatialTableName(tableName); + + if (!string.IsNullOrEmpty(typeName)) + yield return provider.VectorTableName(tableName, typeName); + } + // RFC 7396 JSON Merge Patch support. // Providers that lack a native deep-merge function (PostgreSQL, SQL Server) return false; // DocumentStore then performs a read-merge-write fallback inside a row-locked transaction. @@ -509,6 +560,15 @@ string BuildHistoryPruneByCountSql(string tableName) // Spatial (optional — only SQLite implements these) bool SupportsSpatial => false; + + /// + /// The bare (unquoted) name of the per-table spatial sidecar. Every provider follows the same + /// {table}_spatial convention and differs only in quoting and in what it hangs off it (SQLite + /// adds a rowid map plus the R*Tree shadows), so the convention is defined once here — the + /// counterpart to , and + /// . + /// + string SpatialTableName(string tableName) => tableName + "_spatial"; string? BuildCreateSpatialTablesSql(string tableName) => null; /// diff --git a/src/ShinyDocDbMyAdmin.Core/Models/AdminModels.cs b/src/ShinyDocDbMyAdmin.Core/Models/AdminModels.cs index 8e0996f..14197f9 100644 --- a/src/ShinyDocDbMyAdmin.Core/Models/AdminModels.cs +++ b/src/ShinyDocDbMyAdmin.Core/Models/AdminModels.cs @@ -4,9 +4,32 @@ namespace ShinyDocDbMyAdmin.Models; /// A table in the target database, classified by what ShinyDocDbMyAdmin can do with it. -public sealed record TableInfo(string Name, TableRole Role, bool HasTenantColumn = false) +/// +/// The documents table this one belongs to, for a sidecar. Null for a documents table and for a foreign +/// one - a sidecar with no owner is not a sidecar, it is someone else's table that happens to be named +/// like ours. +/// +/// +/// How sure the classification is. Only ever for a documents table +/// that carries the envelope and nothing else; every other role is named from a table we found, so it is +/// by construction. +/// +/// +/// What this table is beyond its role - the sidecar's flavour ("R*Tree shadow", "FTS5 shadow"), or why a +/// documents table is only probable. Null when the role says everything there is to say. +/// +public sealed record TableInfo( + string Name, + TableRole Role, + bool HasTenantColumn = false, + string? Owner = null, + TableConfidence Confidence = TableConfidence.Confirmed, + string? Feature = null) { public bool IsBrowsable => this.Role == TableRole.Documents; + + /// True when this table is DocumentDb's - a documents table or one of its sidecars. + public bool IsOwned => this.Role != TableRole.Foreign; } public enum TableRole @@ -20,13 +43,90 @@ public enum TableRole /// A {table}_blobs binary sidecar. Blobs, - /// A spatial or vector sidecar. - Sidecar, + /// A {table}_spatial sidecar, or one of the engine shadows behind it. + Spatial, + + /// A {table}_vec_{type} sidecar, its id map, or one of the engine shadows behind it. + Vector, + + /// A full-text index table or one of its shadows. + FullText, /// Not a DocumentDb table at all. Foreign } +/// How sure a table classification is. See . +public enum TableConfidence +{ + /// The envelope plus at least one signal only DocumentDb leaves behind. + Confirmed, + + /// The envelope columns and nothing else - most likely ours, but nothing proves it. + Probable +} + +/// +/// The answer to "is this database participating in DocumentDb, and what of it is ours?" - computed from +/// the same single catalog read that classifies the tables, so asking costs nothing extra. +/// +/// True when at least one table carries the document envelope. +/// +/// when at least one document table is itself confirmed, +/// when every one of them is only probable, and +/// when there are none. +/// +/// +/// The DocumentDb features this database provably uses - temporal, blobs, spatial, vector, full text, +/// outbox, tenant, encryption, soft delete. Only what the evidence supports: soft delete appears here +/// exclusively when it has been declared on the connection, because nothing in the database records it. +/// +/// Why the verdict reads the way it does, in the operator's words. +public sealed record DatabaseIdentity( + bool Participates, + IdentityConfidence Confidence, + int DocumentTables, + int TypeCount, + int OwnedSidecars, + int ForeignTables, + IReadOnlyList Features, + IReadOnlyList Reasons) +{ + /// The verdict as one sentence - the banner, the TUI status line and the connection test all use it. + public string Summary + { + get + { + if (!this.Participates) + return $"Not a DocumentDb database - {this.ForeignTables} table(s), none carry the envelope."; + + var parts = new List + { + this.Confidence == IdentityConfidence.Probable ? "Probably a DocumentDb store" : "DocumentDb store", + $"{this.DocumentTables} document table(s)", + $"{this.TypeCount} type(s)" + }; + + if (this.Features.Count > 0) + parts.Add(string.Join(", ", this.Features)); + + return string.Join(" · ", parts); + } + } +} + +public enum IdentityConfidence +{ + /// Nothing in this database carries the document envelope. + None, + + /// Every document table is envelope-only - no index, no data, nothing that proves authorship. + Probable, + + /// At least one document table carries a signal only DocumentDb leaves behind. + Confirmed +} + /// One distinct TypeName value inside a documents table - the equivalent of a "table" in phpMyAdmin. public sealed record DocumentTypeInfo(string TypeName, long Count); @@ -81,19 +181,29 @@ public sealed record DocumentPage(IReadOnlyList Rows, long TotalCou /// /// Set when at least one sampled value on this path was a field-level encryption envelope; null otherwise. /// +/// +/// The kind of soft-delete flag this path looks like, or null. Never a verdict: soft delete writes +/// no DDL, so nothing in the database can prove it, and this is only a suggestion for the operator to +/// confirm into ConnectionProfile.SoftDeleteFlags. It changes no role, no browsability and no +/// delete behaviour on its own. See DocumentAdminService.SoftDeleteCandidateKind. +/// public sealed record InferredField( string Path, string Types, int Occurrences, int SampleSize, string? Example, - EncryptedFieldInfo? Encryption = null) + EncryptedFieldInfo? Encryption = null, + SoftDeleteFlagKind? SoftDeleteCandidate = null) { public int PercentPresent => this.SampleSize == 0 ? 0 : (int)Math.Round(this.Occurrences * 100.0 / this.SampleSize); public bool IsOptional => this.Occurrences < this.SampleSize; /// True when the sample proves this path holds encrypted values. public bool IsEncrypted => this.Encryption is not null; + + /// True when this path looks like a soft-delete flag - a suggestion, never a statement. + public bool IsSoftDeleteCandidate => this.SoftDeleteCandidate is not null; } /// diff --git a/src/ShinyDocDbMyAdmin.Core/Models/ConnectionProfile.cs b/src/ShinyDocDbMyAdmin.Core/Models/ConnectionProfile.cs index b24e4d8..77c568b 100644 --- a/src/ShinyDocDbMyAdmin.Core/Models/ConnectionProfile.cs +++ b/src/ShinyDocDbMyAdmin.Core/Models/ConnectionProfile.cs @@ -42,10 +42,57 @@ public class ConnectionProfile /// public List EncryptionKeys { get; set; } = []; + /// + /// Document types this connection's application has configured for soft delete, and where each keeps its + /// flag. Declared by the operator, because nothing in the database records it: + /// AddSoftDelete registers an interceptor and a query filter, and the flag is an ordinary JSON + /// property - there is no column, table or index for the tool to read. + /// + /// + /// Until a type is declared here, this tool's delete is a real DELETE - which for an application + /// that only ever flags is a broken invariant with no warning. Declaring a type turns the delete button + /// into a flag write, gives Browse its live / deleted / all filter, and puts soft delete in the + /// database verdict. The Structure tab suggests candidates; confirming one writes an entry here. + /// + public List SoftDeleteFlags { get; set; } = []; + + /// + /// Hides tables that are neither documents nor DocumentDb sidecars from the overview. On by default, so + /// "ignore the tables that aren't ours" is a stated behaviour rather than an accident of filtering - + /// which matters when this tool is pointed at a schema it shares with an application's own tables. + /// + public bool HideForeignTables { get; set; } = true; + public DateTimeOffset CreatedAt { get; set; } = DateTimeOffset.UtcNow; public DateTimeOffset? LastOpenedAt { get; set; } } +/// +/// One declared soft-delete flag: the type, the JSON path its flag lives at, and which of the two shapes +/// AddSoftDelete accepts it is - which is what tells a restore whether to write false or +/// null. +/// +public class SoftDeleteFlag +{ + /// The stored TypeName, exactly as it appears in the documents table. + public string TypeName { get; set; } = ""; + + /// Dotted JSON path to the flag inside the document body, e.g. isDeleted. + public string PropertyPath { get; set; } = ""; + + public SoftDeleteFlagKind FlagKind { get; set; } = SoftDeleteFlagKind.Boolean; +} + +/// The two flag shapes SoftDeleteMapping.Build accepts, and nothing else. +public enum SoftDeleteFlagKind +{ + /// A bool: deleted is true, restored is false. + Boolean, + + /// A nullable timestamp: deleted is the time of deletion, restored is null. + Timestamp +} + /// One entry in a connection's read-only key ring. public class EncryptionKeyEntry { diff --git a/src/ShinyDocDbMyAdmin.Core/Services/AdminConnection.cs b/src/ShinyDocDbMyAdmin.Core/Services/AdminConnection.cs index b827d5e..a4628b5 100644 --- a/src/ShinyDocDbMyAdmin.Core/Services/AdminConnection.cs +++ b/src/ShinyDocDbMyAdmin.Core/Services/AdminConnection.cs @@ -61,7 +61,14 @@ public async Task Open(string profileId, CancellationToken ct = ?? throw new InvalidOperationException($"Connection '{profileId}' no longer exists."); // Re-created whenever any input to the provider changes, so an edited profile takes effect at once. - var fingerprint = $"{resolved.Provider}|{resolved.ConnectionString}|{resolved.Password}"; + // The soft-delete declarations are in here for the same reason even though the provider never sees + // them: they decide whether a delete flags or deletes, and a cached connection carrying yesterday's + // list would decide it with the wrong answer. + var softDelete = string.Join( + ";", + resolved.Profile.SoftDeleteFlags.Select(f => $"{f.TypeName}:{f.PropertyPath}:{f.FlagKind}")); + + var fingerprint = $"{resolved.Provider}|{resolved.ConnectionString}|{resolved.Password}|{softDelete}"; await this.gate.WaitAsync(ct).ConfigureAwait(false); try diff --git a/src/ShinyDocDbMyAdmin.Core/Services/AiToolSurface.cs b/src/ShinyDocDbMyAdmin.Core/Services/AiToolSurface.cs index db048c6..915a872 100644 --- a/src/ShinyDocDbMyAdmin.Core/Services/AiToolSurface.cs +++ b/src/ShinyDocDbMyAdmin.Core/Services/AiToolSurface.cs @@ -55,6 +55,7 @@ public sealed class AiToolSurface(DocumentAdminService admin, ProfileStore profi public static readonly IReadOnlyList ToolNames = [ "list_connections", + "database_identity", "list_tables", "list_types", "describe_type", @@ -72,6 +73,7 @@ public IReadOnlyList Build() var tools = new List { AIFunctionFactory.Create(this.ListConnections, "list_connections"), + AIFunctionFactory.Create(this.DatabaseIdentity, "database_identity"), AIFunctionFactory.Create(this.ListTables, "list_tables"), AIFunctionFactory.Create(this.ListTypes, "list_types"), AIFunctionFactory.Create(this.DescribeType, "describe_type"), @@ -99,14 +101,26 @@ async Task> ListConnections(CancellationToken c return [.. all.Select(p => new ConnectionSummary(p.Id, p.Name, p.Provider.ToString(), p.ReadOnly))]; } - [Description("Lists the tables on a connection, marking which carry documents and which are " + - "history/blob/spatial/vector sidecars.")] + [Description("Reports whether a connection's database is a DocumentDb store at all, how sure that is, " + + "which DocumentDb features it uses, and why. Ask this before drawing conclusions from a " + + "table listing - a database can hold tables that merely look like ours.")] + Task DatabaseIdentity( + [Description("Connection id from list_connections.")] string connectionId, + CancellationToken ct) + => admin.GetIdentity(connectionId, ct: ct); + + [Description("Lists the tables on a connection. 'role' says what each one is - documents, or a " + + "history/blob/spatial/vector/full-text sidecar, or foreign (not DocumentDb's at all). " + + "'owner' is the documents table a sidecar belongs to. 'confidence' is Probable when a " + + "table carries the document envelope but nothing proves DocumentDb wrote it; only a table " + + "with carriesDocuments = true can be browsed.")] async Task> ListTables( [Description("Connection id from list_connections.")] string connectionId, CancellationToken ct) { var tables = await admin.ListTables(connectionId, refresh: false, ct); - return [.. tables.Select(t => new TableSummary(t.Name, t.Role.ToString(), t.IsBrowsable))]; + return [.. tables.Select(t => new TableSummary( + t.Name, t.Role.ToString(), t.IsBrowsable, t.Owner, t.Confidence.ToString(), t.Feature))]; } [Description("Lists the document types stored in a table, with how many documents each has. " + @@ -146,7 +160,9 @@ Task TableStats( [Description("Reads a page of documents, optionally filtered and sorted. Use describe_type first " + "so the field paths you filter on actually exist. Results are capped - check the " + - "'truncated' and 'totalCount' fields before stating a total.")] + "'truncated' and 'totalCount' fields before stating a total. When the type is declared " + + "for soft delete this returns live documents only, and says so in 'scope' - do not " + + "describe such a listing as everything the table holds.")] async Task BrowseDocuments( [Description("Connection id from list_connections.")] string connectionId, [Description("Table name from list_tables.")] string table, @@ -163,6 +179,8 @@ async Task BrowseDocuments( [Description("Optional field path to sort by. Defaults to UpdatedAt.")] string? sortPath = null, [Description("Sort descending. Defaults to true.")] bool? sortDescending = null, [Description("Rows to return, 1 to 50. Defaults to 25.")] int? limit = null, + [Description("Include documents the application has soft-deleted. Only has an effect for a type " + + "with a declared soft-delete flag, where the default excludes them.")] bool? includeDeleted = null, CancellationToken ct = default) { var take = Math.Clamp(limit ?? 25, 1, MaxRows); @@ -196,13 +214,25 @@ async Task BrowseDocuments( Sort = sort, Filters = filters, Search = search, - SearchPaths = searchPaths + SearchPaths = searchPaths, + Deleted = includeDeleted == true ? DeletedFilter.All : DeletedFilter.Live }, ct); + // Said out loud rather than left implicit: a filtered listing the model reports as "the documents" + // is how a soft-deleted document gets described as gone, or a live one as missing. + var flag = await admin.GetSoftDeleteFlag(connectionId, typeName, ct); + var scope = flag is null + ? null + : includeDeleted == true + ? $"All documents, including ones flagged deleted on '{flag.PropertyPath}'." + : $"Live documents only - '{typeName}' is declared for soft delete on '{flag.PropertyPath}'. " + + "Pass includeDeleted to see the flagged ones."; + return new DocumentResults( [.. page.Rows.Select(Summarize)], page.TotalCount, - page.TotalCount > page.Rows.Count); + page.TotalCount > page.Rows.Count, + scope); } [Description("Reads one document by its id.")] @@ -295,12 +325,22 @@ static string Clip(string json) // model provider, so each field is one someone chose to send. public sealed record ConnectionSummary(string ConnectionId, string Name, string Provider, bool ReadOnly); - public sealed record TableSummary(string Name, string Role, bool CarriesDocuments); + public sealed record TableSummary( + string Name, + string Role, + bool CarriesDocuments, + string? Owner, + string Confidence, + string? Detail); public sealed record TypeSummary(string TypeName, long DocumentCount); public sealed record SchemaFieldSummary(string Path, string JsonTypes, int PercentPresent, string? Example); public sealed record TypeSchema(string TypeName, int SampleSize, IReadOnlyList Fields); public sealed record DocumentSummary(string Id, DateTimeOffset? CreatedAt, DateTimeOffset? UpdatedAt, string Json); - public sealed record DocumentResults(IReadOnlyList Documents, long TotalCount, bool Truncated); + public sealed record DocumentResults( + IReadOnlyList Documents, + long TotalCount, + bool Truncated, + string? Scope = null); public sealed record IndexSummary(string Name, string TypeName, IReadOnlyList Paths); public sealed record SearchHit(string DocumentId, double Score, string Json); diff --git a/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Browse.cs b/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Browse.cs index 0fdddcd..fbf7242 100644 --- a/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Browse.cs +++ b/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Browse.cs @@ -32,6 +32,14 @@ public async Task Browse( where.Add("TypeName = @typeName", ("@typeName", typeName)); ApplyFilters(where, provider, query); + // A declared soft-delete type is partitioned by its flag. Undeclared types have no flag to partition + // on, so the tri-state is inert for them and the grid keeps showing everything, as it always has. + var softDelete = connection.Profile.Profile.SoftDeleteFlags + .FirstOrDefault(f => f.TypeName.Equals(typeName, StringComparison.Ordinal)); + + if (softDelete is not null && query.Deleted != DeletedFilter.All) + where.Add(SoftDeletePredicate(provider, softDelete, query.Deleted == DeletedFilter.Deleted)); + var page = Math.Max(1, query.Page); var pageSize = Math.Clamp(query.PageSize, 1, 500); var offset = (page - 1) * pageSize; @@ -248,4 +256,12 @@ public sealed record BrowseQuery public string? TenantId { get; init; } public bool IncludeTenant { get; init; } + + /// + /// Which partition of a declared soft-delete type to show. Defaults to , + /// which is what the application's own reads see - a grid that mixed flagged documents in with live ones + /// and said nothing would be reporting deleted documents as current. Inert for a type with no declared + /// flag: there is nothing to partition on. + /// + public DeletedFilter Deleted { get; init; } = DeletedFilter.Live; } diff --git a/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Crud.cs b/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Crud.cs index 89f85ce..12f5f1b 100644 --- a/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Crud.cs +++ b/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Crud.cs @@ -156,11 +156,25 @@ static VectorSyncOutcome Outcome(VectorSidecarInfo sidecar, float[]? vector) => : vector is null ? VectorSyncOutcome.Ambiguous : VectorSyncOutcome.Synced; + /// + /// Removes documents outright, with their blob, vector and spatial sidecar rows, and records a Removed + /// tombstone where the type is temporal. + /// + /// + /// Required to be true for a type with a declared soft-delete flag. The application only ever flags such + /// a document, so a plain DELETE here breaks an invariant it relies on - and it used to do so + /// silently. Callers choose or say permanently, and nothing decides it + /// for them. + /// + /// + /// The type is declared for soft delete and is false. + /// public async Task DeleteDocuments( string profileId, string table, string typeName, IReadOnlyList ids, + bool permanent = false, CancellationToken ct = default) { if (ids.Count == 0) @@ -169,6 +183,14 @@ public async Task DeleteDocuments( var connection = await this.Connect(profileId, ct); connection.AssertWritable(); + if (!permanent && await this.GetSoftDeleteFlag(profileId, typeName, ct) is { } declared) + { + throw new InvalidOperationException( + $"'{typeName}' is declared for soft delete on flag '{declared.PropertyPath}', so deleting it " + + "would remove a document the application would only have flagged. Flag it with " + + "SoftDeleteDocuments, or delete it permanently on purpose."); + } + var provider = connection.Provider; var safeTable = Ado.SafeIdentifier(table); @@ -186,7 +208,7 @@ public async Task DeleteDocuments( // exists. var spatialDelete = provider.BuildSpatialDeleteSql(safeTable); var hasSpatial = spatialDelete is not null - && tables.Any(t => t.Name.Equals(SpatialTableName(safeTable), StringComparison.OrdinalIgnoreCase)); + && tables.Any(t => t.Name.Equals(provider.SpatialTableName(safeTable), StringComparison.OrdinalIgnoreCase)); // Full text is deliberately absent: every provider that supports it has the engine maintain the // index (FTS5 triggers, or a rebuild at query time where FullTextIndexRequiresRebuild), so there is @@ -269,11 +291,20 @@ await RecordVersion( /// Deliberately does not write history, matching the library: Clear<T> is a bulk delete /// and is the one mutation temporal tracking skips. /// - public async Task ClearType(string profileId, string table, string typeName, CancellationToken ct = default) + public async Task ClearType(string profileId, string table, string typeName, bool permanent = false, CancellationToken ct = default) { var connection = await this.Connect(profileId, ct); connection.AssertWritable(); + // Same guard, same reason: the library's Clear is intercepted into a bulk flag write for a + // soft-delete type, so clearing one here without saying "permanently" would not match it. + if (!permanent && await this.GetSoftDeleteFlag(profileId, typeName, ct) is { } declared) + { + throw new InvalidOperationException( + $"'{typeName}' is declared for soft delete on flag '{declared.PropertyPath}'. Clearing it here " + + "would permanently remove documents the application would only have flagged - pass permanent to do that on purpose."); + } + var provider = connection.Provider; var safeTable = Ado.SafeIdentifier(table); var quoted = provider.QuoteTable(safeTable); @@ -285,7 +316,7 @@ public async Task ClearType(string profileId, string table, string typeName var hasBlobs = tables.Any(t => t.Name.Equals(provider.BlobTableName(safeTable), StringComparison.OrdinalIgnoreCase)); var spatialClear = provider.BuildSpatialClearSql(safeTable); var hasSpatial = spatialClear is not null - && tables.Any(t => t.Name.Equals(SpatialTableName(safeTable), StringComparison.OrdinalIgnoreCase)); + && tables.Any(t => t.Name.Equals(provider.SpatialTableName(safeTable), StringComparison.OrdinalIgnoreCase)); var deleted = await connection.Execute(async (db, token) => { diff --git a/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Geometry.cs b/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Geometry.cs index 9829f16..41b83c7 100644 --- a/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Geometry.cs +++ b/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Geometry.cs @@ -37,13 +37,6 @@ public sealed partial class DocumentAdminService /// How many documents a geometry view will read before it stops and says so. public const int GeometryScanLimit = 500; - /// - /// The spatial sidecar's name. Unlike history, blobs and vectors, this one is not on - /// IDatabaseProvider - every provider interpolates it inside its own spatial SQL - so the - /// convention is written down once here rather than at each use. - /// - internal static string SpatialTableName(string safeTable) => safeTable + "_spatial"; - /// /// True when the table has a spatial sidecar. Distinct from having geometry: a type can carry GeoJSON /// in its body and never have been mapped with MapSpatial, in which case there is no index row @@ -55,7 +48,7 @@ public async Task HasSpatialSidecar(string profileId, string table, Cancel if (connection.Provider.BuildSpatialDeleteSql(Ado.SafeIdentifier(table)) is null) return false; - var sidecar = SpatialTableName(Ado.SafeIdentifier(table)); + var sidecar = connection.Provider.SpatialTableName(Ado.SafeIdentifier(table)); var tables = await this.ListTables(profileId, refresh: false, ct); return tables.Any(t => t.Name.Equals(sidecar, StringComparison.OrdinalIgnoreCase)); diff --git a/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Identity.cs b/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Identity.cs new file mode 100644 index 0000000..1c92bb0 --- /dev/null +++ b/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Identity.cs @@ -0,0 +1,530 @@ +using System.Data.Common; +using System.Text.Json.Nodes; +using System.Text.RegularExpressions; +using Shiny.DocumentDb; +using ShinyDocDbMyAdmin.Models; + +namespace ShinyDocDbMyAdmin.Services; + +/// +/// Deciding what a database is: which of its tables are DocumentDb's, which belong to something +/// else, and whether the database participates at all. +/// +/// +/// +/// This replaced a classifier that ran name-substring rules first and probed the database second. Both +/// halves were wrong. The substring rules are unanchored, so a business table called audit_history +/// or geo_spatial_index was reported as a DocumentDb sidecar, and a documents table called +/// orders_history never reached the probe at all - it was classified as history and vanished from +/// the explorer, the filter console and the assistant. The probe was a failing +/// SELECT … WHERE 1 = 0 per table, which costs a DbException per foreign table (~300 of them +/// in a shared schema) and cannot run inside a transaction on PostgreSQL. +/// +/// +/// What replaces it is evidence. Two catalog reads describe the whole database - the table list and every +/// column of every table, with types. A table is a candidate when it carries all five envelope +/// columns, and it is only when something DocumentDb leaves behind +/// corroborates it. Sidecars are then found by computing the names DocumentDb would have created +/// () and looking them up: name matching becomes +/// confirmation of a name we derived rather than a guess at one we found. Everything left over is foreign, +/// whatever it is called. +/// +/// +public sealed partial class DocumentAdminService +{ + /// Rows read per document table when a caller asks for the opt-in data signal. + const int IdentitySampleRows = 5; + + /// Column types that can hold a JSON document across the nine dialects. + static readonly string[] JsonColumnTypes = + ["json", "jsonb", "text", "clob", "nclob", "varchar", "nvarchar", "char", "string", "longtext", "mediumtext"]; + + /// + /// A primary key or unique index over (Id, TypeName) as the catalogs spell it. Word-boundary + /// matched, because an index definition mentioning idx_json_… contains "id" as a + /// substring and would otherwise pass. + /// + static readonly Regex EnvelopeKeyDefinition = + new(@"\bunique\b(?=.*\bid\b)(?=.*\btypename\b)", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + + /// + /// The verdict for a whole database. Computed from the same catalog read that classifies the tables, + /// so a caller that has already listed them pays nothing for asking. + /// + /// + /// Reads a handful of rows from each candidate table, so a table with the envelope but no index can + /// still be confirmed by what is in it, and so field-level encryption shows up in the feature + /// list. Off by default: it is the only part of this that reads data rather than catalog. + /// + public async Task GetIdentity( + string profileId, + bool sampleRows = false, + bool refresh = false, + CancellationToken ct = default) + => (await this.Catalog(profileId, refresh, sampleRows, ct)).Identity; + + /// The classified tables, cached per profile until . + public async Task> ListTables(string profileId, bool refresh = false, CancellationToken ct = default) + => (await this.Catalog(profileId, refresh, sampleRows: false, ct)).Tables; + + async Task Catalog(string profileId, bool refresh, bool sampleRows, CancellationToken ct) + { + // A sampled snapshot is strictly better informed than an unsampled one, so it satisfies a caller + // that did not ask for sampling - but not the other way round. + if (!refresh && this.catalogCache.TryGetValue(profileId, out var cached) && (cached.Sampled || !sampleRows)) + return cached; + + var connection = await this.Connect(profileId, ct); + var snapshot = await connection.Execute( + (db, token) => Classify(db, connection, sampleRows, token), ct); + + this.catalogCache[profileId] = snapshot; + return snapshot; + } + + /// What one catalog read produced: every table classified, and the database's own verdict. + sealed record CatalogSnapshot(IReadOnlyList Tables, DatabaseIdentity Identity, bool Sampled); + + static async Task Classify(DbConnection db, AdminConnection connection, bool sampleRows, CancellationToken ct) + { + var provider = connection.Provider; + + var names = await ReadTableNames(db, provider, ct); + var catalog = await ReadColumns(db, provider, ct); + + // ── Which tables carry the envelope ───────────────────────────────── + var candidates = names.Where(n => catalog.TryGetValue(n, out var c) && HasEnvelope(c)).ToList(); + + // Carrying the envelope is not enough on its own, and the blob sidecar is the proof: it is + // Id / TypeName / Data / CreatedAt / UpdatedAt plus BlobKey, so by columns alone it reads as a + // documents table. A candidate that is a name another candidate would have created is that other + // table's sidecar, not a documents table of its own. Only the type-independent names are needed + // here - no per-type sidecar carries the envelope - which is what keeps this from needing the type + // lists it would otherwise have to read first. + var sidecarNames = new HashSet(StringComparer.OrdinalIgnoreCase); + foreach (var candidate in candidates) + { + foreach (var sidecar in provider.OwnedTableNames(candidate, null)) + { + if (!sidecar.Equals(candidate, StringComparison.OrdinalIgnoreCase)) + sidecarNames.Add(sidecar); + } + } + + var documents = new List(); + foreach (var name in candidates.Where(c => !sidecarNames.Contains(c))) + { + var columns = catalog[name]; + var evidence = await Evidence(db, provider, name, columns, sampleRows, ct); + documents.Add(new DocumentTable( + name, + columns.ContainsKey("TenantId"), + columns, + evidence, + await ReadTypeNames(db, provider, name, ct))); + } + + // ── The sidecars, by computed name rather than by substring ───────── + var owned = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var table in documents) + { + foreach (var candidate in OwnedNames(provider, table)) + { + // First owner wins. Two document tables can only collide here by being named such that one + // is the other's sidecar prefix, and the shorter name is the one the convention derives from. + if (!owned.ContainsKey(candidate) && !documents.Any(d => d.Name.Equals(candidate, StringComparison.OrdinalIgnoreCase))) + owned[candidate] = new OwnedTable(table.Name, OwnedRole(provider, table.Name, candidate)); + } + } + + var tables = new List(names.Count); + foreach (var name in names) + { + var document = documents.FirstOrDefault(d => d.Name.Equals(name, StringComparison.Ordinal)); + if (document is not null) + { + tables.Add(new TableInfo( + name, + TableRole.Documents, + document.HasTenant, + Owner: null, + document.Evidence.Confidence, + document.Evidence.Confidence == TableConfidence.Probable + ? "envelope only - nothing here proves DocumentDb wrote it" + : null)); + continue; + } + + tables.Add(owned.TryGetValue(name, out var sidecar) + ? new TableInfo(name, sidecar.Role, false, sidecar.Owner, TableConfidence.Confirmed, Detail(sidecar.Owner, name, sidecar.Role)) + : new TableInfo(name, TableRole.Foreign)); + } + + return new CatalogSnapshot(tables, Verdict(connection, tables, documents), sampleRows); + } + + // ── Reads ─────────────────────────────────────────────────────────── + + static async Task> ReadTableNames(DbConnection db, IDatabaseProvider provider, CancellationToken ct) + { + var names = new List(); + await using var cmd = Ado.Command(db, provider.BuildListTablesSql()); + await using var reader = await cmd.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + names.Add(Ado.Text(reader, 0)); + + return [.. names.OrderBy(x => x, StringComparer.OrdinalIgnoreCase)]; + } + + /// + /// Every column of every table, as one read. The table list is read separately rather than derived from + /// this, because a provider's column view can legitimately not cover everything the table list does - + /// SQLite's excludes virtual tables, whose columns cannot be read without the module that created them + /// loaded. Such a table simply has no columns here, which is exactly right: it has no envelope, and it + /// is still recognised by name if it is one we created. + /// + static async Task>> ReadColumns( + DbConnection db, IDatabaseProvider provider, CancellationToken ct) + { + var catalog = new Dictionary>(StringComparer.OrdinalIgnoreCase); + + await using var cmd = Ado.Command(db, provider.BuildListColumnsSql()); + await using var reader = await cmd.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + var table = Ado.Text(reader, 0); + if (!catalog.TryGetValue(table, out var columns)) + catalog[table] = columns = new Dictionary(StringComparer.OrdinalIgnoreCase); + + columns[Ado.Text(reader, 1)] = Ado.NullableText(reader, 2) ?? ""; + } + + return catalog; + } + + /// The distinct stored types in a documents table - what the per-type sidecar names are derived from. + static async Task> ReadTypeNames(DbConnection db, IDatabaseProvider provider, string table, CancellationToken ct) + { + var quoted = provider.QuoteTable(Ado.SafeIdentifier(table)); + var types = new List(); + + await using var cmd = Ado.Command(db, $"SELECT DISTINCT TypeName FROM {quoted}"); + await using var reader = await cmd.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + var type = Ado.NullableText(reader, 0); + if (!string.IsNullOrEmpty(type)) + types.Add(type); + } + + return types; + } + + // ── Evidence ──────────────────────────────────────────────────────── + + static bool HasEnvelope(Dictionary columns) + => columns.ContainsKey("Id") + && columns.ContainsKey("TypeName") + && columns.ContainsKey("Data") + && columns.ContainsKey("CreatedAt") + && columns.ContainsKey("UpdatedAt"); + + /// + /// Scores a candidate. The envelope is required but never sufficient - any table with those five column + /// names has it, whatever the types and whatever is in it. What separates a DocumentDb table + /// from a coincidence is the rest: a JSON-shaped Data column, the key the library declares, the + /// indexes it creates, and - when the caller asks for it - a row that actually holds a document. + /// + static async Task Evidence( + DbConnection db, + IDatabaseProvider provider, + string table, + Dictionary columns, + bool sampleRows, + CancellationToken ct) + { + var signals = new List(); + + var dataType = columns["Data"]; + if (JsonColumnTypes.Any(t => dataType.Contains(t, StringComparison.OrdinalIgnoreCase))) + signals.Add($"Data is {dataType.ToLowerInvariant()}"); + + foreach (var signal in await IndexSignals(db, provider, table, ct)) + signals.Add(signal); + + if (!sampleRows) + return new TableEvidence(signals); + + var sample = await Sample(db, provider, table, ct); + if (sample.HoldsDocuments) + signals.Add("holds documents"); + + return new TableEvidence(signals) { Encrypted = sample.Encrypted }; + } + + static async Task> IndexSignals(DbConnection db, IDatabaseProvider provider, string table, CancellationToken ct) + { + var sql = provider.BuildListAllIndexesSql(Ado.SafeIdentifier(table)); + if (sql is null) + return []; + + var signals = new List(); + var typeNameIndex = $"idx_{table}_typename"; + var jsonIndexes = 0; + + try + { + await using var cmd = Ado.Command(db, sql); + await using var reader = await cmd.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + var name = Ado.Text(reader, 0); + var definition = Ado.NullableText(reader, 1); + + if (name.Equals(typeNameIndex, StringComparison.OrdinalIgnoreCase)) + signals.Add($"{typeNameIndex} present"); + else if (name.StartsWith(IndexPrefix, StringComparison.OrdinalIgnoreCase)) + jsonIndexes++; + else if (IsEnvelopeKey(table, name, definition)) + signals.Add("keyed on (Id, TypeName)"); + } + } + catch (DbException) + { + // An index view a connection is not granted, or a catalog that disagrees with its own docs. The + // classification does not depend on this - it only corroborates - so an unreadable index list + // costs a signal, not an answer. + return signals; + } + + if (jsonIndexes > 0) + signals.Add($"{jsonIndexes} JSON property index(es)"); + + return signals; + } + + /// + /// Whether an index is the (Id, TypeName) key the library declares. Every engine records it + /// differently - some name it after the table, some only list its columns - and SQLite and DuckDB do not + /// surface it at all, which is why its absence proves nothing. + /// + static bool IsEnvelopeKey(string table, string name, string? definition) + => name.Equals("PRIMARY", StringComparison.OrdinalIgnoreCase) + || name.Equals($"PK_{table}", StringComparison.OrdinalIgnoreCase) + || name.Equals($"{table}_pkey", StringComparison.OrdinalIgnoreCase) + || (definition is { Length: > 0 } && EnvelopeKeyDefinition.IsMatch(definition)); + + /// + /// Reads a handful of rows: whether any of them looks like a stored document (parseable JSON under a + /// non-empty type), and whether any field in them is a field-level encryption envelope. The one read + /// here that touches data rather than catalog, which is why the caller has to ask for it. + /// + static async Task Sample(DbConnection db, IDatabaseProvider provider, string table, CancellationToken ct) + { + var quoted = provider.QuoteTable(Ado.SafeIdentifier(table)); + var holdsDocuments = false; + var encrypted = false; + + try + { + await using var cmd = Ado.Command( + db, $"SELECT TypeName, Data FROM {quoted} {provider.BuildPaginationClause(0, IdentitySampleRows)}"); + + await using var reader = await cmd.ExecuteReaderAsync(ct); + while (await reader.ReadAsync(ct)) + { + if (string.IsNullOrEmpty(Ado.NullableText(reader, 0))) + continue; + + JsonNode? body; + try + { + body = JsonNode.Parse(Ado.Text(reader, 1)); + } + catch (System.Text.Json.JsonException) + { + // Not a document. Keep looking: one unparseable body does not settle the table. + continue; + } + + if (body is not JsonObject) + continue; + + holdsDocuments = true; + encrypted |= EncryptedFields.PathsIn(body).Count > 0; + } + } + catch (DbException) + { + // The one read here that can fail on a table we do not own - a column of an incompatible type + // behind a matching name. Absence of the signal is the answer. + } + + return new SampleOutcome(holdsDocuments, encrypted); + } + + sealed record SampleOutcome(bool HoldsDocuments, bool Encrypted); + + // ── Owned names ───────────────────────────────────────────────────── + + static IEnumerable OwnedNames(IDatabaseProvider provider, DocumentTable table) + { + // Type-independent sidecars (history, blobs, spatial and whatever hangs off them) exist for a table + // with no rows in it too, so they are asked for once with no type. + foreach (var name in provider.OwnedTableNames(table.Name, null)) + yield return name; + + foreach (var type in table.Types) + { + foreach (var name in provider.OwnedTableNames(table.Name, type)) + yield return name; + } + } + + /// + /// Which feature a computed name belongs to. Name matching is safe here in a way it is not in a + /// classifier: these are names this provider just generated, so the match confirms a name we derived + /// rather than guessing at one we found. + /// + static TableRole OwnedRole(IDatabaseProvider provider, string table, string name) + { + if (name.Equals(provider.HistoryTableName(table), StringComparison.OrdinalIgnoreCase)) + return TableRole.History; + + if (name.Equals(provider.BlobTableName(table), StringComparison.OrdinalIgnoreCase)) + return TableRole.Blobs; + + var suffix = name.Length > table.Length ? name[table.Length..] : ""; + if (suffix.StartsWith("_spatial", StringComparison.OrdinalIgnoreCase)) + return TableRole.Spatial; + if (suffix.StartsWith("_vec", StringComparison.OrdinalIgnoreCase)) + return TableRole.Vector; + + return TableRole.FullText; + } + + /// The half of a sidecar's identity the role cannot carry: which of several tables it is. + static string? Detail(string owner, string name, TableRole role) + { + var suffix = name.Length > owner.Length ? name[owner.Length..] : ""; + + return role switch + { + TableRole.Spatial when suffix.StartsWith("_spatial_map", StringComparison.OrdinalIgnoreCase) => "rowid map", + TableRole.Spatial when !suffix.Equals("_spatial", StringComparison.OrdinalIgnoreCase) => "R*Tree shadow", + TableRole.Vector when suffix.StartsWith("_vec_map", StringComparison.OrdinalIgnoreCase) => "document id map", + TableRole.Vector when suffix.Contains("_chunks", StringComparison.OrdinalIgnoreCase) + || suffix.EndsWith("_rowids", StringComparison.OrdinalIgnoreCase) + || suffix.EndsWith("_info", StringComparison.OrdinalIgnoreCase) => "vec0 shadow", + TableRole.FullText when suffix.StartsWith("_ftssrc", StringComparison.OrdinalIgnoreCase) => "indexed text", + TableRole.FullText when !suffix.Equals("_fts", StringComparison.OrdinalIgnoreCase) => "FTS5 shadow", + _ => null + }; + } + + // ── The verdict ───────────────────────────────────────────────────── + + static DatabaseIdentity Verdict(AdminConnection connection, IReadOnlyList tables, IReadOnlyList documents) + { + var foreign = tables.Count(t => t.Role == TableRole.Foreign); + var sidecars = tables.Count(t => t.IsOwned && t.Role != TableRole.Documents); + var types = documents.SelectMany(d => d.Types).Distinct(StringComparer.Ordinal).ToList(); + + if (documents.Count == 0) + { + return new DatabaseIdentity( + false, IdentityConfidence.None, 0, 0, 0, foreign, [], + [$"{tables.Count} table(s) read; none carry the Id / TypeName / Data / CreatedAt / UpdatedAt envelope."]); + } + + var confirmed = documents.Count(d => d.Evidence.Confidence == TableConfidence.Confirmed); + var reasons = new List + { + $"{documents.Count} table(s) carry the document envelope." + }; + + foreach (var table in documents.Where(d => d.Evidence.Signals.Count > 0)) + reasons.Add($"{table.Name}: {string.Join(", ", table.Evidence.Signals)}."); + + if (confirmed < documents.Count) + reasons.Add($"{documents.Count - confirmed} of them carry the envelope and nothing else."); + + if (sidecars > 0) + reasons.Add($"{sidecars} table(s) match names DocumentDb would have created."); + + if (foreign > 0) + reasons.Add($"{foreign} table(s) belong to something else."); + + return new DatabaseIdentity( + true, + confirmed > 0 ? IdentityConfidence.Confirmed : IdentityConfidence.Probable, + documents.Count, + types.Count, + sidecars, + foreign, + Features(connection, tables, documents, types), + reasons); + } + + /// + /// What this database provably uses. Everything here is read from the catalog or from the type list - + /// except soft delete, which no database records, and which therefore appears only when the operator + /// has declared it on the connection. + /// + static IReadOnlyList Features( + AdminConnection connection, + IReadOnlyList tables, + IReadOnlyList documents, + IReadOnlyList types) + { + var features = new List(); + + if (tables.Any(t => t.Role == TableRole.History)) features.Add("temporal"); + if (tables.Any(t => t.Role == TableRole.Blobs)) features.Add("blobs"); + if (tables.Any(t => t.Role == TableRole.Spatial)) features.Add("spatial"); + if (tables.Any(t => t.Role == TableRole.Vector)) features.Add("vectors"); + + // Only two providers give full text a table of its own; the rest add a generated column to the + // documents table, which the same catalog read already has. + if (tables.Any(t => t.Role == TableRole.FullText) || documents.Any(HasFullTextColumn)) + features.Add("full text"); + + if (types.Any(IsOutboxTypeName)) features.Add("outbox"); + if (documents.Any(d => d.HasTenant)) features.Add("tenant"); + if (documents.Any(d => d.Evidence.Encrypted)) features.Add("encryption"); + + var declared = connection.Profile.Profile.SoftDeleteFlags; + if (declared.Any(f => types.Contains(f.TypeName, StringComparer.Ordinal))) + features.Add("soft delete"); + + return features; + } + + /// The generated column full text adds on every provider that does not build a table for it. + static bool HasFullTextColumn(DocumentTable table) + => table.Columns.Keys.Any(c => + c.StartsWith("fts_", StringComparison.OrdinalIgnoreCase) || + c.StartsWith("ftcc_", StringComparison.OrdinalIgnoreCase)); + + // ── Working state ─────────────────────────────────────────────────── + + sealed record DocumentTable( + string Name, + bool HasTenant, + Dictionary Columns, + TableEvidence Evidence, + IReadOnlyList Types); + + sealed record OwnedTable(string Owner, TableRole Role); + + /// + /// The signals found for one candidate. The envelope is the entry fee, so it is not in here; these are + /// the things that make the difference between "carries those five columns" and "DocumentDb wrote this". + /// + sealed record TableEvidence(IReadOnlyList Signals) + { + public TableConfidence Confidence => this.Signals.Count > 0 ? TableConfidence.Confirmed : TableConfidence.Probable; + + public bool Encrypted { get; init; } + } +} diff --git a/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Schema.cs b/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Schema.cs index 81d1a0d..2c03ffe 100644 --- a/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Schema.cs +++ b/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.Schema.cs @@ -49,7 +49,8 @@ public async Task InferSchema( var fields = order .Select(path => accumulators[path]) - .Select(a => new InferredField(a.Path, a.TypeSummary, a.Occurrences, parsed, a.Example, a.Encryption)) + .Select(a => new InferredField( + a.Path, a.TypeSummary, a.Occurrences, parsed, a.Example, a.Encryption, a.SoftDeleteCandidate)) .ToList(); return new InferredSchema(typeName, parsed, fields); @@ -147,6 +148,47 @@ static void Walk(JsonObject obj, string prefix, Dictionary + /// Names a soft-delete flag is spelled with. Half of the candidate test - the value shape is the other + /// half, and neither alone is allowed to suggest anything. + /// + const string DeletedNameFragment = "deleted"; + + /// + /// Whether a path looks like the soft-delete flag of a type, and which shape it looks like. + /// + /// + /// + /// Soft delete is the one DocumentDb feature that leaves nothing in the database: AddSoftDelete + /// registers an interceptor and a query filter, and the flag is an ordinary JSON property. So there is + /// no catalog signal to read and no honest way to detect it - only a way to suggest it. + /// + /// + /// This is name matching, which the classifier rejects outright - the difference is what it is allowed + /// to do. There a name decided a classification on its own; here it is one of two required signals, and + /// the output is a suggestion the operator confirms into ConnectionProfile.SoftDeleteFlags. A + /// candidate never changes a role, never changes browsability, never adds a feature to the verdict and + /// never changes what the delete button does. A domain property called IsDeleted that means + /// something else is therefore harmless: the worst case is a prompt the operator declines. + /// + /// + static SoftDeleteFlagKind? SoftDeleteCandidateKind(string path, IReadOnlySet types, bool stringsAreTimestamps) + { + var name = path[(path.LastIndexOf('.') + 1)..]; + if (!name.Contains(DeletedNameFragment, StringComparison.OrdinalIgnoreCase)) + return null; + + // The two shapes SoftDeleteMapping.Build accepts, and nothing else: a bool, or a nullable timestamp + // that has actually been seen both set and unset. + if (types.Count == 1 && types.Contains("bool")) + return SoftDeleteFlagKind.Boolean; + + if (types.Count == 2 && types.Contains("null") && types.Contains("string") && stringsAreTimestamps) + return SoftDeleteFlagKind.Timestamp; + + return null; + } + sealed class FieldAccumulator(string path) { readonly SortedSet types = new(StringComparer.Ordinal); @@ -161,6 +203,7 @@ sealed class FieldAccumulator(string path) int encryptedCount; int plaintextCount; bool deterministicObserved; + bool stringsAreTimestamps = true; public string Path { get; } = path; public int Occurrences { get; private set; } @@ -172,6 +215,10 @@ sealed class FieldAccumulator(string path) /// public string TypeSummary => this.types.Count == 0 ? "null" : string.Join(" | ", this.types); + /// See - a suggestion, never a verdict. + public SoftDeleteFlagKind? SoftDeleteCandidate + => SoftDeleteCandidateKind(this.Path, this.types, this.stringsAreTimestamps); + public EncryptedFieldInfo? Encryption => this.encryptedCount == 0 ? null : new EncryptedFieldInfo(this.keyIds, this.encryptedCount, this.plaintextCount, this.deterministicObserved); @@ -182,6 +229,15 @@ public void Observe(JsonNode? value) var encrypted = EncryptedFields.TryRead(value, out var info); this.types.Add(encrypted ? "encrypted" : Describe(value)); + if (!encrypted && value is JsonValue { } stringValue && stringValue.GetValueKind() == JsonValueKind.String) + { + this.stringsAreTimestamps &= DateTimeOffset.TryParse( + stringValue.GetValue(), + System.Globalization.CultureInfo.InvariantCulture, + System.Globalization.DateTimeStyles.RoundtripKind, + out _); + } + if (encrypted) this.ObserveEnvelope(value!, info); else if (value is JsonValue { } scalar && scalar.GetValueKind() != JsonValueKind.Null) diff --git a/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.SoftDelete.cs b/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.SoftDelete.cs new file mode 100644 index 0000000..f8518b9 --- /dev/null +++ b/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.SoftDelete.cs @@ -0,0 +1,192 @@ +using System.Text.Json.Nodes; +using Shiny.DocumentDb; +using ShinyDocDbMyAdmin.Models; + +namespace ShinyDocDbMyAdmin.Services; + +/// +/// Soft delete, as far as an admin tool can honestly go with it. +/// +/// +/// +/// AddSoftDelete creates no column, table or index - it registers an interceptor that turns a +/// Remove into a flag write, and a named query filter that hides flagged documents from every read. +/// Both live in the application's process. Nothing about it reaches the database, so no amount of catalog +/// reading can discover it, and the schema sample can only ever suggest it +/// (SoftDeleteCandidateKind). +/// +/// +/// That left this tool violating the invariant it could not see: flagged documents appeared in the browser +/// mixed in with live ones with nothing to say which was which, and the delete button issued a real +/// DELETE - hard-deleting a document the application would only have flagged, with no warning. The +/// fix is a declaration on the connection (ConnectionProfile.SoftDeleteFlags): the operator states +/// what their application configured, and the tool then treats it as fact. Until they do, +/// behaves exactly as it always has. +/// +/// +/// Restore / PurgeDeleted / HardDelete are library calls this tool cannot make - it +/// writes raw JSON over ADO and has no CLR type to bind to. Their equivalents here are ordinary document +/// writes over the declared path, which is why the declaration has to carry +/// : restoring a boolean writes false, restoring a timestamp writes +/// null. +/// +/// +public sealed partial class DocumentAdminService +{ + /// + /// The soft-delete flag declared for a type on this connection, or null when none is. Null means the + /// tool has no reason to believe the type uses soft delete - not that it does not. + /// + public async Task GetSoftDeleteFlag(string profileId, string typeName, CancellationToken ct = default) + { + var connection = await this.Connect(profileId, ct); + return connection.Profile.Profile.SoftDeleteFlags + .FirstOrDefault(f => f.TypeName.Equals(typeName, StringComparison.Ordinal)); + } + + /// + /// Flags documents as deleted instead of removing them - what the application's interceptor would have + /// done. The flag is written as a document update, so temporal history records it as the version it is. + /// + /// How many documents were flagged. A document that no longer exists is skipped, not an error. + /// The type has no declared soft-delete flag. + public Task SoftDeleteDocuments( + string profileId, + string table, + string typeName, + IReadOnlyList ids, + CancellationToken ct = default) + => this.WriteSoftDeleteFlag(profileId, table, typeName, ids, deleted: true, ct); + + /// + /// Clears the declared flag - false for a boolean, null for a timestamp, matching + /// SoftDeleteMapping.RestoreValue. + /// + /// The type has no declared soft-delete flag. + public Task RestoreDocuments( + string profileId, + string table, + string typeName, + IReadOnlyList ids, + CancellationToken ct = default) + => this.WriteSoftDeleteFlag(profileId, table, typeName, ids, deleted: false, ct); + + async Task WriteSoftDeleteFlag( + string profileId, + string table, + string typeName, + IReadOnlyList ids, + bool deleted, + CancellationToken ct) + { + if (ids.Count == 0) + return 0; + + var flag = await this.GetSoftDeleteFlag(profileId, typeName, ct) + ?? throw new InvalidOperationException( + $"'{typeName}' has no declared soft-delete flag on this connection, so there is nothing to write. " + + "Declare one in the connection's settings, or from the Structure tab."); + + var now = DateTimeOffset.UtcNow; + var written = 0; + + foreach (var id in ids) + { + var row = await this.GetDocument(profileId, table, typeName, id, ct); + if (row?.Body is not JsonObject body) + continue; + + ApplyFlag(body, flag, deleted, now); + + // Through SaveDocument rather than an UPDATE of its own: a flag write is a document write, and + // the history sidecar, the vector sidecar and the encryption guard all have to see it as one. + await this.SaveDocument(profileId, table, typeName, id, body.ToJsonString(Compact), isNew: false, ct: ct); + written++; + } + + logger.LogInformation( + "{Action} {Count} {Type} document(s) in {Table} on declared flag '{Path}'", + deleted ? "Soft-deleted" : "Restored", written, typeName, table, flag.PropertyPath); + + return written; + } + + /// + /// Writes the flag at its declared path, creating the intermediate objects a nested path needs. The + /// values match SoftDeleteMapping exactly: true / false for a boolean, the current + /// UTC instant / null for a timestamp. + /// + static void ApplyFlag(JsonObject body, SoftDeleteFlag flag, bool deleted, DateTimeOffset now) + { + var segments = flag.PropertyPath.Split('.', StringSplitOptions.RemoveEmptyEntries); + if (segments.Length == 0) + throw new InvalidOperationException("The declared soft-delete flag has no property path."); + + var target = body; + for (var i = 0; i < segments.Length - 1; i++) + { + if (target[segments[i]] is JsonObject nested) + { + target = nested; + continue; + } + + var created = new JsonObject(); + target[segments[i]] = created; + target = created; + } + + target[segments[^1]] = (flag.FlagKind, deleted) switch + { + (SoftDeleteFlagKind.Boolean, _) => JsonValue.Create(deleted), + (SoftDeleteFlagKind.Timestamp, true) => JsonValue.Create(now), + _ => null + }; + } + + /// + /// Whether a row the grid already holds is flagged - read from the body it parsed, so badging every row + /// in an "all" listing costs no queries. + /// + public static bool IsFlagged(DocumentRow row, SoftDeleteFlag flag) + { + var node = row.Read(flag.PropertyPath); + + return flag.FlagKind == SoftDeleteFlagKind.Timestamp + ? node is JsonValue timestamp && timestamp.GetValueKind() != System.Text.Json.JsonValueKind.Null + : node is JsonValue boolean && boolean.GetValueKind() == System.Text.Json.JsonValueKind.True; + } + + /// + /// The predicate that partitions a declared type into live and deleted documents, in the same shape the + /// library's query filter uses: a boolean flag is deleted when it is true, a timestamp when it is set. + /// A document with no flag at all is live, which is what an application that added the flag later has. + /// + internal static string SoftDeletePredicate(IDatabaseProvider provider, SoftDeleteFlag flag, bool deleted) + { + if (flag.FlagKind == SoftDeleteFlagKind.Timestamp) + return provider.JsonNullCheck("Data", flag.PropertyPath, isNull: !deleted); + + var isTrue = provider.BoolCondition(provider.JsonExtractTyped("Data", flag.PropertyPath, typeof(bool))); + + return deleted + ? isTrue + : $"({provider.JsonNullCheck("Data", flag.PropertyPath, isNull: true)} OR NOT ({isTrue}))"; + } +} + +/// +/// Which partition of a declared soft-delete type the Browse grid is showing. Only meaningful for a type +/// with a declared flag; an undeclared type is always because there is nothing to split on. +/// +public enum DeletedFilter +{ + /// Live documents only - what the application's own reads see. + Live, + + /// Flagged documents only - what the application has deleted. + Deleted, + + /// Everything, flagged rows included and badged as such. + All +} diff --git a/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.cs b/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.cs index c81973c..0c5470f 100644 --- a/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.cs +++ b/src/ShinyDocDbMyAdmin.Core/Services/DocumentAdminService.cs @@ -21,7 +21,7 @@ public sealed partial class DocumentAdminService(ConnectionManager connections, /// The envelope columns, in the order every SELECT in this class uses. const string EnvelopeColumns = "Id, TypeName, Data, CreatedAt, UpdatedAt"; - readonly ConcurrentDictionary> tableCache = new(); + readonly ConcurrentDictionary catalogCache = new(); public Task Connect(string profileId, CancellationToken ct = default) => connections.Open(profileId, ct); @@ -29,7 +29,7 @@ public Task Connect(string profileId, CancellationToken ct = de /// Forces the next schema read to hit the database. Call after any DDL or a profile edit. public void InvalidateSchema(string profileId) { - this.tableCache.TryRemove(profileId, out _); + this.catalogCache.TryRemove(profileId, out _); // The vector sidecar probe is derived from the table list, so it cannot outlive it. foreach (var key in this.vectorSidecarCache.Keys.Where(k => k.StartsWith(profileId + "|", StringComparison.Ordinal))) @@ -38,84 +38,18 @@ public void InvalidateSchema(string profileId) // ── Discovery ─────────────────────────────────────────────────────── - /// Opens the connection and reports whether the database answers, without changing anything. + /// + /// Opens the connection and reports what the database is, without changing anything. The sentence is + /// - the same verdict the overview banner shows, so the test + /// button and the screen it leads to cannot disagree. + /// public async Task TestConnection(string profileId, CancellationToken ct = default) { - var connection = await this.Connect(profileId, ct); - var tables = await this.ListTables(profileId, refresh: true, ct); - var documentTables = tables.Count(t => t.IsBrowsable); - - return documentTables == 0 - ? $"Connected. No DocumentDb tables found among {tables.Count} table(s) - the store may not be initialised yet." - : $"Connected. Found {documentTables} document table(s) of {tables.Count} total."; - } - - public async Task> ListTables(string profileId, bool refresh = false, CancellationToken ct = default) - { - if (!refresh && this.tableCache.TryGetValue(profileId, out var cached)) - return cached; - - var connection = await this.Connect(profileId, ct); - var tables = await connection.Execute(async (db, token) => - { - var names = new List(); - await using (var cmd = Ado.Command(db, connection.Provider.BuildListTablesSql())) - await using (var reader = await cmd.ExecuteReaderAsync(token)) - { - while (await reader.ReadAsync(token)) - names.Add(Ado.Text(reader, 0)); - } - - var results = new List(names.Count); - foreach (var name in names.OrderBy(x => x, StringComparer.OrdinalIgnoreCase)) - results.Add(await Classify(db, connection, name, token)); - - return (IReadOnlyList)results; - }, ct); - - this.tableCache[profileId] = tables; - return tables; - } - - static async Task Classify(DbConnection db, AdminConnection connection, string name, CancellationToken ct) - { - // The sidecars are named by convention, so recognise them before paying for a probe. - var role = name switch - { - _ when name.EndsWith("_history", StringComparison.OrdinalIgnoreCase) => TableRole.History, - _ when name.EndsWith("_blobs", StringComparison.OrdinalIgnoreCase) => TableRole.Blobs, - // Covers the sidecar itself plus SQLite's R*Tree shadow tables (_spatial_node, - // _spatial_rowid, _spatial_parent), which are created by the engine, not the library. - _ when name.Contains("_spatial", StringComparison.OrdinalIgnoreCase) => TableRole.Sidecar, - _ when name.Contains("_vec", StringComparison.OrdinalIgnoreCase) => TableRole.Sidecar, - _ when name.Contains("_fts", StringComparison.OrdinalIgnoreCase) => TableRole.Sidecar, - _ => TableRole.Foreign - }; - - if (role != TableRole.Foreign) - return new TableInfo(name, role); - - // Otherwise ask the database: a table carrying the whole envelope is one we can browse. - var quoted = connection.Provider.QuoteTable(Ado.SafeIdentifier(name)); - if (!await ColumnsExist(db, $"SELECT {EnvelopeColumns} FROM {quoted} WHERE 1 = 0", ct)) - return new TableInfo(name, TableRole.Foreign); - - var hasTenant = await ColumnsExist(db, $"SELECT TenantId FROM {quoted} WHERE 1 = 0", ct); - return new TableInfo(name, TableRole.Documents, hasTenant); - } + await this.Connect(profileId, ct); + var identity = await this.GetIdentity(profileId, sampleRows: false, refresh: true, ct); - static async Task ColumnsExist(DbConnection db, string probeSql, CancellationToken ct) - { - try - { - await using var cmd = Ado.Command(db, probeSql); - await using var reader = await cmd.ExecuteReaderAsync(ct); - return true; - } - catch (DbException) - { - return false; - } + var reason = identity.Reasons.FirstOrDefault(); + return reason is null ? $"Connected. {identity.Summary}" : $"Connected. {identity.Summary} {reason}"; } /// diff --git a/src/ShinyDocDbMyAdmin.Core/Services/ProfileStore.cs b/src/ShinyDocDbMyAdmin.Core/Services/ProfileStore.cs index 2b5b989..396396b 100644 --- a/src/ShinyDocDbMyAdmin.Core/Services/ProfileStore.cs +++ b/src/ShinyDocDbMyAdmin.Core/Services/ProfileStore.cs @@ -94,6 +94,40 @@ .. profile.EncryptionKeys await this.store.Update(profile, cancellationToken: ct); } + /// + /// Replaces the connection's soft-delete declarations, leaving every other field - including the + /// secrets - untouched. Separate from because declaring a flag is something an + /// operator does from the Structure tab, and it must not require re-typing a connection string. + /// + /// + /// Entries missing a type or a path are dropped rather than stored blank: a declaration with no path + /// would turn every delete of that type into a write to nowhere. + /// + public async Task SaveSoftDeleteFlags(string profileId, IReadOnlyList flags, CancellationToken ct = default) + { + this.AssertNotProvided(profileId); + + var profile = await this.store.Get(profileId, cancellationToken: ct) + ?? throw new InvalidOperationException($"Connection '{profileId}' no longer exists."); + + profile.SoftDeleteFlags = + [ + .. flags + .Where(f => !string.IsNullOrWhiteSpace(f.TypeName) && !string.IsNullOrWhiteSpace(f.PropertyPath)) + .GroupBy(f => f.TypeName.Trim(), StringComparer.Ordinal) + .Select(g => new SoftDeleteFlag + { + // One flag per type, exactly as the library allows - SoftDelete.Register throws on a + // second mapping for the same document type. + TypeName = g.Key, + PropertyPath = g.Last().PropertyPath.Trim(), + FlagKind = g.Last().FlagKind + }) + ]; + + await this.store.Update(profile, cancellationToken: ct); + } + public async Task Delete(string id, CancellationToken ct = default) { this.AssertNotProvided(id); diff --git a/src/ShinyDocDbMyAdmin.Core/Services/ProvidedConnections.cs b/src/ShinyDocDbMyAdmin.Core/Services/ProvidedConnections.cs index 22855c0..7bbafa6 100644 --- a/src/ShinyDocDbMyAdmin.Core/Services/ProvidedConnections.cs +++ b/src/ShinyDocDbMyAdmin.Core/Services/ProvidedConnections.cs @@ -35,6 +35,23 @@ public sealed class ProvidedConnections /// Prefix that keeps provided profile ids from colliding with saved (GUID) ones. public const string IdPrefix = "provided-"; + /// + /// Per-store section declaring which document types the application configured for soft delete, keyed by + /// stored type name. Either shape works: + /// + /// Shiny:DocumentDb:store:SoftDelete:Customer = isDeleted + /// Shiny:DocumentDb:store:SoftDelete:Invoice:PropertyPath = deletedAt + /// Shiny:DocumentDb:store:SoftDelete:Invoice:FlagKind = Timestamp + /// + /// + /// + /// It has to be here rather than only in the UI: a provided connection is never written to the profile + /// store, so there would otherwise be nowhere to declare one - and an Aspire-hosted admin would keep + /// hard-deleting documents its application only ever flags. The host declares the connection, so the + /// host declares its soft-deleted types. + /// + public const string SoftDeleteSetting = "SoftDelete"; + readonly Dictionary byId = new(StringComparer.OrdinalIgnoreCase); public ProvidedConnections(IConfiguration configuration, ILogger logger) @@ -79,7 +96,8 @@ public ProvidedConnections(IConfiguration configuration, ILogger this.byId.GetValueOrDefault(profileId); + /// Reads the declared soft-delete flags for one store. See . + static IEnumerable ReadSoftDeleteFlags(IConfigurationSection section, string name, ILogger logger) + { + foreach (var entry in section.GetChildren()) + { + // The compact form is the value itself; the long form spells the two fields out. A boolean flag + // is the common case, so it is what the compact form means. + var path = entry.Value ?? entry["PropertyPath"]; + if (string.IsNullOrWhiteSpace(path)) + { + logger.LogWarning( + "Ignoring soft-delete declaration '{Type}' on provided connection '{Name}': it names no property path.", + entry.Key, name); + continue; + } + + var kindName = entry.Value is null ? entry["FlagKind"] : null; + if (!string.IsNullOrWhiteSpace(kindName) && !Enum.TryParse(kindName, ignoreCase: true, out _)) + { + logger.LogWarning( + "Ignoring soft-delete declaration '{Type}' on provided connection '{Name}': '{Kind}' is not Boolean or Timestamp.", + entry.Key, name, kindName); + continue; + } + + yield return new SoftDeleteFlag + { + TypeName = entry.Key, + PropertyPath = path.Trim(), + FlagKind = string.IsNullOrWhiteSpace(kindName) + ? SoftDeleteFlagKind.Boolean + : Enum.Parse(kindName, ignoreCase: true) + }; + } + } + /// /// The literal connection string if there is one, otherwise the base64 form decoded. A literal /// wins: it is the ordinary contract, and the encoded form exists only for hosts that cannot diff --git a/src/ShinyDocDbMyAdmin.Tui/Panels/BrowsePanel.cs b/src/ShinyDocDbMyAdmin.Tui/Panels/BrowsePanel.cs index fdda14f..98de1df 100644 --- a/src/ShinyDocDbMyAdmin.Tui/Panels/BrowsePanel.cs +++ b/src/ShinyDocDbMyAdmin.Tui/Panels/BrowsePanel.cs @@ -37,6 +37,12 @@ public sealed class BrowsePanel(WorkspaceContext context) : WorkspacePanel(conte BrowseSort sort = new("UpdatedAt", true, true); DocumentPage? current; + /// The flag declared for this type on the connection, or null. Never inferred - see + /// DocumentAdminService.SoftDelete. + SoftDeleteFlag? softDelete; + + DeletedFilter deleted = DeletedFilter.Live; + protected override Visual Build() { this.grid.OnActivate(this.Edit); @@ -50,6 +56,12 @@ protected override Visual Build() Ui.Primary("New", this.Insert).IsVisible(() => this.Context.CanWrite), Ui.Action("Edit", () => this.WithSelection(this.Edit)), Ui.Danger("Delete", () => this.WithSelection(this.Delete)).IsVisible(() => this.Context.CanWrite), + // Both appear only for a declared type: with no declaration there is nothing to restore from + // and nothing to partition on. + Ui.Action("Restore", () => this.WithSelection(this.Restore)) + .IsVisible(() => this.Context.CanWrite && this.softDelete is not null), + Ui.Action("Live / deleted / all", this.CycleDeleted) + .IsVisible(() => this.softDelete is not null), Ui.Action("Filters", this.EditFilters), Ui.Action("Columns", this.EditColumns), Ui.Action("Sort", this.EditSort), @@ -104,6 +116,9 @@ async Task Fetch(bool reinfer, CancellationToken ct) { try { + var flag = await this.Context.Admin.GetSoftDeleteFlag(this.Context.ProfileId, this.Context.TypeName, ct); + this.Context.Post(() => this.softDelete = flag); + if (reinfer || this.schema is null) { var inferred = await this.Context.Admin.InferSchema(this.Context.ProfileId, this.Context.Table, this.Context.TypeName, ct: ct); @@ -132,7 +147,8 @@ async Task Fetch(bool reinfer, CancellationToken ct) Sort = this.sort, Filters = [.. this.filters.Where(f => !string.IsNullOrWhiteSpace(f.Path))], Search = this.searchText, - SearchPaths = this.schema is null ? ["Id"] : DocumentAdminService.SearchableColumns(this.schema) + SearchPaths = this.schema is null ? ["Id"] : DocumentAdminService.SearchableColumns(this.schema), + Deleted = this.softDelete is null ? DeletedFilter.All : this.deleted }; var result = await this.Context.Admin.Browse(this.Context.ProfileId, this.Context.Table, this.Context.TypeName, query, ct); @@ -141,8 +157,12 @@ async Task Fetch(bool reinfer, CancellationToken ct) { this.current = result; this.error.Value = ""; - this.grid.SetRows(result.Rows); - this.pageLabel.Value = $"[dim]page {result.Page} of {result.PageCount} · {Ui.Number(result.TotalCount)} document(s)[/]"; + this.grid.SetRows(result.Rows, flag); + + // The scope is part of the count: "12 documents" under a live-only filter is a different + // claim from "12 documents", and only one of them is true. + var scope = flag is null ? "" : $" · {this.deleted.ToString().ToLowerInvariant()}"; + this.pageLabel.Value = $"[dim]page {result.Page} of {result.PageCount} · {Ui.Number(result.TotalCount)} document(s){scope}[/]"; this.Shell.Status.Value = $"sorted by {this.sort.Path} {(this.sort.Descending ? "desc" : "asc")}"; }); } @@ -177,6 +197,25 @@ void Delete(DocumentRow row) return; } + if (this.softDelete is { } flag) + { + // A declared type is only ever flagged by the application, so that is what delete means here. + // Removing the row for real is the Purge command, which says so. + Modal.Confirm( + this.Shell, + "Flag deleted", + $"Flag '{row.Id}' deleted? '{this.Context.TypeName}' is declared for soft delete on '{flag.PropertyPath}', so the document stays in the table and can be restored.", + "Flag deleted", + () => this.Context.Run(async ct => + { + await this.Context.Admin.SoftDeleteDocuments(this.Context.ProfileId, this.Context.Table, this.Context.TypeName, [row.Id], ct); + await this.Fetch(reinfer: false, ct); + this.Context.Post(() => this.Shell.Success($"Flagged '{row.Id}' deleted.")); + }, "Could not flag the document") + ); + return; + } + Modal.Confirm( this.Shell, "Delete document", @@ -184,13 +223,42 @@ void Delete(DocumentRow row) "Delete", () => this.Context.Run(async ct => { - await this.Context.Admin.DeleteDocuments(this.Context.ProfileId, this.Context.Table, this.Context.TypeName, [row.Id], ct); + await this.Context.Admin.DeleteDocuments(this.Context.ProfileId, this.Context.Table, this.Context.TypeName, [row.Id], permanent: true, ct); await this.Fetch(reinfer: false, ct); this.Context.Post(() => this.Shell.Success($"Deleted '{row.Id}'.")); }, "Could not delete the document") ); } + /// Clears the declared flag - false for a boolean, null for a timestamp. + void Restore(DocumentRow row) + { + if (this.softDelete is null) + { + this.Shell.Info($"'{this.Context.TypeName}' has no declared soft-delete flag, so there is nothing to restore."); + return; + } + + this.Context.Run(async ct => + { + await this.Context.Admin.RestoreDocuments(this.Context.ProfileId, this.Context.Table, this.Context.TypeName, [row.Id], ct); + await this.Fetch(reinfer: false, ct); + this.Context.Post(() => this.Shell.Success($"Restored '{row.Id}'.")); + }, "Could not restore the document"); + } + + void CycleDeleted() + { + this.deleted = this.deleted switch + { + DeletedFilter.Live => DeletedFilter.Deleted, + DeletedFilter.Deleted => DeletedFilter.All, + _ => DeletedFilter.Live + }; + + this.GoTo(1); + } + // ── Filters, columns and sort ─────────────────────────────────────── IReadOnlyList AllPaths => diff --git a/src/ShinyDocDbMyAdmin.Tui/Screens/DatabaseOverviewScreen.cs b/src/ShinyDocDbMyAdmin.Tui/Screens/DatabaseOverviewScreen.cs index 4d123d3..6bced1b 100644 --- a/src/ShinyDocDbMyAdmin.Tui/Screens/DatabaseOverviewScreen.cs +++ b/src/ShinyDocDbMyAdmin.Tui/Screens/DatabaseOverviewScreen.cs @@ -14,6 +14,7 @@ public sealed partial class TableRow { [Bindable] public partial string Table { get; set; } [Bindable] public partial string Role { get; set; } + [Bindable] public partial string Owner { get; set; } [Bindable] public partial string Documents { get; set; } [Bindable] public partial string Types { get; set; } [Bindable] public partial string Size { get; set; } @@ -34,18 +35,24 @@ public sealed class DatabaseOverviewScreen(AdminShell shell, string profileId) : readonly RowGrid grid = new( (TableRow.Accessor.Table, "Table", ColumnWidth.Fill), (TableRow.Accessor.Role, "Role", ColumnWidth.Fit), + (TableRow.Accessor.Owner, "Belongs to", ColumnWidth.Fit), (TableRow.Accessor.Documents, "Documents", ColumnWidth.Fit), (TableRow.Accessor.Types, "Types", ColumnWidth.Fit), (TableRow.Accessor.Size, "JSON size", ColumnWidth.Fit) ); readonly State heading = new(""); + readonly State verdict = new(""); readonly State error = new(""); readonly State hasOutbox = new(false); readonly State hasStreams = new(false); + readonly State showForeign = new(false); ConnectionProfile? profile; + /// Every table, classified. The grid shows a filtered view of this. + IReadOnlyList<(TableInfo Table, TableStats? Stats)> all = []; + public string ProfileId { get; } = profileId; public override string Title => this.profile?.Name ?? "Database"; @@ -56,8 +63,10 @@ protected override Visual Build() { if (row.Info.IsBrowsable) this.Shell.Push(new TableOverviewScreen(this.Shell, this.ProfileId, row.Info.Name)); + else if (row.Info.IsOwned) + this.Shell.Info($"'{row.Info.Name}' is {row.Info.Owner}'s {row.Role} sidecar - it is maintained alongside that table, not browsed on its own."); else - this.Shell.Info($"'{row.Info.Name}' is a {row.Role} sidecar - it is maintained alongside a document table, not browsed on its own."); + this.Shell.Info($"'{row.Info.Name}' is not a DocumentDb table. Nothing here reads or writes it."); }); var toolbar = Ui.Toolbar( @@ -72,11 +81,16 @@ protected override Visual Build() Ui.Action("Assistant", this.OpenAssistant).IsVisible(this.Shell.AiAvailability.IsEnabled), Ui.Action("Settings", () => this.Shell.Push(new ConnectionEditScreen(this.Shell, this.ProfileId))), Ui.Action("New table", this.CreateTable), + // Counted in the verdict either way; this is what makes them visible without making them noise. + Ui.Action("Foreign tables", this.ToggleForeign), Ui.Action("Refresh", () => this.Shell.Reload()) ); return new Padder(new VStack( new Markup(() => this.heading.Value), + // The question an operator arrives with, answered above the list rather than left to be + // inferred from it. + new Markup(() => this.verdict.Value), toolbar, new ComputedVisual(() => this.error.Value.Length == 0 ? Ui.Nothing() @@ -123,6 +137,7 @@ public override async Task Load(CancellationToken ct) try { var tables = await this.Shell.Admin.ListTables(this.ProfileId, refresh: true, ct); + var identity = await this.Shell.Admin.GetIdentity(this.ProfileId, ct: ct); rows = []; foreach (var table in tables) @@ -155,19 +170,12 @@ public override async Task Load(CancellationToken ct) this.error.Value = ""; this.hasOutbox.Value = outboxes.Count > 0; this.hasStreams.Value = streams.Count > 0; - this.grid.Replace(rows.Select(entry => new TableRow - { - Table = entry.Table.Name, - Role = entry.Table.Role == TableRole.Documents - ? (entry.Table.HasTenantColumn ? "documents · tenant" : "documents") - : entry.Table.Role.ToString().ToLowerInvariant(), - Documents = entry.Stats is null ? "" : Ui.Number(entry.Stats.DocumentCount), - Types = entry.Stats is null ? "" : Ui.Number(entry.Stats.TypeCount), - Size = entry.Stats is null ? "" : Ui.Bytes(entry.Stats.TotalJsonBytes), - Info = entry.Table - })); - - this.Shell.Status.Value = $"{rows.Count(r => r.Table.IsBrowsable)} document table(s)"; + this.all = rows; + this.showForeign.Value = !(loaded.HideForeignTables); + this.verdict.Value = Verdict(identity); + this.Fill(); + + this.Shell.Status.Value = identity.Summary; }); } catch (Exception ex) @@ -180,6 +188,72 @@ public override async Task Load(CancellationToken ct) } } + /// + /// Rebuilds the grid from the classified list. Foreign tables are filtered here rather than dropped at + /// read time, so toggling them is instant and the verdict's counts always describe the whole database. + /// + void Fill() + { + var visible = this.showForeign.Value + ? this.all + : [.. this.all.Where(e => e.Table.IsOwned)]; + + this.grid.Replace(visible.Select(entry => new TableRow + { + Table = entry.Table.Name, + Role = Describe(entry.Table), + Owner = entry.Table.Owner ?? "", + Documents = entry.Stats is null ? "" : Ui.Number(entry.Stats.DocumentCount), + Types = entry.Stats is null ? "" : Ui.Number(entry.Stats.TypeCount), + Size = entry.Stats is null ? "" : Ui.Bytes(entry.Stats.TotalJsonBytes), + Info = entry.Table + })); + } + + void ToggleForeign() + { + this.showForeign.Value = !this.showForeign.Value; + this.Fill(); + + var foreign = this.all.Count(e => !e.Table.IsOwned); + this.Shell.Info(this.showForeign.Value + ? $"Showing {foreign} table(s) that are not DocumentDb's." + : $"Hiding {foreign} table(s) that are not DocumentDb's."); + } + + static string Describe(TableInfo table) + { + if (table.Role == TableRole.Foreign) + return "foreign"; + + if (table.Role != TableRole.Documents) + { + var role = table.Role.ToString().ToLowerInvariant(); + return table.Feature is { } detail ? $"{role} · {detail}" : role; + } + + var parts = new List { "documents" }; + if (table.HasTenantColumn) parts.Add("tenant"); + if (table.Confidence == TableConfidence.Probable) parts.Add("probable"); + + return string.Join(" · ", parts); + } + + static string Verdict(DatabaseIdentity identity) + { + var colour = identity switch + { + { Participates: false } => "red", + { Confidence: IdentityConfidence.Probable } => "yellow", + _ => "green" + }; + + var reason = identity.Reasons.FirstOrDefault(); + var tail = reason is null ? "" : $" [dim]{Ui.Escape(reason)}[/]"; + + return $"[{colour}]{Ui.Escape(identity.Summary)}[/]{tail}"; + } + void CreateTable() => Modal.Prompt( this.Shell, "New documents table", diff --git a/src/ShinyDocDbMyAdmin.Tui/Screens/TableOverviewScreen.cs b/src/ShinyDocDbMyAdmin.Tui/Screens/TableOverviewScreen.cs index fa0cf1e..4873e2f 100644 --- a/src/ShinyDocDbMyAdmin.Tui/Screens/TableOverviewScreen.cs +++ b/src/ShinyDocDbMyAdmin.Tui/Screens/TableOverviewScreen.cs @@ -116,22 +116,33 @@ public override async Task Load(CancellationToken ct) } } - void ClearType(TypeRow row) => Modal.Confirm( - this.Shell, - "Empty type", - $"Delete every '{row.Info.TypeName}' document in {this.Table}? {row.Documents} document(s) go, and so do their history, blob and vector sidecar rows. This cannot be undone from here.", - "Delete them all", - () => this.Shell.RunAsync(async ct => - { - var deleted = await this.Shell.Admin.ClearType(this.ProfileId, this.Table, row.Info.TypeName, ct); - await this.Load(ct); - this.Shell.Post(() => + void ClearType(TypeRow row) => this.Shell.RunAsync(async ct => + { + // Asked before the prompt is written, because a declared soft-delete type needs a different one: + // the application's own Clear flags these documents, and this removes them. + var flag = await this.Shell.Admin.GetSoftDeleteFlag(this.ProfileId, row.Info.TypeName, ct); + + var warning = flag is null + ? "This cannot be undone from here." + : $"'{row.Info.TypeName}' is declared for soft delete on '{flag.PropertyPath}' - the application's own Clear would only flag these documents. This removes them for real, and cannot be undone from here."; + + this.Shell.Post(() => Modal.Confirm( + this.Shell, + "Empty type", + $"Delete every '{row.Info.TypeName}' document in {this.Table}? {row.Documents} document(s) go, and so do their history, blob and vector sidecar rows. {warning}", + flag is null ? "Delete them all" : "Delete them all permanently", + () => this.Shell.RunAsync(async token => { - this.Shell.Success($"Deleted {deleted} document(s)."); - this.Shell.ReloadExplorer(); - }); - }, "Could not empty the type") - ); + var deleted = await this.Shell.Admin.ClearType(this.ProfileId, this.Table, row.Info.TypeName, permanent: true, token); + await this.Load(token); + this.Shell.Post(() => + { + this.Shell.Success($"Deleted {deleted} document(s)."); + this.Shell.ReloadExplorer(); + }); + }, "Could not empty the type") + )); + }, "Could not empty the type"); public override IEnumerable Commands() => [ diff --git a/src/ShinyDocDbMyAdmin.Tui/Widgets/DocumentGrid.cs b/src/ShinyDocDbMyAdmin.Tui/Widgets/DocumentGrid.cs index 151f48d..de0bb79 100644 --- a/src/ShinyDocDbMyAdmin.Tui/Widgets/DocumentGrid.cs +++ b/src/ShinyDocDbMyAdmin.Tui/Widgets/DocumentGrid.cs @@ -10,10 +10,17 @@ namespace ShinyDocDbMyAdmin.Tui.Widgets; /// A browse row: the envelope, plus whichever JSON paths are currently shown as columns. -public sealed class DocumentGridRow(DocumentRow row) +/// +/// The flag declared for this type, or null. Present only so a flagged row can say so in a grid that has +/// no colour to spare - the value comes from the body already parsed, so it costs no query. +/// +public sealed class DocumentGridRow(DocumentRow row, SoftDeleteFlag? softDelete) { public DocumentRow Row { get; } = row; + /// True when the application would consider this document deleted. + public bool IsFlagged { get; } = softDelete is not null && DocumentAdminService.IsFlagged(row, softDelete); + /// /// One column's text. Envelope columns come from the envelope; anything else is a dotted path into /// the body. @@ -26,7 +33,7 @@ public sealed class DocumentGridRow(DocumentRow row) /// public string Read(string path) => path switch { - "Id" => this.Row.Id, + "Id" => this.IsFlagged ? this.Row.Id + " (deleted)" : this.Row.Id, "TypeName" => this.Row.TypeName, "CreatedAt" => Ui.Timestamp(this.Row.CreatedAt), "UpdatedAt" => Ui.Timestamp(this.Row.UpdatedAt), @@ -130,7 +137,7 @@ public void SetColumns(IReadOnlyList paths) /// sorting a page rather than the query would reorder twenty-five rows out of thousands and look /// like it had done something. /// - public void SetRows(IEnumerable rows) + public void SetRows(IEnumerable rows, SoftDeleteFlag? softDelete = null) { using (this.document.BeginUpdate()) { @@ -139,7 +146,7 @@ public void SetRows(IEnumerable rows) this.document.RemoveRows(0, existing); foreach (var row in rows) - this.document.AddRow(new DocumentGridRow(row)); + this.document.AddRow(new DocumentGridRow(row, softDelete)); } this.revision.Value++; diff --git a/src/ShinyDocDbMyAdmin/Components/Pages/ConnectionEdit.razor b/src/ShinyDocDbMyAdmin/Components/Pages/ConnectionEdit.razor index 77f5671..aff02a3 100644 --- a/src/ShinyDocDbMyAdmin/Components/Pages/ConnectionEdit.razor +++ b/src/ShinyDocDbMyAdmin/Components/Pages/ConnectionEdit.razor @@ -87,6 +87,49 @@ else Read-only (block every write, and refuse non-SELECT statements in the SQL console) + + +
+ Soft-deleted types (optional) + + AddSoftDelete writes no column, table or index — it is an interceptor and a query + filter inside your application — so this tool cannot discover it. Until a type + is declared here, deleting one of its documents from this tool is a permanent delete, and the + application would only have flagged it. Declare a type and the delete button flags instead, + Browse gains a live / deleted / all filter, and flagged documents are badged. The Structure tab + suggests candidates it recognises. + + + @foreach (var flag in this.softDeleteFlags) + { +
+ + + + +
+ } + +
+ + + The stored TypeName, and the JSON path of the flag — + bool restores to false, + timestamp to null. + +
+
+
🔒 Field-level encryption keys (optional) @@ -133,6 +176,7 @@ else string connectionString = ""; string password = ""; List encryptionKeys = []; + List softDeleteFlags = []; UploadResult? uploaded; string? message; string messageClass = ""; @@ -151,6 +195,7 @@ else this.profile = new ConnectionProfile(); this.connectionString = this.Descriptor.ConnectionStringTemplate; this.encryptionKeys = []; + this.softDeleteFlags = []; return; } @@ -166,6 +211,16 @@ else this.connectionString = this.Profiles.RevealConnectionString(existing); this.password = this.Profiles.RevealPassword(existing) ?? ""; this.encryptionKeys = [.. this.Profiles.RevealEncryptionKeys(existing)]; + + // Copied for the same reason as the keys: the form edits its own list, and Save replaces the + // profile's with what survived validation. + this.softDeleteFlags = + [ + .. existing.SoftDeleteFlags.Select(f => new SoftDeleteFlag + { + TypeName = f.TypeName, PropertyPath = f.PropertyPath, FlagKind = f.FlagKind + }) + ]; } void OnProviderChanged(ChangeEventArgs e) @@ -232,6 +287,16 @@ else // Bound to the form's own list rather than edited in place on the profile: Save replaces the // list with wrapped copies, and re-editing a wrapped value would double-wrap it. this.profile.EncryptionKeys = this.encryptionKeys; + this.profile.SoftDeleteFlags = + [ + .. this.softDeleteFlags + .Where(f => !string.IsNullOrWhiteSpace(f.TypeName) && !string.IsNullOrWhiteSpace(f.PropertyPath)) + .Select(f => new SoftDeleteFlag + { + TypeName = f.TypeName.Trim(), PropertyPath = f.PropertyPath.Trim(), FlagKind = f.FlagKind + }) + ]; + await this.Profiles.Save(this.profile, this.connectionString.Trim(), this.password); // The provider instance is built from these values, so any cached one is now wrong. diff --git a/src/ShinyDocDbMyAdmin/Components/Pages/DatabaseOverview.razor b/src/ShinyDocDbMyAdmin/Components/Pages/DatabaseOverview.razor index 9e06722..ff54d8e 100644 --- a/src/ShinyDocDbMyAdmin/Components/Pages/DatabaseOverview.razor +++ b/src/ShinyDocDbMyAdmin/Components/Pages/DatabaseOverview.razor @@ -58,7 +58,26 @@ else else { var documents = this.tables.Where(t => t.IsBrowsable).ToList(); - var others = this.tables.Where(t => !t.IsBrowsable).ToList(); + var internals = this.tables.Where(t => t.IsOwned && !t.IsBrowsable).ToList(); + var foreign = this.tables.Where(t => t.Role == TableRole.Foreign).ToList(); + + @* The question an operator arrives with, answered before the table list rather than left to be + inferred from a filtered one. *@ + @if (this.identity is { } verdict) + { +
+ @verdict.Summary +
+ Why +
    + @foreach (var reason in verdict.Reasons) + { +
  • @reason
    • + } +
+
+
+ }
@@ -102,6 +121,12 @@ else { tenant } + @if (table.Confidence == TableConfidence.Probable) + { + @* Browsable either way. The badge says why the tool is unsure + instead of pretending it is not. *@ + probable + } @if (this.stats.TryGetValue(table.Name, out var s)) { @@ -138,24 +163,36 @@ else }
- @if (others.Count > 0) + @if (internals.Count > 0) {
-
Other tables
+
+ DocumentDb internals + + @internals.Count table(s) this store created +
+ - @foreach (var table in others) + @foreach (var table in internals) { - + + } @@ -163,6 +200,46 @@ else } + + @if (foreign.Count > 0) + { + @* Counted, never hidden outright. These are someone else's tables and this tool will not touch + them - but a database that silently omitted half of what is in it would be lying about it. *@ +
+
+ Other tables in this database + + +
+ @if (this.showForeign) + { +
+
Table RoleBelongs to
@table.Name@Describe(table.Role) + @Describe(table.Role) + @if (table.Feature is { } detail) + { + @detail + } + @table.Owner
+ + + + + @foreach (var table in foreign) + { + + } + +
Table
@table.Name
+
+ } + else + { +
+

+ @foreign.Count table(s) here are not DocumentDb's - neither documents nor a sidecar + of one. Nothing in this tool reads or writes them. +

+
+ } +
+ } } } @@ -171,6 +248,8 @@ else ConnectionProfile? profile; IReadOnlyList tables = []; + DatabaseIdentity? identity; + bool showForeign; readonly Dictionary stats = []; string? error; bool busy; @@ -227,6 +306,8 @@ else try { this.tables = await this.Admin.ListTables(this.ProfileId, refresh); + this.identity = await this.Admin.GetIdentity(this.ProfileId, ct: default); + this.showForeign = !(this.profile?.HideForeignTables ?? true); // Render the table list first, then fill the counts in - a slow COUNT(*) on a large table // should not hold up the page. @@ -274,9 +355,12 @@ else static string Describe(TableRole role) => role switch { - TableRole.History => "Temporal history sidecar", - TableRole.Blobs => "Blob sidecar", - TableRole.Sidecar => "Spatial / vector / full-text sidecar", + TableRole.History => "Temporal history", + TableRole.Blobs => "Blobs", + TableRole.Spatial => "Spatial index", + TableRole.Vector => "Vector index", + TableRole.FullText => "Full-text index", + TableRole.Documents => "Documents", _ => "Not a DocumentDb table" }; } diff --git a/src/ShinyDocDbMyAdmin/Components/Panels/BrowseTab.razor b/src/ShinyDocDbMyAdmin/Components/Panels/BrowseTab.razor index 08b5781..3e363e7 100644 --- a/src/ShinyDocDbMyAdmin/Components/Panels/BrowseTab.razor +++ b/src/ShinyDocDbMyAdmin/Components/Panels/BrowseTab.razor @@ -21,6 +21,17 @@ + @if (this.softDelete is not null) + { + @* Only for a declared type. Nothing in the database says a type is soft-deleted, so this + control appears exactly when the operator has told us it is. *@ + + }