docs: expand clowder/database annotation reconciliation details

docs/faq.md

@@ -76,19 +76,73 @@ database will receive a new hostname. With the legacy way of choosing database s
 which will end up causing much disruption. 
 
 In Clowder 0.16.0, a new way of declaring database secrets in *app-interface* mode was added, this
-makes use of annotations to decide which database credential to expose to the application. This
-takes the form of the following example and should be applied in app-interface
+makes use of annotations to decide which database credential to expose to the application. The
+annotation is added to the RDS resource definition in app-interface, and the value **must match
+the ClowdApp's `.metadata.name`** — not the database name, namespace, or RDS identifier.
+
+For example, given a ClowdApp named `app-d`:
+
+```yaml
+apiVersion: cloud.redhat.com/v1alpha1
+kind: ClowdApp
+metadata:
+  name: app-d       # <-- this is the value the annotation must match
+spec:
+  database:
+    name: inventory  # <-- this is the database name, NOT used for annotation matching
+```
+
+The corresponding RDS resource in app-interface would be annotated like this:
 
 ```yaml
 terraformResources:
   - name: my_resource
     annotations:
-      clowder/database: app-d
+      clowder/database: app-d  # matches ClowdApp .metadata.name, not database.name
 ```
 
-In this example, the ClowdApp would have the database name set to ``app-d``. To migrate to a new
-database, bring up the second secret with the annotation applied to it, and remove it from the
-first.
+To migrate to a new database, add the annotation to the new resource's secret and remove it from
+the old one. Clowder will pick up the newly annotated secret on the next reconciliation.
+
+#### Why the annotation is required for restores
+
+Without the annotation, Clowder falls back to matching secrets by hostname using the pattern
+`<name>-<env>`. A restored RDS instance typically has a different identifier (e.g.
+`mydb-restore`) that does not follow this pattern, so the fallback will not find it. The
+`clowder/database` annotation bypasses hostname matching entirely and directly associates the
+secret with the correct ClowdApp.
+
+#### Important details
+
+- The annotation value **must match the ClowdApp's `.metadata.name`**, not the namespace or the
+  RDS identifier. This is typically the `resourceTemplates[].name` value in the service's
+  saas-file. You can verify by checking the deployment template for the `.metadata.name` of the
+  ClowdApp resource.
+- The annotation **must be present before Clowder reconciles the ClowdApp** during the restore.
+  Adding it after the fact does not trigger re-reconciliation.
+- Move the annotation in a single change: remove it from the old resource and add it to the new
+  one so that the next reconciliation picks up the correct secret.
+
+#### Example: Swapping the annotation during a restore
+
+```yaml
+resources:
+  # Original database — remove annotation
+  - provider: rds
+    identifier: mydb
+    defaults: defaults.yml
+
+  # Restored database — add annotation
+  - provider: rds
+    identifier: mydb-restore
+    defaults: defaults.yml
+    annotations:
+      clowder/database: "my-clowdapp-name"
+    overrides:
+      restore_to_point_in_time:
+        restore_time: '<timestamp>'
+        source_db_instance_identifier: mydb
+```
 
 
 ## Web Services

docs/providers/database.md

@@ -76,14 +76,52 @@ ClowdEnv Config options available:
 #### app-interface
 
 In app-interface mode, the Clowder operator does not create any resources and
-simply passes through configuration from a secret to the client config. First
-the provider will search for any secrets that have the annotation of
-``clowder/database: <app-name>`` where the app-name matches the ClowdApp name.
-If this cannot be found then the provider will search all secrets in the same
-namespace looking for a hostname which is of the form
-`<name>-<env>.*********` where `name` is the name defined in the
-`ClowdApp` `database` stanza, and `env` is usually one of either
-`stage` or `prod`.
+simply passes through configuration from a secret to the client config.
+
+##### Secret Lookup Order
+
+Clowder uses a two-tier lookup to find the database secret for each ClowdApp:
+
+1. **Annotation match (preferred):** Clowder searches all secrets in the
+   namespace for one with the annotation `clowder/database: <app-name>`, where
+   `<app-name>` matches the ClowdApp's `.metadata.name`. If a matching
+   annotated secret is found, it is used immediately and the fallback is
+   skipped.
+
+2. **Hostname fallback:** If no annotated secret is found, Clowder falls back
+   to searching all secrets in the namespace for one whose hostname matches the
+   pattern `<name>-<env>.*`, where `name` is the value of the `database.name`
+   field in the ClowdApp spec and `env` is usually `stage` or `prod`. The
+   hostname is extracted from the secret data and split on `-` to derive the
+   database name.
+
+If neither method finds a matching secret, Clowder reports a missing
+dependency and the ClowdApp will not become ready.
+
+##### Why the Annotation Matters
+
+The hostname fallback relies on the RDS instance identifier following the
+`<name>-<env>` naming convention. This breaks in scenarios where the actual
+hostname does not match, such as:
+
+- **Database snapshot restores** where the restored instance has a different
+  identifier (e.g. `mydb-restore` instead of `mydb`).
+- **Database migrations** where a new RDS instance is provisioned alongside the
+  original.
+
+In these cases the `clowder/database` annotation is the only reliable way to
+associate the correct secret with the ClowdApp.
+
+##### Key Details
+
+- The annotation value **must match the ClowdApp's `.metadata.name`** (e.g.
+  `export-service`), not the namespace, RDS identifier, or database name.
+- The annotation **must be present on the secret before Clowder reconciles the
+  ClowdApp**. Adding it after the initial reconciliation does not trigger a new
+  reconciliation automatically.
+- When migrating databases, **remove the annotation from the old secret and add
+  it to the new one** in a single change so that the next reconciliation picks
+  up the correct secret.
 
 ## Generated App Configuration