Skip to content

[Async AE 4] Complete query metadata cache asynchronously - #5081

Open
cheenamalhotra wants to merge 5 commits into
dev/cheena/ae-async-parameter-discoveryfrom
dev/cheena/ae-async-metadata-cache
Open

cheenamalhotra wants to merge 5 commits into
dev/cheena/ae-async-parameter-discoveryfrom
dev/cheena/ae-async-metadata-cache

Conversation

@cheenamalhotra

@cheenamalhotra cheenamalhotra commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Description

Layer 4 of 5 split from #4585. Stacked on #5080.

  • SqlQueryMetadataCache: splits the lookup into an in-memory probe (TryGetCachedQueryMetadata) and CompleteCachedQueryMetadataAsync, which loads keys one by one with async key store calls.
    • Stale key: the entry is evicted and the lookup is a miss.
    • Canceled: the exception propagates and good metadata stays cached.
    • Other errors: the entry is evicted and the error is rethrown.
  • SqlCommand.PrepareForTransparentEncryption: on an async cache hit, keys load on a worker (Task.Run). A stale key falls back to describe parameter encryption on that same worker, never on the caller thread. The sync path is unchanged.

Review fixes, for the window while cached keys load:

  • _enclaveAttestationInProgress stays set until execution starts and is cleared on success, failure and cancellation. Until then, changing Connection or Transaction, or starting a second execution, throws.
  • Execution is rejected if the connection was replaced, closed and reopened, or switched with ChangeDatabase. This is checked before the stale-key fallback and again before execution.
  • ValidateConnectionForExecute runs before the stale-key fallback describe.

Out of scope: the gap between the write completing and the read stage publishing PendingAsyncOperation already exists on main for every async path (open thread).

Deferred to layer 5: async enclave package and key loading, and retry changes.

Issues

Part of #4585.

Testing

  • New UnitTests/Microsoft/Data/SqlClient/SqlQueryMetadataCacheTests.cs: async hit, stale and cancel cases, plus sync hit and stale cases.
  • SqlCommandTests: worker dispatch on an async cache hit, and stale-key fallback.
  • New SimulatedServerTests/SqlCommandQueryMetadataCacheAsyncTests.cs: 7 guard tests using a gated key store provider. They cover setter and second-execution rejection, the guard being released on cancel, reopen before execution and before the fallback, a reader opened before the fallback, and ChangeDatabase.
  • dotnet test src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft.Data.SqlClient.UnitTests.csproj -f net10.0 -c Debug|Release --filter "FullyQualifiedName~SqlQueryMetadataCacheTests|FullyQualifiedName~SymmetricKeyCacheTests|FullyQualifiedName~SqlSecurityUtilityTests|FullyQualifiedName~SqlCommandTests|FullyQualifiedName~SqlCommandQueryMetadataCacheAsyncTests|FullyQualifiedName~SqlCommandEnclaveAsyncTests": 160/160 passed in each configuration. On macOS this ran with a local, uncommitted TDS listener patch, because the simulated server hangs on shutdown there.
  • dotnet build src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft.Data.SqlClient.UnitTests.csproj -f net462 -c Debug|Release: succeeded, 0 warnings, 0 errors.
  • CI for this PR at 8dbd424: all 39 checks passed, including sqlclient-pr 192584.
  • Full-stack CI ([Async AE 5] Load enclave keys asynchronously and route package retries #5085, build 192590, aa9fadb):
    • UnitTests: Linux net10 2668 passed, Windows net10 2692 passed, net462 2588 passed, 0 failed. Simulated-server tests ran and none of the stack's tests were skipped.
    • AE manual tests: 246 passed, 0 failed. Only the enclave tests were skipped, because CI has no enclave server.

Guidelines

Please review the contribution guidelines before submitting a pull request:

Split the metadata cache lookup into an in-memory probe and an async key
load. Async execution loads cached column encryption keys on a worker,
falls back to describe parameter encryption when keys are stale, and
keeps good metadata cached when the load is canceled.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 19:19
@github-project-automation github-project-automation Bot moved this to To triage in SqlClient Board Nov 16, 2026
@cheenamalhotra cheenamalhotra added this to the 8.1.0-preview1 milestone Nov 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical concurrency and connection-lifetime gaps during cached-key loading can let execution use changed connection state or fallback RPCs corrupt an active reader’s results.

3 open issues
What changed in this PR

This fourth layer of async Always Encrypted support extends asynchronous key loading to query metadata cache hits, while preserving synchronous provider dispatch.

Changes:

  • Splits cache lookup into an in-memory probe and asynchronous key-loading completion.
  • Dispatches async cache completion and stale-key discovery fallback to a worker.
  • Adds coverage for cache hits, stale keys, cancellation, and worker dispatch.
File Description
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Utilities/​QueryMetadataCacheTestCommands.cs Adds isolated commands and cached encryption metadata for tests.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Microsoft/​Data/​SqlClient/​SqlQueryMetadataCacheTests.cs Tests async completion, cancellation, and sync compatibility.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Microsoft/​Data/​SqlClient/​SqlCommandTests.cs Tests worker dispatch for cache hits and stale-key fallback.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlQueryMetadataCache.cs Separates metadata probing from key loading and completion.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlCommand.Encryption.cs Integrates worker-dispatched async cache completion into encryption preparation.

Review effort: Balanced

cheenamalhotra and others added 2 commits November 16, 2026 13:21
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.11009% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.81%. Comparing base (86b2cdc) to head (8dbd424).

Files with missing lines Patch % Lines
.../Microsoft/Data/SqlClient/SqlCommand.Encryption.cs 79.78% 19 Missing ⚠️
.../Microsoft/Data/SqlClient/SqlQueryMetadataCache.cs 85.33% 11 Missing ⚠️
.../src/Microsoft/Data/SqlClient/SqlCommand.Reader.cs 81.63% 9 Missing ⚠️
Additional details and impacted files
@@                             Coverage Diff                             @@
##           dev/cheena/ae-async-parameter-discovery    #5081      +/-   ##
===========================================================================
+ Coverage                                    70.15%   72.81%   +2.65%     
===========================================================================
  Files                                          314      314              
  Lines                                        70098    71546    +1448     
===========================================================================
+ Hits                                         49178    52093    +2915     
+ Misses                                       20920    19453    -1467     
Flag Coverage Δ
sqlclient-pr 72.81% <82.11%> (+2.65%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Keep _enclaveAttestationInProgress set while cached keys load and until
execution reserves the connection, so connection changes and overlapping
executions are rejected. Reject a close/reopen or connection replacement
before the stale-key fallback and before execution, and revalidate the
connection before the fallback describe.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 21:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical races allow command state changes before pending-operation publication and database changes during cached-key loading, potentially executing with altered state or writing to the wrong database.

2 open issues
3 resolved since last review

Review effort: Balanced

{
if (clearInProgress)
{
sqlCommand._enclaveAttestationInProgress = false;

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This window exists on main (2bda3c9) already, so I'm leaving the thread open.

  • In SqlCommand.Reader.cs, BeginExecuteReaderInternal runs BeginExecuteReaderInternalReadStage only after writeTask completes (lines 290-334). SetActiveConnectionAndResult, which sets PendingAsyncOperation, runs only in the read stage (line 400).
  • The setters and validation (SqlCommand.cs 605, 758, 2902, 2924) check only _enclaveAttestationInProgress or PendingAsyncOperation. On main nothing sets either during an async write, plain or after the uncached describe continuation (Reader.cs ~1837). Only enclave attestation sets the flag.

This PR doesn't widen the window. The cached path now holds the guard until RunExecuteReaderTds returns. After that it has the same write-to-read-stage tail as the uncached and plain async-write paths on main. Closing that tail for every path is a separate change.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tracked separately in #5092, since the fix covers every async reader path, not just this stack.

Capture the connection database before cached-key completion and reject
execution if ChangeDatabase ran while keys loaded, because the query
metadata cache key includes the database.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 22:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cached async execution can bypass required transaction-binding checks and skip the recovery retry for missing or expired enclave sessions.

1 open issue
2 previously missed issues

In code that hasn't changed since last review

High severity Revalidate transaction binding after cached key load resumes

src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlCommand.Encryption.cs:660

If connection.BeginTransaction() succeeds while a cached key load is pending, the command can resume inside that transaction even though command.Transaction is null, instead of throwing TransactionRequired. Revalidate transaction binding before stale-cache discovery and final execution, using the transaction checks from ValidateCommand without invoking its in-progress guard. Add a gated-provider regression test that begins a transaction before releasing the key load.

Medium severity Retry deferred enclave-package failures on metadata-cache hits

src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlCommand.Reader.cs:1871

On an async metadata-cache hit, a missing or expired enclave session can now make GenerateEnclavePackage fail in this continuation, bypassing the synchronous retry catch in RunExecuteReader. CreateLocalCompletionTask forwards that fault before reaching its retry handler, so a previously recoverable cached execution fails. Preserve metadata/session invalidation and the single retry for these deferred package failures, even if the remaining retry refactoring stays in layer 5.

1 resolved since last review

Review effort: Balanced

@cheenamalhotra
cheenamalhotra marked this pull request as ready for review November 17, 2026 06:28
@cheenamalhotra
cheenamalhotra requested a review from a team as a code owner November 17, 2026 06:28
@cheenamalhotra cheenamalhotra changed the title Async Always Encrypted (4/5): async query metadata cache completion [Async AE 4] Complete query metadata cache asynchronously Nov 17, 2026
@cheenamalhotra cheenamalhotra added the Hotfix 8.0.0 PRs that must be backported to release/8.0 label Nov 17, 2026

// Block command and connection changes until execution has been handed off.
_enclaveAttestationInProgress = true;
returnTask = Task.Run(() => CompleteCachedQueryMetadataAsync(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Every async cache hit now pays a Task.Run hop, even when KeysToLoad is empty (plaintext-only parameters, which the code notes is common) and likely when the symmetric key cache is warm, so the load completes synchronously. Consider calling the cache's CompleteCachedQueryMetadataAsync inline first: if it already completed with a hit, return returnTask = null as before, and offload only when it's pending or stale. That keeps the steady-state path free of the extra hop/allocations and keeps GenerateEnclavePackage inside RunExecuteReader's retry try for the common case.

@github-project-automation github-project-automation Bot moved this from To triage to Waiting for customer in SqlClient Board Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Hotfix 8.0.0 PRs that must be backported to release/8.0

Projects

Status: Waiting for customer

Development

Successfully merging this pull request may close these issues.

4 participants