Skip to content

[Async AE 5] Load enclave keys asynchronously and route package retries - #5085

Open
cheenamalhotra wants to merge 4 commits into
dev/cheena/ae-async-metadata-cachefrom
dev/cheena/ae-async-enclave-key-loading
Open

cheenamalhotra wants to merge 4 commits into
dev/cheena/ae-async-metadata-cachefrom
dev/cheena/ae-async-enclave-key-loading

Conversation

@cheenamalhotra

@cheenamalhotra cheenamalhotra commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Description

Layer 5 of 5 split from #4585. Stacked on #5081.

  • EnclaveDelegate: adds GenerateEnclavePackageAsync and GetDecryptedKeysToBeSentToEnclaveAsync. Keys load one at a time with async key store calls. Sync and async paths share BuildEnclavePackage and CreateColumnEncryptionKeyInfo.
  • SqlCommand.Reader: after describe, ContinueRunExecuteReaderTdsAsync (on a Task.Run worker) generates the enclave package asynchronously, then runs the TDS execution. This replaces the sync GenerateEnclavePackage call. The connection is revalidated after each await. Cancellation propagates unwrapped.
  • SqlCommand.CreateLocalCompletionTask: a retryable enclave package fault before the read stage now goes through the same bounded retry as read-stage failures. This is the first production reader of _asyncExecutionCancellationToken.

Fixes from review feedback:

  • Guard handoff: the layer 4 in-progress guard (_enclaveAttestationInProgress) is held from the post-describe continuation through package key loading, package generation and the RunExecuteReaderTds handoff. It is cleared on failure or cancellation. Changing Connection or starting another execution while package keys load now throws.
  • Database check: ValidateConnectionLifetime gets the database captured before dispatch, after describe and after the package-key await. A database change during key loading throws.
  • XML retry task: BeginExecuteXmlReader now attaches its callback to and returns the global completion task, like Reader and NonQuery. Callers see the retry's outcome, not the faulted first attempt.

Deviations from #4585:

  • A faulted write already released its async-count reservation, so the retry does not take it over. This avoids a double decrement.
  • The connection is also revalidated after the describe await, before package generation.
  • EnclaveDelegate.Crypto.cs comments mention the session-lock wait on a cache miss.

Issues

Part of #4585.

Testing

  • New EnclaveDelegateTests (6 tests): async vs sync provider API, all keys resolved, sequential loading, cancellation, provider failure.
  • SqlCommandTests: new CreateLocalCompletionTask_* tests (retryable write fault retries, retry fails, retry canceled, non-retryable fault, canceled write).
  • SimulatedServerTests/SqlCommandEnclaveAsyncTests: Execute_PendingPackageKey_* (revalidation, connection setter, second execution, database change), Prepare_Completed_HandsOffInProgressFlag, Retry_PackageFault_DoesNotTransferReservation, BeginExecuteXmlReader_RetryablePackageFault_CompletesWithRetry.
  • ManualTests/AlwaysEncrypted/ApiShould.cs: re-enables debug stale-enclave retry coverage.

Local results at aa9fadb:

  • dotnet test src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft.Data.SqlClient.UnitTests.csproj -f net10.0 -c Debug|Release --filter "FullyQualifiedName~SqlCommandEnclaveAsyncTests|FullyQualifiedName~SqlCommandQueryMetadataCacheAsyncTests|FullyQualifiedName~EnclaveDelegateTests|FullyQualifiedName~SqlCommandTests|FullyQualifiedName~SqlQueryMetadataCacheTests|FullyQualifiedName~SqlSecurityUtilityTests|FullyQualifiedName~SymmetricKeyCacheTests|FullyQualifiedName~AsyncKeyStoreProviderTests": passed, 178/178 in each configuration. Simulated-server tests were run with a local-only macOS listener workaround (not committed).
  • dotnet build of UnitTests and ManualTests with -f net462 -c Debug|Release: 0 warnings, 0 errors.
  • dotnet build build.proj -t:BuildSqlClient -p:Configuration=Debug|Release: 0 warnings, 0 errors.

CI (build 192590, aa9fadb):

  • UnitTests: Linux net10 2668 passed / 0 failed; Windows net10 2692 / 0; net462 2588 / 0. Simulated-server tests ran; none of the stack's tests were skipped.
  • AE manual tests: 246 passed / 0 failed. Only enclave tests were skipped (no enclave server in CI).

Pending: enclave end-to-end testing against a real enclave server.

Guidelines

Please review the contribution guidelines before submitting a pull request:

Generate enclave packages with the async key store APIs after the
describe round trip, revalidate the connection after each await, and
route retryable package faults into the bounded enclave retry without
transferring an already released reservation.

Final layer of the #4585 split.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A critical concurrency gap during the key-store await allows connection reassignment or overlapping execution to redirect pending work or overwrite encryption state.

1 open issue
What changed in this PR

Completes the fifth layer of async Always Encrypted integration in SqlClient, building on #5081’s metadata-cache work to load enclave keys asynchronously and route package failures through the existing retry mechanism.

Changes:

  • Adds sequential async enclave-key loading while sharing validation and package construction with synchronous execution.
  • Awaits package generation before command execution, revalidates the connection, and handles retryable package faults without transferring an already-released reservation.
  • Adds provider, retry, and connection-revalidation tests; re-enables debug stale-enclave retry coverage.
File Description
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​SimulatedServerTests/​SqlCommandEnclaveAsyncTests.cs Tests connection revalidation during key loading and retry reservation handling.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Microsoft/​Data/​SqlClient/​SqlCommandTests.cs Covers retry success, failure, cancellation, and non-retryable write faults.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Microsoft/​Data/​SqlClient/​EnclaveDelegateTests.cs Tests provider dispatch, sequential key loading, cancellation, and errors.
src/​Microsoft.Data.SqlClient/​tests/​ManualTests/​AlwaysEncrypted/​ApiShould.cs Re-enables debug package-failure retry scenarios.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlCommand.Reader.cs Adds async package generation and an await-based post-describe execution path.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlCommand.cs Routes retryable package faults through bounded retry with adjusted reservation ownership.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​EnclaveDelegate.cs Adds async key loading and shared key-info validation.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​EnclaveDelegate.Crypto.cs Adds async package generation and shared package assembly; clarifies session-lock behavior.

Review effort: Balanced

Merge dev/cheena/ae-async-metadata-cache (9fee7e7) and reuse its
in-progress guard on the async enclave path. Successful async preparation
now hands the flag to the reader continuation, which holds it through
package key loading and the RunExecuteReaderTds reservation, revalidates
the connection lifetime after each await, and clears it on success,
failure or cancellation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 22:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

ExecuteXmlReaderAsync still exposes the original task during retries, so callers can receive an exception and release command state while a retry is using it.

1 open issue
1 resolved since last review

Review effort: Balanced

Pass the captured database to ValidateConnectionLifetime after describe and
after the package key await. Share UseDatabaseQueryEngine and add a
ChangeDatabase test while the package key is pending.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 22:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Async enclave retry changes affect connection-reservation ownership and command reuse, while unexecuted simulated-server and real enclave stale-session retry tests leave key behavior needing CI/runtime confirmation.

1 open issue

Review effort: Balanced

BeginExecuteXmlReader now attaches its callback to and returns the
global completion task, like Reader and NonQuery, so callers observe
the bounded enclave retry instead of the faulted first attempt.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 22:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Async enclave changes span command guards, reservation ownership, and XML retry completion, while simulated-server and live SQL Server/enclave validation remain unverified and require human confirmation.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.62376% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.68%. Comparing base (8dbd424) to head (aa9fadb).

Files with missing lines Patch % Lines
.../src/Microsoft/Data/SqlClient/SqlCommand.Reader.cs 86.77% 16 Missing ⚠️
...nt/src/Microsoft/Data/SqlClient/EnclaveDelegate.cs 80.00% 6 Missing ⚠️
...Microsoft/Data/SqlClient/EnclaveDelegate.Crypto.cs 90.32% 3 Missing ⚠️
Additional details and impacted files
@@                          Coverage Diff                           @@
##           dev/cheena/ae-async-metadata-cache    #5085      +/-   ##
======================================================================
- Coverage                               72.81%   70.68%   -2.13%     
======================================================================
  Files                                     314      314              
  Lines                                   71546    70348    -1198     
======================================================================
- Hits                                    52093    49724    -2369     
- Misses                                  19453    20624    +1171     
Flag Coverage Δ
sqlclient-pr 70.68% <87.62%> (-2.13%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cheenamalhotra cheenamalhotra changed the title Async AE (5/5): async enclave key loading and package retry routing [Async AE 5] Load enclave keys asynchronously and route package retries Nov 17, 2026
@cheenamalhotra cheenamalhotra added the Hotfix 8.0.0 PRs that must be backported to release/8.0 label Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Hotfix 8.0.0 PRs that must be backported to release/8.0

Projects

Status: To triage

Development

Successfully merging this pull request may close these issues.

4 participants