Repository navigation
Conversation
Build one candidate-owned harness against both drivers and retain matched regression, baseline-debt and invalid-evidence outcomes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Sequential package comparisons in both scripts can reuse baseline binaries for the current pass, hiding candidate regressions, and the health build fails for the specified preview baseline because required dependency overrides are missing.
3 open issues
What changed in this PR
Adds a bounded OpenAsync health phase before BenchmarkDotNet in SqlClient’s existing performance runners, without changing production driver code or pipeline triggers. This is separate from the broader experiments in #5014.
Changes:
- Adds 20 workload cases with three paired repetitions, using the same harness against candidate and baseline drivers.
- Introduces matched regression gating, raw evidence, summaries, and optional Linux network-delay testing; default runs add no delay.
- Integrates both runners, increases timeout defaults, and adds tests and documentation.
| File | Description |
|---|---|
src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft.Data.SqlClient.UnitTests.csproj |
Links harness sources into modern .NET unit tests. |
src/Microsoft.Data.SqlClient/tests/UnitTests/Measurements/ThreadPoolProbeRecorderTests.cs |
Tests bounded probe accounting. |
src/Microsoft.Data.SqlClient/tests/UnitTests/Measurements/ThreadPoolHealthObserverTests.cs |
Tests observer failure and incomplete evidence. |
src/Microsoft.Data.SqlClient/tests/UnitTests/Measurements/OpenAsyncHealthSettingsTests.cs |
Tests workload-setting validation. |
src/Microsoft.Data.SqlClient/tests/UnitTests/Measurements/OpenAsyncHealthRecorderTests.cs |
Tests outcome retention and counter snapshots. |
src/Microsoft.Data.SqlClient/tests/PerformanceTests/OpenAsyncHealth/Program.cs |
Adds the standalone command-line entrypoint. |
src/Microsoft.Data.SqlClient/tests/PerformanceTests/OpenAsyncHealth/OpenAsyncHealth.csproj |
Supports source and package driver references. |
src/Microsoft.Data.SqlClient/tests/PerformanceTests/Microsoft.Data.SqlClient.PerformanceTests.csproj |
Excludes standalone harness sources from BenchmarkDotNet. |
src/Microsoft.Data.SqlClient/tests/PerformanceTests/Measurements/ThreadPoolProbeRecorder.cs |
Records probe timing and missing callbacks. |
src/Microsoft.Data.SqlClient/tests/PerformanceTests/Measurements/ThreadPoolHealthObserver.cs |
Samples scheduling and process resources. |
src/Microsoft.Data.SqlClient/tests/PerformanceTests/Measurements/OpenAsyncHealthRecorder.cs |
Retains operation outcomes and counters. |
src/Microsoft.Data.SqlClient/tests/PerformanceTests/Measurements/OpenAsyncHealthExperiment.cs |
Executes bounded connection workloads. |
eng/pipelines/perf/sqlclient-perf-pr-pipeline.yml |
Adds 30 minutes to the PR timeout default. |
eng/pipelines/perf/sqlclient-perf-pipeline.yml |
Adds 30 minutes to the standard timeout default. |
eng/pipelines/perf/sqlclient-perf-experiment.yml |
Adds 30 minutes to the experiment timeout default. |
eng/pipelines/perf/scripts/tests/test_openasync_health.py |
Tests evidence validation, comparisons, and orchestration. |
eng/pipelines/perf/scripts/tests/test_openasync_health_runners.py |
Tests both runner integrations and exit handling. |
eng/pipelines/perf/scripts/tests/test_openasync_health_process.py |
Adds opt-in executable lifecycle tests. |
eng/pipelines/perf/scripts/tests/test_openasync_health_netem.py |
Tests network-rule ownership and cleanup. |
eng/pipelines/perf/scripts/run-perf-tests.sh |
Runs health coverage before Linux benchmarks. |
eng/pipelines/perf/scripts/run-perf-tests.ps1 |
Runs health coverage before Windows benchmarks. |
eng/pipelines/perf/scripts/openasync_health.py |
Builds, pairs, validates, and reports health runs. |
eng/pipelines/perf/scripts/netem.py |
Adds optional SQL-only loopback delay. |
eng/pipelines/perf/README.md |
Documents automatic coverage and timeout changes. |
eng/pipelines/perf/openasync-health.md |
Explains workloads, gates, artifacts, and limitations. |
Review effort: Balanced
Observe marked BDN workloads in extra iterations, run burst regression cases automatically, and build matched candidate benchmark sources against source baselines. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The new performance gates and BDN lifecycle changes need human review of authenticated SQL pipeline runs on Linux and Windows, since SQL-free checks do not establish end-to-end measurement validity for this revision.
2 resolved since last review
Review effort: Balanced
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5068 +/- ##
==========================================
+ Coverage 68.99% 72.51% +3.52%
==========================================
Files 314 314
Lines 45857 71062 +25205
==========================================
+ Hits 31638 51533 +19895
- Misses 14219 19529 +5310
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Remove the standalone pre-benchmark sweep and network shaping. Record startup/warmup and extra-iteration queue/open/outcome signals in the perf tests, retaining missing work and resource evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
The oversubscribed benchmark may never create pool waiters, leaving the advertised waiter-wakeup regression untested, and the PR description claims health checks and gates that are not implemented.
2 open issues
1 resolved since last review
Review effort: Balanced
| Async-health measurements are part of the ordinary BenchmarkDotNet perf tests. | ||
| There is no standalone sweep, ramp prerequisite, separate case harness, or network shaping. | ||
| The existing pipeline flow and trigger conventions are unchanged. |
| var invocation = AsyncHealthDiagnoser.BeginInvocation(Workers); | ||
| var tasks = new Task[Workers]; | ||
| for (int i = 0; i < tasks.Length; i++) | ||
| { | ||
| var operation = invocation?.Offer(i); | ||
| tasks[i] = Task.Run(async () => | ||
| { | ||
| operation?.Start(); | ||
| try | ||
| { | ||
| using (var connection = new SqlConnection(_connectionString)) | ||
| { | ||
| Task open = connection.OpenAsync(); | ||
| operation?.OpenReturned(open.IsCompleted); | ||
| await open.ConfigureAwait(false); | ||
| operation?.Opened(); | ||
| } | ||
| operation?.Complete(); | ||
| } | ||
| catch (Exception error) | ||
| { | ||
| operation?.Fail(error); |
Skip preview-only automatic JIT tier-promotion workload, retaining configured warmups and measurement counts. Reserve collection grace inside the pipeline task budget and retain partial per-unit logs and timings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>


Description
Capture async-health regression signals inside the existing BenchmarkDotNet performance-test flow. The previously added standalone pre-BDN sweep, CLI experiment harness, network-shaping helpers, custom sweep gates and their tests have been removed from this PR. Pipeline timeout defaults are unchanged; there is no separate ramp or sweep before the perf tests.
FirstBenchmarkInProcessidentifies the first case rather than claiming every empty-pool case is process-cold.benchmark-async-health.md. Existing triggers and BDN timing-regression policies remain unchanged.OpenAsyncBurstcases: 64 queued workers against empty, warm, oversubscribed 16-slot and non-pooled connections, each performing open/close or open/query/close. Worker minimum 16 is explicitly set/read back and restored afterward.Benchmark operations record offer-to-worker-start delay, open duration, offer-to-open latency, useful throughput, inline/pending opens, failures/cancellations, queued/in-flight work and missing first-worker outcomes. Every worker's first record survives the normal retention cap; dropped-record counts remain explicit. Probe/resource data includes dispatch and producer delay, missing probes, ThreadPool counts and all process threads, plus allocation/GC counters. Exception messages and connection strings are not retained.
Short/censored windows are insufficient, not healthy. Failed or incomplete instrumented work fails explicitly. The 100ms dispatch finding is diagnostic, not a new unconditional pipeline gate. Diagnostic resource/allocation data includes instrumentation bookkeeping; use identical harnesses for comparisons. BDN's batch-normalized mean is amortized cost, not individual request latency; the operation records supply those latency distributions.
Candidate-owned benchmark/diagnoser sources are compiled against both driver versions. Source baselines select their driver and matching TestCommon via
MdsSourceRoot; package baselines retainMdsPackageVersion. Sequential mode rebuilds the candidate after a baseline build. Pool/SNI settings remain process-wide selections; no experimental driver switch or production driver/API change is included.Test-only BenchmarkDotNet packages use
0.16.0-preview.1for the required lifecycle hooks; per-case timeouts remain enforced. The shared job explicitly usesJitTieringMode.Skipto avoid the preview's additional automatic tier-promotion workload. This does not disable runtime tiered compilation or reduce configured warmups, measurement counts, or diagnostic iterations. Existing run strategies remain unchanged; marking additional long-running benchmarks asMonitoringis deferred to a separate pass.The VM runner reserves the final ten minutes of the existing task timeout for completion polling and artifact collection. Build/setup elapsed time reduces the remaining benchmark budget. Budget expiry terminates owned benchmark processes and fails explicitly while retaining raw logs/reports under
results/runsand per-unit elapsed/exit-status JSON. This protects benchmark-timeout evidence, not an indefinitely hung SDK installation or an externally killed agent/VM.See
eng/pipelines/perf/async-health.md. Automated pipelines use zero added RTT. The 100ms added-RTT reproduction below was local Linux validation only; the exact latency-injected scenario does not automatically run. No TUI, study data, standalone sweep, or artificial RTT machinery is added to the pipeline.Issues
Related to #5014. The experiments remain preserved there; this PR does not replace or close it.
Testing
Actual local SQL reproduction
The BDN benchmarks reproduced the reported Linux scheduler stall in all three fresh-process repetitions. Tested benchmark revision
f2ad8a688, local Ubuntu 24.04 WSL, .NET 10.0.12, managed SNI, 16 logical CPUs, 64 workers, worker minimum/maximum 16/32767, pool 100, SELECT 1, SQL authentication, Encrypt=true/TrustServerCertificate=true, 15-second connection timeout, and 100ms added RTT (50ms each direction). The subsequent JIT-preparation fix does not alter this unit's existingMonitoringstrategy, which already bypasses that preparation stage.Four conditions x three fresh processes produced 12 processes and 24 diagnostic windows. Each window completed 64 operations with zero failures, cancellations, queued/in-flight work or missing first-worker outcomes.
The V2 startup result matches the earlier sweep's 2690–2791ms symptom. Its first-worker queue p99 was 2754.5–2791.4ms; offer-to-open p99 was 3451.7–3597.9ms, versus 1137.0–1229.5ms within-worker open p99. Thus successful-open timing alone misses the scheduling component. V1 completed 9.17–9.22 operations/s versus V2's 17.29–18.01 in these first batches, retaining the scheduler/throughput tradeoff. These batch-throughput values are not claimed numerically equivalent to an 8-second continuous sweep.
Cohort/censoring qualifications: cold V2 Startup windows lasted 3.672–3.821s, with 100–107 executed probes and 46–48 capacity-dropped probes explicitly retained in the counters (no producer misses or outstanding callbacks). Their observed percentile is a failure/lower-bound finding, not a qualified healthy estimate. Warm Startup includes about 39s of synchronous fixture prewarming; its large healthy probe cohort is not used as a workload-only warm/cold comparison. The table instead shows the warm ExtraIteration cohort, which has only 15–16 probes and correctly remains
Insufficient, not a statistical health pass. Its fully observed per-operation offer-to-open p99 was 2.908–3.350ms. Operation throughput excludes fixture time.Process-thread costs also remain visible: non-pooled Startup peaked at 45–46 process threads while ThreadPool count was 16; V2 Startup peaked at 31 process threads/18 ThreadPool threads. Raw phase, duration, counters and samples are preserved. Local shaping recorded traffic with zero packet drops, cleanup reported
Cleaned, and a livetccheck confirmed loopback returned tonoqueue. The credential-bearing config existed only in a memfd and the validation process exited; no password was saved.Normal execution ordering was tested too: three fresh Windows integrated-auth, managed-SNI, zero-added-RTT runs executed the entire unfiltered
OpenAsyncBurstunit (all 8 cases each). The normal first case,OpenClose(State=Empty), captured 281–319ms startup dispatch p99. Capture therefore does not depend on the special reproduction filter. Separate Windows query-case controls reproduced V2-empty 363–408ms versus V1-empty 24–27ms across three repetitions.The local validation helper only launches the real BDN executable and is not in this PR/pipeline. Linux used one warmup, one normal timing iteration and the normal diagnostic extra iteration per case; Windows full-unit checks used the default three timing iterations. The actual BDN commands were:
Preparation overhead and timeout fix
Run 192544 reached the 180-minute outer task timeout during its first pass, before
OpenAsyncBurstand before result collection. VM teardown succeeded; detailed result artifacts were unavailable.The preview's automatic tier-promotion stage repeatedly invokes full benchmark workloads and fixtures beyond configured warmup. A real local SQL
ExecuteScalarText/MARS=false A/B confirmed the effect:The production-config regression test executes BDN and checks bounded preparation plus unchanged explicit warmup/actual iteration counts. Timeout tests cover shared deadlines, real owned-process termination, partial retention, invalid budgets and both shell entrypoints.
Automated checks
Broader script validation:
python -B -m unittest discover -s eng\pipelines\perf\scripts\tests -qran 49 tests on Windows: 46 passed and the same 3 pre-existing Git Bashtest_tool_restoretemp-path assertions failed. Previously reproduced on untouched main; not modified or suppressed. In WSL,python3 -B -m unittest discover -s eng/pipelines/perf/scripts/tests -qagainst a complete LF Git export passed 27 tests and skipped 22 unavailable platform/tool cases. Windows checks above exercised the PowerShell counterparts. Initial Windows-export CRLF failures were corrected in the validation export, not by modifying committed code.The f2ad8a6 source was exported into an isolated local WSL directory,
dotnet tool restorewas run there, and the BDN host built with-c Release -f net10.0 -p:DisableSourceLink=true: passed with 0 warnings/errors.Successful current-head pipeline and final timings
Run 192609 passed on exact head
d64140e0c1e4d5d4ec119a2f21758c581b4ea68c. Artifact publication and VM teardown succeeded. All 40 PR checks are green. The separate standard CI build 192608 had one failure in an unchanged MultiSubnetFailover simulated-server test; that Linux job passed on one retry without source changes or suppression. Both Windows test stages passed on their first attempts.Per-process timing JSON verifies 74 successful processes: all 19 units for baseline and candidate in the first pass, followed by two confirmation repetitions for 9 units on both sides. First-pass process time was 60.84 minutes; confirmation added 24.88 minutes. Existing strategies, warmups and measurement counts were not reduced. Both baseline and candidate logs explicitly show
JitTieringMode=Skip.OpenAsyncBursttook 0.89 minutes for its first pair and 2.69 minutes across all three repetitions on both sides. The large data-reader matrices still dominate; reclassifying long-running cases asMonitoringremains separate work. Earlier successful pre-preview executions took approximately 105–121 minutes for the run-tests task; their bulk-flushed logs do not support trustworthy per-unit timestamp comparisons.Artifact verification: the published ZIP passed CRC verification. It contains 204 unique schema-2 diagnostic windows (102 Startup, 102 ExtraIteration), with identical duplicate copies, complete operation evidence, zero failed/cancelled/open-failure outcomes, zero queued/in-flight work, and zero missing first-worker outcomes. Raw evidence includes 3,082 probe samples, 3,082 resource samples and 67,104 operation records. All eight burst cases ran three times per side. The two sides have distinct driver SHA-256 values and matching harness/runtime/SNI/pool settings.
Interpretation limits: this successful run used the pipeline's default V1 pool, managed SNI, zero added RTT, .NET 10.0.15 and 16 CPUs on both sides. It validates automatic execution and retention, not the separate local V2/100ms starvation reproduction. Of its diagnostic windows, 202 correctly report
Insufficientand 2 reportPass; completed operations do not make short probe cohorts statistically healthy. The retained BDN comparison covers 1,237 benchmarks and reports 7 confirmed timing regressions, 52 unconfirmed flags and 58 improvements versus 7.1.2. Pipeline success is not a claim that those report findings are absent; no comparison policy or finding was suppressed to obtain green execution.Kusto ingestion was deliberately disabled for this validation run. The earlier feed issue was repaired by saving missing Abstractions/Logging 7.1.2 dependencies, then verifying anonymous downloads and clean-cache restores. The obsolete sweep-bearing run 192494 was cancelled and its VM teardown succeeded.
Guidelines
Please review the contribution guidelines before submitting a pull request: