Repository navigation
rustls: dual-provider tree (ring + aws-lc-rs) has no process-level CryptoProvider default — panics on the VSS TLS path #991
Description
Activity
@hash-money dug into this on main. The root cause isn't quite where the report puts it, and there's a second bug.
aws-lc-rs comes from
chain-electrum, not the VSS/HTTP path.cargo tree -e no-devwithchain-electrumoff:rustls v0.23.43 feats=ring,std,tls12, and noaws-lc-rsin the tree at all. With it on:feats=aws-lc-rs,aws_lc_rs,default,log,logging,prefer-post-quantum,ring,std,tls12. bitreq (the VSS/HTTP path) asks only for["ring","std","tls12"].electrum-client 0.25 has an optional dep
rustlsand a same-named featurerustls = ["webpki-roots","dep:rustls","rustls/default"]. The explicit feature shadows the implicit one, sorustls-ring's"rustls/ring"also enables therustlsfeature →rustls/default→ aws-lc-rs. The resolved tree showselectrum-client feats=…,rustls,rustls-ring,use-rustls-ring,…even though bdk_electrum and lightning-transaction-sync both pindefault-features = false(notedefaultis absent from that list). So 4b45d7c ("Switch to userustls-ringeverywhere") never actually produced the ring-only tree it advertises — which matters for the step 2 @tnull sketched in #600.The safeguard runs after the first TLS request on the VSS path.
optionally_install_rustls_cryptoprovider()is called inbuild_with_store_internal(builder.rs:1494), but all fourbuild_with_vss_store*methods construct the store first (builder.rs:803, 841, 869, 895).VssStore::newdoes a blockingdetermine_and_write_schema_versionon a scoped thread (io/vss_store.rs:128-143 → :810), and vss-client-ng → bitreq builds itsClientConfigfrom the process default. No corepc#661 needed — this is reachable on the current pin against anhttps://VSS URL. The scoped-threadjoin()catches the unwind and turns it intoBuildError::KVStoreSetupFailed, so it surfaces as a confusing setup failure rather than the rustls message; underpanic = "abort"profiles it aborts.Happy to take: hoist the install ahead of VSS store construction, declare
rustls'sringfeature explicitly in our own manifest (builder.rs:2508 callsrustls::crypto::ring::default_provider(), but Cargo.toml:108 declaresrustlswithdefault-features = falseand no features, so that only compiles because something else in the tree turnsringon), and carry the manifest fix upstream to electrum-client — renaming the dep torustls_dep = { package = "rustls" }removes the shadowing and yieldsrustls feats=ring,std,tls12.Do you want provider choice behind
tls-ring/tls-aws-lc-rsfeatures, along the lines of what @TheBlueMatt described on #600, or keep a single pinned provider?@big14way thanks — we re-derived all of this independently and it holds. Corrections to our own report first, then the one thing you may not have seen, then your question.
Confirmed (cargo tree on main
6480bff):# default features rustls v0.23.43 feats=aws-lc-rs,aws_lc_rs,default,log,logging,prefer-post-quantum,ring,std,tls12 electrum-client v0.25.0 feats=byteorder,libc,proxy,rustls,rustls-ring,use-rustls-ring,webpki-roots,winapi # --no-default-features --features chain-esplora,chain-bitcoind,storage-sqlite,storage-filesystem,storage-vss,unified-payments rustls v0.23.43 feats=ring,std,tls12 # no aws-lc-rs in the treeMinimal repro of the shadowing, one dependency only:
electrum-client = { version = "=0.25.0", default-features = false, features = ["use-rustls-ring"] }resolves rustls withaws-lc-rs,…,default,…,ring; the same crate withrustls-ring = [..., "rustls?/ring", "rustls?/logging", "rustls?/std", "rustls?/tls12"]resolvesrustls feats=log,logging,ring,std,tls12.Ordering: read as you describe —
build_with_vss_store(builder.rs:797–809) runsVssStoreBuilder::build_with_sigs_auth→VssStore::new→ the scoped-threaddetermine_and_write_schema_version(io/vss_store.rs:128–143) beforebuild_with_store_internalreachesoptionally_install_rustls_cryptoprovider()(:1494); thejoin()error becomesKVStoreSetupFailed(:805). One nuance: the rustls panic text still reaches stderr through the default panic hook — it's only the returnedBuildErrorthat loses it. We have not executed this path on main; see the offer below.Two corrections to the opening report:
- The
aws-lc-rsattribution was wrong. It is not "rustls' default on the VSS/HTTP path" — in our tree, exactly as in yours, the only source is electrum-client'srustlsfeature leakingrustls/default. Substituting therustls?/…manifest (and no default features on our own rustls dep) leaves our Android target atrustls feats=log,logging,ring,std,tls12—aws-lc-rsgone entirely. - "The released stack doesn't hit it" was a lockfile artifact, not a property of the code. Our
Cargo.lockholdsbitreq 0.3.4, which is on rustls 0.21 (noCryptoProviderconcept).bitreq ≥ 0.3.5requires rustls^0.23.38, and 0.3.7 — what a fresh resolve of main'sbitreq = "0.3"gets — builds its config withClientConfig::builder()(connection/rustls_stream.rs:36–50). So corepc#661 was incidental: it was simply the first rustls-0.23 bitreq we ran. No patch needed to reach it on main; your reading is right.
Already in flight upstream: the electrum-client half exists as bitcoindevkit/rust-electrum-client#220 (open since 2026-07-02, same
rustls?/ringdiff, tACK'd, awaiting maintainer review), and electrum-client#183 already prefersringinternally when both features are on — so no second electrum-client PR is needed, and the ldk-node-side exposure reduces to the VSS ordering plus the explicitringfeature in ldk-node's own manifest.On
tls-ring/tls-aws-lc-rsfeatures vs. a single pinned provider: that's @tnull's and @TheBlueMatt's call (they already leaned toward downstream-selectable features on #600), and we don't need the toggle either way. What we need is narrower: (a) the install — or an explicit-providerClientConfig— runs before any TLS client is constructed,VssStore::newincluded; (b) a provider already installed by the host process keeps winning, as the currentis_none()check does. Two data points, not asks: we consume ldk-node through orange-sdk, so an ldk-node feature only reaches us if orange-sdk re-exports it — the app-levelinstall_default()is what we rely on regardless; and with #220 landed,ringis what everything else in our tree (spark-sdk, cdk, reqwest 0.12, and bitreq ≥0.3.5 itself) already selects, so a single pinnedringwould be the zero-config outcome for us and would dropaws-lc-sysfrom our mobile build.Offer stands: our harness (Galaxy A12 +
wallet-cliagainst our staging VSS overhttps://) is scripted; point us at a branch with the hoist and we'll post before/after on the current tree. No urgency from our side — we install at app init, so we route around the ordering today.- The
Context: @tnull asked us to split this out of lightningdevkit/orange-sdk#90 (comment: "Can you expand on that and maybe open a separate issue for it? LDK Node should do that for you, so we should investigate what's going on exactly."). Filing here per that request — and flagging honestly below where the trigger actually lives, in case you conclude it belongs in
bitreq/corepc instead.Symptom
When a binary embedding ldk-node compiles both rustls crypto providers —
aws-lc-rs(rustls' default) andring— rustls has no unambiguous process-level default, so any code that builds aClientConfigvia the process-default-dependentClientConfig::builder()panics:Reproduced standalone to isolate it to rustls itself (v0.23.37, our pin):
Where the two providers come from (our tree)
ring— pulled in byelectrum-client(our chain source, under ldk-node).aws-lc-rs— rustls' default, on the VSS / HTTP path.ldk-node is where these two meet in a single process, which is why the collision surfaces at wallet init rather than in either dependency alone.
Where it triggers today
We hit this while integrating rust-bitcoin/corepc#661 — the
bitreqconnection-persistence fix you and @TheBlueMatt asked us to benchmark on #90. The patchedbitreqbuilds its rustlsClientConfigfrom the process-level default, so with both providers present it panics at the first VSS request.On the released stack (without corepc#661) we do not hit it — nothing on that path calls the process-default builder — so today it's latent, and gets exposed by the very patch that fixes the VSS latency. That's the part worth investigating: the perf fix and this panic arrive together.
Our workaround (the shim referenced on #90)
At wallet init, before any TLS client is constructed:
It works, but
install_default()makes a process-global choice from library/app init — the decision rustls says the top-level binary should own. Pushing that onto every downstream app embedding ldk-node is the awkward part.Fix direction (deferring to you on the right layer)
Rather than depend on the process-level default, the code that builds these
ClientConfigs (ldk-node's own TLS clients, orbitreq's) could construct them with an explicit provider viaClientConfig::builder_with_provider(...), so a multi-provider tree neither panics nor forces downstream to install a process-global default.If you conclude the right place is
bitreq, we're glad to carry this straight to corepc#661 — we just surfaced it here per your request, since ldk-node is where the two providers collide.Repro / standing offer
The standalone snippet above pins the panic to rustls. Our full-stack capture (A12 device + desktop wallet-cli against staging VSS) is scripted — point us at a branch and we'll confirm the panic is gone, before/after, same as we've done for #93 / #94 / corepc#661.