[dghgit]

Summary

RSA digital signatures, plus the RSASVE KEM, across a limited range of modulus sizes up to the maximum originally requested, 8192 bits: RSASSA-PKCS1-v1_5 and RSASSA-PSS (RFC 8017 §8), the latter with its parameters as a type, the conventional MGF1 pairing, PSS-SHAKE128/256 (RFC 8702 §3.2.1), another MGF1 hash or salt length (§8.1, Appendix A.2.3) and the zero salt (§9.1 note 4, deterministic), over RSA-1024/1536 (verify-only) through RSA-2048/3072/4096/8192 (sign+verify), on bouncycastle_core's key roles. The size is a marker type (RSA2048), a scheme is one generic marker over the size, the hash and its parameters (RSASSA_PSS<RSA2048, SHA256>, RSASSA_PSS<RSA2048, SHAKE128>, RSASSA_PSS<RSA2048, SHA512, Mgf1<SHA256, 32>>), and a key is a key for one of them: RsaPrivateKey<A> implements SigningKey or DeterministicSigningKey by what the scheme draws and RsaPublicKey<A> VerifyingKey, handing out SignatureSigner / SignatureVerifier workers, with the material behind them as RsaPrivateComponents<K> / RsaPublicComponents<K>. RSASVE (SP 800-56B Rev. 2 §7.2.1) is the KEM roles EncapsulatingKey / DecapsulatingKey over () on KEM-only key types at 2048/3072/4096/8192, and RSASVE_HKDF<K, H> the same roles over a KdfParams<HkdfDescription> for a derived secret. Key pair generation is FIPS 186-5 Appendix A.1.3, from the size marker for one scheme (RSA2048::generate_pss_signature_key_pair::<SHA256>(&mut rng)). A new crate, bouncycastle-rsa, with its CLI subcommands, criterion benches and stack-usage harness.

Description

• Reuses bouncycastle_ec's constant-time fixed-width limb arithmetic (bouncycastle_ec::nat, bouncycastle_ec::montgomery) rather than a dedicated BigInteger type, for the same constant-time-on-secret-exponent reasoning that motivates that crate's own custom-curve arithmetic: a modular exponentiation over a secret exponent has to run in the same time regardless of the exponent's value, which needs fixed-width operands. modexp is Montgomery exponentiation over a runtime modulus (f7f2f12); keys holds the CRT key types and rsa_core the CRT-based RSASP1/RSAVP1 (182eb3c).
• 2048-8192 bit moduli support both signing and verification; 1024- and 1536-bit moduli are verification-only, enforced at compile time: the size markers implement size::RsaSize, and only those from 2048 up size::RsaSigningSize, on which the private key's constructors and encoder, the signing roles, RSASP1, key generation and the KEM private key are all written, so no private key value and no private-key arithmetic exists at the two legacy sizes (a compile_fail doctest in rsa_1024.rs pins it).
• API shape: the EMSA/RSASSA layers take the message hash rather than the message (2c0166e), so the streaming workers share one encoding path; the crate is written once per scheme over the size marker (eb4aefd), the key types carry their algorithm (010cf1a) and RSASSA-PSS its parameters (5c31875). RSASSA_PKCS1_v1_5<K, H> and RSASSA_PSS<K, H, P = Conventional> (spec spelling per QUALITY_AND_STYLE) are the markers; P is rsassa_pss::PssParams<H>, the mask and the salt length: Conventional reads the hash's conventional pairing off PssHash (MGF1 over the hash with a salt one output long for SHA-2 and SHA-3, the XOF itself as the mask for SHAKE), Mgf1<M, SALT_LEN> spells another, NoSalt<M> is the zero salt and puts the key on the deterministic role. A key is RsaPrivateKey<A> / RsaPublicKey<A> over the components, so sk.sign(..) and pk.verify(..) resolve from the key, a PSS key cannot sign PKCS#1 v1.5 (SP 800-57 Part 1 Rev. 5's one scheme per key pair), and the components become a key for a scheme through key::<A>(); the roles are implemented once in rsassa over three algorithm traits, with three workers (RsaSigner with set_salt, RsaDeterministicSigner, RsaVerifier) and the *_from_hash entry points. Per-size alias modules (rsa_2048::RSASSA_PSS_SHA256, SIG_LEN) serve code at one size. The schemes were first wired as Signer / SignatureVerifier types (a996164), the per-size free functions dropped once the traits covered them (c84e7a6), and the whole ported onto the key role traits when those landed on the release (c6ea90d).
• Key pair generation (db9c026): FIPS 186-5 Appendix A.1.3 (probable primes: trial division against the primes below 2^10, then Appendix B.3.1 Miller-Rabin with Table B.1's round counts; e = 65537; |p - q| and d > 2^(nlen/2) criteria from A.1.1), structured after BC Java's RSAKeyPairGenerator, on the crate's own limb arithmetic, no bignum dependency. Exposed on the signing sizes as size::SignatureKeyPair (RSA2048::generate_pkcs1_v1_5_signature_key_pair::<SHA256>(&mut rng), generate_pss_signature_key_pair::<H>, generate_pss_signature_key_pair_with::<H, P>, a pair for one algorithm, the hash spelled and never defaulted) and size::KemKeyPair, with RsaSigningSize::keygen giving the components; the RNG must be at least the modulus's strength. The private-key paths scrub what they derive from the primes (57c5c09).
• RSASVE (b539612, e075ea6, c3129a3): the scheme BC FIPS Java 2.0 exposes as FipsRSA.KTS_SVE (JCA "RSA-KAS-KEM") and OpenSSL 3 as its "RSASVE" KEM operation. b539612 is a behaviour-preserving refactor that splits the exponentiation out of rsavp1/rsasp1 into crate-private public_exp/crt_exp, so RSASVE can reuse it under SP 800-56B's 1 < x < n - 1 range checks instead of RFC 8017's. The KEM-only key types RsaKEMPublicKey / RsaKEMPrivateKey keep key establishment and signing apart (§6.1 item 5) and add the §6.2.1 checks (e >= 65537, n of exactly nlen bits). After the squash of PR KEM encapsulation and decapsulation move onto the key objects (EncapsulatingKey / DecapsulatingKey, ML-KEM ported, KeyType::SharedSecret) #192 they implement the KEM roles over (), naming RSASVE as the algorithm, with RsasveEncapsulator::encaps_out as RSASVE.GENERATE and RsasveDecapsulator::decaps_out as RSASVE.RECOVER into the caller's buffers; the ciphertext is a [u8; CT_LEN], so a wrong length is a type error. The shared secret is the raw Z at the modulus's full length, typed CryptographicRandom at the modulus's strength, for the caller's KDF; RSASVE_HKDF<K, H> (rsasve_kdf, in eb4aefd) gives the same key types the KEM roles over KdfParams<HkdfDescription<H>>, SP 800-56B Rev. 2 Section 5.5's derivation of the secret from Z, the parameters being the library's HSM-readable description.
• CLI subcommands rsa-<size> and rsasve-<size>, which stream stdin through the workers in this CLI's usual 1 KB chunks; criterion benches (including keygen at 2048/3072); a stack-usage harness.

Scope and Risk

New crate only (bouncycastle-rsa) plus its CLI wiring and mem_usage_benches harness, based on release/0.1.4alpha. Two changes outside those: utils::const_str::ConstStr::decimal, the digits of a number for an algorithm name (-MGF1-SHA256-32), with its tests; and core-test-framework's KEM suite (dce5490): its fixed seed was a 256-byte counting pattern, so two consecutive draws of 256, 512 or 1024 bytes, RSASVE's secret at 2048, 4096 and 8192 bits, were identical and the "two encapsulations differ" check failed; it is a 4096-byte hashed pattern now. The branch has taken each release tip by merge, most recently f213c9e (the ECDSA-over-Curve and KDF-description squashes).

Validation

• cargo test --workspace passes at the head, all targets build, rustfmt is clean, and the rsa-* / rsasve-* CLI tests pass. For the crate: Wycheproof PKCS1-v1.5/PSS/PSS-SHAKE vectors per size and hash, its rsa_pss_2048_sha512_mgf1sha256_32_params and rsa_pss_2048_sha256_mgf1_0 files for the spelled parameters (178 and 103 vectors), BC Java cross-check vectors at 8192, core-test-framework's signature conformance suite for every pairing at every size, and its KEM conformance and key suites for RSASVE at every size.
• RSASVE (tests/rsasve_tests.rs, vectors in bc-test-data/crypto/rsa): NIST ACVP-Server KAS-IFC-SSC-Sp800-56Br2, the KAS1 group (2048, CRT) and the KAS2 group (3072, CRT, both directions), decaps recovering Z from C and an encapsulation fed the vector's Z reproducing its C, with the failChangedZ cases not matching; ACVP-Server RSA-DecryptionPrimitive-Sp800-56Br2, CRT groups at 2048/3072/4096 including the out-of-range ciphertexts; OpenSSL 3.0's "RSASVE" KEM operation at every size, 8192 included.
• ./dev_scripts/quality_stats.sh ./crypto/rsa at the head: 4383 core lines, 1290 docstring lines, test ratio 0.95, 1 unwrap and 36 Err() sites in core code.
• cargo mutants, as reported in the comments below (the hashes there belong to the pre-rebase history): the full-crate run of the pre-trait-wiring state, 646 mutants, 446 caught, 6 missed (documented OR/XOR and boundary equivalents), 3 timeouts, 191 unviable; the scoped run over the 14 trait-wiring and keygen files, 751 mutants, 476 caught, 0 missed, 180 unviable, 95 timeouts, of which the --iterate --timeout 7200 re-run left 14 genuine infinite loops, all itemised; RSASVE's rsasve.rs and the KEM code of the four size modules, 168 mutants, 72 caught, 0 missed, 95 unviable, one real hang (attempts += 1 to *= 1 removes the draw cap). Since then, each squash carries its own scoped run in its message: the size markers and RSASVE-HKDF (eb4aefd), 1168 mutants, 639 caught, 362 unviable, 25 timeouts, 142 missed and itemised, keygen.rs's slow nondeterministic ones among them; the scheme keys (010cf1a), 505 mutants, 162 caught, 330 unviable, 2 timeouts, 11 missed; the PSS parameters (5c31875), 481 mutants, 187 caught, 264 unviable, no timeouts, 27 missed after the targeted re-runs, plus utils const_str.rs, 39 mutants, 19 caught, 19 unviable, 1 build-time hang. Each squash message itemises its misses.

AI Usage Statement

• Yes, non-trivial code changes were generated by AI

Assisted-by: Claude:claude-sonnet-5
Assisted-by: Claude:claude-fable-5-1
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Claude:claude-mythos-5-1

🤖 Generated with Claude Code

[dghgit]

cargo mutants -- full-crate confirmation run, pre-trait-wiring state (61aa840)

Completed on separate hardware (mirroring PR #143's approach): 646/646 mutants, 446 caught, 6 missed, 3 timeout, 191 unviable.

All 9 non-caught findings are already-documented equivalents/artifacts, consistent with the two-pass investigation described in the PR body:

Missed (6), all documented OR/XOR or boundary equivalents:

• emsa_pkcs1_v1_5.rs:124:18, :151:28, :192:20 (decode_digest_info/emsa_pkcs1_v1_5_verify) -- boundary-condition and short-circuit equivalents already covered by the crate's own vector-driven tests.
• modexp.rs:109:54, :185:36, :193:54 (redc_normalize/ct_shift_in_bit_mod) -- disjoint-bit-range OR/XOR equivalences (the two operands never overlap by construction, so | and ^ agree).

Timeout (3), genuine infinite loops, not coverage gaps:

• emsa_pkcs1_v1_5.rs:182:11 (emsa_pkcs1_v1_5_verify) -- a +=→*= mutation stalls a length-tracking loop forever.
• mgf1.rs:24:18, :34:16 -- similarly stall MGF1's output-generation loop.

This confirms the pre-trait-wiring baseline (everything through commit 61aa840) is clean: no silent gaps, every non-caught mutant independently verified via code reasoning to be either a strict semantic equivalent or a loud (non-hanging-forever-undetected) infinite loop.

The later trait-wiring and key-generation commits (f1a87e2..1efdd64) have their own scoped mutants run in progress, covering the 14 files those commits touch; that result will follow as a separate comment once complete.

[dghgit]

cargo mutants -- scoped run over the trait-wiring + keygen commits (f1a87e2..1efdd64)

Scoped to the 14 files those commits touch (emsa_pkcs1_v1_5.rs, emsa_pss.rs, emsa_pss_shake.rs, keygen.rs, keys.rs, rsa_1024/1536/2048/3072/4096/8192.rs, rsassa_pkcs1_v1_5.rs, rsassa_pss.rs, rsassa_pss_shake.rs), --jobs 3 --timeout 3000, single-threaded tests.

Result: 751/751 mutants -- 476 caught, 0 missed, 180 unviable, 95 timeout. Every timeout was individually investigated (not just left as "presumably fine"); none is a silent gap. They break down as:

• 2 build timeouts -- cold scratch-copy compile time exceeding the auto-set cap, unrelated to the mutation.
• ~14 genuine infinite loops -- real hangs, i.e. legitimate kills, just slow ones. All in keygen.rs's retry/gate logic (see below) or has_small_factor's prime-sieve loop.
• ~79 equivalent-at-cap -- the mutated code runs to completion correctly (confirmed by watching many full test binaries pass in the log) but the single-threaded suite's wall-clock is close enough to the 3000s cap that the run gets cut off deep in the 8192-bit tests before finishing everything. Not equivalents in the strict sense for all of these -- see the structural notes below -- but none change the outcome the tests check.

Why so many timeouts turn out harmless

• gcd/div/shl/shr/trailing_zeros/mask_to_bits/inverse_of_e_mod in keygen.rs feed only the FIPS 186-5 criterion-3 "d > 2^(nlen/2)" gate check, never the actual signing exponents (dP/dQ/qInv, computed independently via Fermat's theorem). Breaking these helpers can make the gate wrongly retry (a real infinite loop, ~14 of the above) or wrongly skip a redundant margin check (harmless, since the key is still correct) -- but it can never produce a broken key.
• has_small_factor's prime-batching loop has three distinct ways its index bookkeeping can be broken into a genuine, deterministic infinite loop (confirmed by tracing the fixed batching structure by hand, not just observing the hang).
• is_probable_prime's parity and small-value fast paths are redundant shortcuts whose fallback (full trial division / Miller-Rabin) independently reaches the same correct answer, so breaking the shortcut is invisible.
• candidate()'s odd-forcing bit (out[0] |= 1 mutated to &=) degrades the low-order entropy of generated primes without ever producing an invalid one -- rejected-if-even candidates are simply retried. Functionally harmless, but a real (if narrow) reduction in randomness quality that no functional test could ever catch.

Four things worth your attention (none are silent mutants -- all found by manually reasoning through the timeout set, not something cargo-mutants itself flagged as missed)

1. primes_too_close's rejection mechanism is never tested against a deliberately close p/q pair. keygen_tests.rs independently re-derives |p-q| > 2^(nlen/2-100) on whatever real key came out, rather than checking the rejection fired -- and two random ~1024-bit primes landing within that (astronomically small) distance essentially never happens by chance, so the check's disablement is invisible either way.
2. keygen_from_rng's RNG-security-strength gate is only ever tested strictly above the required strength, never exactly at it. The suite's two RNG scenarios are None (must fail) and _256bit (must succeed) -- nothing tests an RNG at exactly the strength a given modulus size requires, which is the ordinary real-world case, not an edge case.
3. Highest-leverage one: core-test-framework's test_boundary_conditions (used by every RSA size's test_keys call) never asserts a real PK::from_bytes(&pk.encode()) == pk round trip -- only that the encoded array has the declared (compile-time-fixed) length and that malformed-length buffers get rejected. Confirmed exhaustively: encode/encode_out are untested for every RSA size and both key-type flavors except RSA-2048 (which alone has an extra, independent round-trip test in keys_tests.rs). A single added assertion in the shared helper would close this for every algorithm using it, not just RSA.
4. rsassa_pss's and rsassa_pss_shake's public sign_from_hash functions appear to be dead code. The actual Signer::sign_final path calls sign_from_hash_with_salt directly; a crate-wide grep turns up no call site for the plain sign_from_hash anywhere but its own definition and doc comments. (PKCS#1-v1.5's equivalent sign_from_hash is called from its own sign_final -- this is PSS/PSS-SHAKE specific.) Flagging as an open question rather than a bug: could be intentional public API for one-shot, non-streaming callers, or a refactor leftover that should be wired in or removed.

A --iterate --timeout 7200 --build-timeout 900 re-run over just the 95 timeout mutants is in progress to positively confirm the genuine-infinite-loop ones don't terminate even given much more time; a follow-up comment will report that once it completes.

AI Usage Statement

• Yes, this analysis was performed by AI

Assisted-by: Claude:claude-sonnet-5

🤖 Generated with Claude Code

[dghgit]

Follow-up: --iterate --timeout 7200 re-confirmation of the 95 timeout mutants

Re-ran just the 95 timeout mutants from the scoped trait-wiring/keygen run above with a much longer cap (7200s test / 900s build) to positively confirm the genuine-infinite-loop classifications rather than relying on "didn't finish in 3000s."

Final result: 95/95 -- 76 missed, 5 caught, 14 timeout. All 14 remaining timeouts are the genuine infinite loops already itemized above, and every one of them ran the full 7200s (or hit the 900s build-timeout, for the one const-eval compile-time hang) without resolving -- confirming non-termination rather than a slow pass.

One worth flagging precisely because it's the interesting case: keygen.rs:162:5 (lt stubbed to always return true) was originally observed hanging under the 3000s cap and was provisionally grouped with the other infinite loops, but flagged at the time as behaving differently from truly deterministic hangs like is_zero's -- its corruption causes a wrapping-underflow subtraction that's genuinely input-dependent, since gcd() is fed a fresh random p/q on every run. This time it resolved to caught instead: a fast, clean panic (mod_pow needs base < modulus) within a second, on different random primes than the earlier run drew. That's not a contradiction -- it's the input-dependent behavior playing out the other way this time, and either outcome (hang or fast panic) is a legitimate kill, never a silent pass. So the final count is 14 confirmed genuine infinite loops rather than 15, with 162:5 moving to the caught column.

This closes out the mutation-testing investigation for f1a87e2..1efdd64.

[dghgit]

Merged feature/ecdsa at a42ce74 (which carries release/0.1.3alpha 029a17b, the squash of PR #190) into this branch as b5e25d7. The RSA and RSA-SVE CLI commands moved onto the release's input rule from #190: private keys through the zeroized secret readers, public keys, signatures and ciphertexts as raw bytes or hex under the global --hex-input; the RSA CLI tests write raw key files. Workspace suite passes (2408 tests).

[dghgit]

Replayed onto feature/ecdsa as rebased onto release/0.1.4alpha (69df164): feature/rsa-sig is now 7bb63e9, 26 commits on that tip, replayed from b5e25d7 with the ecdsa merges dropped and the tree restored to the old tip plus the 0.1.4 line's setup (version bump and notes file). The base stays feature/ecdsa. Workspace suite passes (2408 tests).