Skip to content

core/txpool, eth: preserve rebroadcast retries and avoid timeout jailing - #2510

Open
0xrukimedo wants to merge 14 commits into
developfrom
vbhattac/tx-propagation-reliability
Open

0xrukimedo wants to merge 14 commits into
developfrom
vbhattac/tx-propagation-reliability

Conversation

@0xrukimedo

@0xrukimedo 0xrukimedo commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Keep transactions eligible for rebroadcast when peer queues refuse them, and prevent downloader timeouts from escalating to peer jailing. Rebroadcast batches rotate fairly by last offer/admission time; actual queue offers enforce retry intervals while queue acceptance determines the maximum-age cutoff.

  • Keep offer and acceptance tracking separate, with bounded batch selection and cleanup when transactions leave the pool. Suppressed, private, or peerless batches do not consume an offer.
  • Preserve retries for old transactions until a peer queue accepts them, including transactions that become executable after a fee drop.
  • Keep downloader timeout backoff at 10 seconds without adding jail strikes. Non-timeout failure escalation remains intact.

Executed tests

On the reduced scope:

  • Full short race suites passed for core/txpool/legacypool (142.708s), core/txpool (2.662s), eth (122.794s), and eth/downloader (206.847s).
  • Scoped golangci-lint v2.11.4 reports zero issues; workflow actionlint and diff checks pass.
  • Diffguard with mutations enabled against the PR merge base exits 0: tier-1 logic 41/41 (100%), overall 93/101 (92.1%). Eight statement/call/boolean mutations survive; tier-2 and existing churn warnings remain. No thresholds were relaxed.

Fresh CI must validate the reduced revision.

Rollout notes

No consensus, wire-format, database, public API, or configuration changes. Existing rebroadcast interval, maximum-age, and batch-size settings apply. Downloader timeout-only failures retain their 10-second backoff without jail escalation.

@codecov

codecov Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.82%. Comparing base (1cdf43c) to head (e7646d3).
⚠️ Report is 4 commits behind head on develop.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #2510      +/-   ##
===========================================
+ Coverage    57.76%   57.82%   +0.05%     
===========================================
  Files          966      969       +3     
  Lines       176609   176705      +96     
===========================================
+ Hits        102025   102182     +157     
+ Misses       68921    68866      -55     
+ Partials      5663     5657       -6     
Files with missing lines Coverage Δ
core/txpool/legacypool/legacypool.go 80.04% <100.00%> (+1.55%) ⬆️
core/txpool/legacypool/legacypool_stranded.go 100.00% <100.00%> (ø)
core/txpool/legacypool/rebroadcast.go 100.00% <100.00%> (ø)
core/txpool/legacypool/rebroadcast_attempt.go 100.00% <100.00%> (ø)
core/txpool/legacypool/rebroadcast_selection.go 100.00% <100.00%> (ø)
core/txpool/rebroadcast_attempt.go 100.00% <100.00%> (ø)
eth/downloader/bor_fetchers_concurrent.go 78.93% <100.00%> (ø)
eth/downloader/peer.go 92.56% <100.00%> (ø)
eth/downloader/peer_response.go 98.07% <100.00%> (ø)
eth/downloader/skeleton.go 79.27% <100.00%> (+0.34%) ⬆️
... and 2 more

... and 24 files with indirect coverage changes

Files with missing lines Coverage Δ
core/txpool/legacypool/legacypool.go 80.04% <100.00%> (+1.55%) ⬆️
core/txpool/legacypool/legacypool_stranded.go 100.00% <100.00%> (ø)
core/txpool/legacypool/rebroadcast.go 100.00% <100.00%> (ø)
core/txpool/legacypool/rebroadcast_attempt.go 100.00% <100.00%> (ø)
core/txpool/legacypool/rebroadcast_selection.go 100.00% <100.00%> (ø)
core/txpool/rebroadcast_attempt.go 100.00% <100.00%> (ø)
eth/downloader/bor_fetchers_concurrent.go 78.93% <100.00%> (ø)
eth/downloader/peer.go 92.56% <100.00%> (ø)
eth/downloader/peer_response.go 98.07% <100.00%> (ø)
eth/downloader/skeleton.go 79.27% <100.00%> (+0.34%) ⬆️
... and 2 more

... and 24 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@0xrukimedo

Copy link
Copy Markdown
Contributor Author

@claude review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-component concurrency and privacy changes need human review, with unresolved propagation issues and the required mutation gate still failing.

2 open issues
What changed in this PR

Improves Bor’s transaction propagation and block-fetch recovery under congestion, initial sync, and peer timeouts.

Changes:

  • Adds fair rebroadcast scheduling and pending announcements after initial sync.
  • Preserves private transaction state through admission and pool residence.
  • Bounds block-fetch requests and separates timeout backoff from failure escalation.
File Description
internal/​ethapi/​transaction_args_test.go Updates backend mock.
internal/​ethapi/​backend.go Adds privacy reservation interface.
internal/​ethapi/​api.go Reserves privacy before admission.
internal/​ethapi/​api_test.go Updates private-submission tests.
internal/​ethapi/​api_private_test.go Tests concurrent private submissions.
eth/​sync.go Shares pending-hash filtering.
eth/​sync_completion_test.go Tests post-sync announcements.
eth/​relay/​relay.go Exposes privacy reservations.
eth/​relay/​private_tx_store.go Retains privacy during pool residence.
eth/​relay/​private_tx_store_test.go Tests retention and sweeping.
eth/​relay/​private_tx_store_submission.go Implements shared admission reservations.
eth/​relay/​private_tx_store_submission_test.go Tests reservation lifecycle.
eth/​rebroadcast_delivery_test.go Adjusts transaction-age fixture.
eth/​handler.go Announces pending transactions after sync.
eth/​filters/​IBackend.go Updates generated backend mock.
eth/​fetcher/​block_fetcher.go Integrates bounded fetch scheduling.
eth/​fetcher/​block_fetcher_timeout_recovery_test.go Tests timeout recovery.
eth/​fetcher/​block_fetcher_schedule.go Implements deferred request scheduling.
eth/​fetcher/​block_fetcher_schedule_test.go Tests scheduling capacity and priority.
eth/​fetcher/​block_fetcher_schedule_edges_test.go Tests scheduling edge cases.
eth/​fetcher/​block_fetcher_request.go Handles request execution and completion.
eth/​fetcher/​block_fetcher_release.go Releases matching request ownership.
eth/​fetcher/​block_fetcher_release_test.go Tests ownership-safe cleanup.
eth/​fetcher/​block_fetcher_pending.go Bounds pending announcement origins.
eth/​fetcher/​block_fetcher_pending_test.go Tests announcement accounting.
eth/​fetcher/​block_fetcher_bounds_test.go Tests blocked-dispatch limits.
eth/​fetcher/​block_fetcher_body_request_test.go Tests body-request cleanup.
eth/​fetcher/​block_fetcher_admission.go Adds request capacity limits.
eth/​fetcher/​block_fetcher_admission_test.go Tests admission limits and retries.
eth/​downloader/​skeleton.go Avoids timeout strike escalation.
eth/​downloader/​peer.go Makes backoff reason-aware.
eth/​downloader/​peer_timeout_test.go Tests repeated timeout handling.
eth/​downloader/​peer_response.go Passes failure reason to backoff.
eth/​downloader/​peer_response_test.go Preserves non-timeout escalation tests.
eth/​downloader/​bor_fetchers_concurrent.go Classifies stale requests as timeouts.
eth/​downloader/​bor_fetchers_concurrent_test.go Tests departed-peer timeout handling.
eth/​downloader/​bor_fetch_helpers_test.go Updates stale-request expectations.
eth/​api_backend.go Excludes private transactions from retry journaling.
eth/​api_backend_private_test.go Tests private admission and journaling.
eth/​api_backend_private_lifecycle_test.go Tests privacy lifecycle and duplicates.
core/​txpool/​legacypool/​rebroadcast.go Tracks attempts separately from acceptance.
core/​txpool/​legacypool/​rebroadcast_test.go Updates attempt-expiry expectations.
core/​txpool/​legacypool/​rebroadcast_selection.go Selects bounded, oldest-activity batches.
core/​txpool/​legacypool/​rebroadcast_scheduling_test.go Tests rotation and queue recovery.
core/​txpool/​legacypool/​rebroadcast_fairness_test.go Tests fairness amid new arrivals.
core/​txpool/​legacypool/​rebroadcast_cleanup_test.go Tests attempt-history cleanup.
core/​txpool/​legacypool/​rebroadcast_boundaries_test.go Tests eligibility and ordering boundaries.
core/​txpool/​legacypool/​legacypool.go Integrates attempt tracking and cleanup.
core/​txpool/​legacypool/​legacypool_stranded.go Clears history for stranded transactions.

Review effort: Balanced

Comment thread eth/fetcher/block_fetcher_schedule.go Outdated
Comment thread eth/handler.go Outdated
Comment thread eth/fetcher/block_fetcher_request.go Outdated
Comment thread eth/relay/private_tx_store.go
Comment thread eth/fetcher/block_fetcher_schedule.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-component concurrency, request ownership, and transaction privacy changes warrant final human review and confirmation of green full-suite CI.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@0xrukimedo

Copy link
Copy Markdown
Contributor Author

@claude review

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Still open from earlier reviews (1):

  • 🔴 eth/relay/private_tx_store.go:197 — Private transactions that block producers reject can now stay hidden in the relay's pool indefinitely. They are never g…

If you have decided not to act on one of these findings, resolve its thread (a reply alone leaves it open) and the next review stops counting it. To review this commit again now, use Re-run on its "Claude Code Review" check.

Additional findings (outside the current diff — GitHub can't attach inline comments there):

  • 🔴 eth/fetcher/block_fetcher.go — Nodes that catch up through the downloader can permanently keep other peers' block announcements and their quota credits. Eventually announcements from new peers are refused. When a fetcher-queued block is popped as already known or too old, line 577 calls only forgetBlock. The stale f.completing[hash] keeps fetchInProgress true, so the hash's alternatives in f.announced are deferred forever. Fix: clear every stage for a discarded queued block (call forgetHash along with forgetBlock, as the done case does), so that alternatives and their f.announces credits are released.

    Why this was flagged

    The fetcher fetches block B (within maxQueueDist) and its body. B is enqueued with f.completing[B] set and waits because number > height+1 (block_fetcher.go:562). While B sits in completing/queued, other peers announce B. hasAnnouncer (block_fetcher_alternates.go:51) only rejects the same origin, so each new announcer is appended to f.announced[B] and f.announces[origin] is incremented (block_fetcher.go:634-635). The downloader then imports B. The import loop sees getBlock(B) != nil or number+maxUncleDist < height and calls only forgetBlock (block_fetcher.go:575-578). From then on, prepareHeaderFetch returns (nil, true) at fetchInProgress (block_fetcher_schedule.go:64-66) before readyToFetch can drop B, so the alternatives and their credits are never released. On the base branch the stale completing entry made the notify handler ignore further announcements of B, so no announcement or credit was retained.

    Verification: When the queued B is popped as already known or too old, block_fetcher.go:575-577 calls only f.forgetBlock(hash). That removes queued/queues and leaves f.completing[B] behind permanently. Each other peer's announce for B is therefore appended, and f.announces[origin] = count is charged (block_fetcher.go:634-635).

Comment thread eth/api_backend.go Outdated
Comment thread eth/fetcher/block_fetcher.go Outdated
@0xrukimedo
0xrukimedo marked this pull request as ready for review October 22, 2026 10:21
Copilot AI balanced review requested due to automatic review settings October 22, 2026 10:21

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Producer-rejected private transactions still lack a bounded automatic recovery policy that preserves the endpoint’s privacy contract.

2 open issues

Review effort: Balanced

Comment thread eth/relay/private_tx_store.go Outdated
Comment thread eth/fetcher/block_fetcher_alternates.go Outdated
Copilot AI balanced review requested due to automatic review settings October 22, 2026 10:49
@0xrukimedo

Copy link
Copy Markdown
Contributor Author

Fixed the out-of-diff queued-block cleanup finding in 2b88eb6. When a queued block is already known or too old, the import loop now calls both forgetHash and forgetBlock. This clears stale completion state, retained alternatives, and their announcement credits as well as queue ownership. TestBlockFetchDiscardReleasesAlternatives drives the real fetcher loop for both cases and verifies stage/quota cleanup.

The private-retention recovery finding remains open. The two reviews request conflicting policies: automatic public fallback versus preserving the private payload. I have asked the PR author to choose the policy; this commit does not silently publish private transactions.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The acknowledged private-transaction recovery policy after producer retry exhaustion still needs an explicit decision, implementation, and tests before merging.

1 open issue
1 resolved since last review

Review effort: Balanced

@0xrukimedo

Copy link
Copy Markdown
Contributor Author

@claude review

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Still open from earlier reviews (1):

  • 🔴 eth/relay/private_tx_store.go:197 — Private transactions that block producers reject can now stay hidden in the relay's pool indefinitely. They are never g…

If you have decided not to act on one of these findings, resolve its thread (a reply alone leaves it open) and the next review stops counting it. To review this commit again now, use Re-run on its "Claude Code Review" check.

Comment thread eth/fetcher/block_fetcher_admission.go Outdated
Comment thread core/txpool/legacypool/rebroadcast.go Outdated
Comment thread core/txpool/legacypool/rebroadcast.go Outdated
Copilot AI balanced review requested due to automatic review settings October 23, 2026 07:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Bounded private-transaction recovery after producer retries exhaust remains unresolved, with implementation and regression tests still outstanding.

No open issues 🎉

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 23, 2026 10:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The concurrent recovery and privacy changes require human review and current-head CI validation, with partial-body-response recovery still unresolved.

1 open issue

Review effort: Balanced

Comment thread eth/fetcher/block_fetcher_release.go Outdated
Copilot AI balanced review requested due to automatic review settings October 23, 2026 12:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-component retry scheduling and confidentiality changes require human sign-off and current-head CI validation.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Low severity Clarify privacy marker expiry for pool-resident transactions

eth/​relay/​CLAUDE.md:84

This guidance says privacy markers remain while the pool retains a transaction, but privateTxExpired now expires them after 30 minutes even for pool-resident transactions. The original payload then becomes eligible for public propagation without a replacement or public resubmission. Document the bounded privacy window and the in-flight admission exception here, consistent with docs/private-transactions.md, so the recovery guidance does not imply indefinite confidentiality.

1 resolved since last review

Review effort: Balanced

@0xrukimedo

Copy link
Copy Markdown
Contributor Author

@claude review

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Comment thread core/txpool/legacypool/rebroadcast_attempt.go
Comment thread eth/relay/CLAUDE.md Outdated
Copilot AI balanced review requested due to automatic review settings October 23, 2026 14:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Privacy-sensitive public fallback and concurrent fetch scheduling require human sign-off and current-head integration validation.

No open issues 🎉

Review effort: Balanced

@0xrukimedo

Copy link
Copy Markdown
Contributor Author

@claude review

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the latest revision and found no new bugs. Because this is a large change (76 files, about 7k lines) covering tx propagation, block-fetch scheduling and the private-transaction privacy lifecycle, a human look would still be worthwhile.

What was reviewed:

  • The rebroadcast max-age gate in core/txpool/legacypool/rebroadcast_selection.go now keys on lastRebroadcast, so a refused queue offer no longer uses up the one late propagation attempt.
  • The updated "Failure recovery" text in eth/relay/CLAUDE.md now describes the 30-minute marker expiry and public fallback, which matches private_tx_store.go and docs/private-transactions.md.
  • The hunt also covered the private admission reserve/commit path, fetcher slot bounds and release ownership, and post-sync announcement retries. It found nothing new there.
Extended reasoning...

The PR changes legacypool rebroadcast selection and attempt tracking, the post-sync pending announcements in eth, the block fetcher and downloader timeout/backoff handling, and private-transaction admission in eth/relay and internal/ethapi. The security-sensitive part is data exposure: private transactions now become eligible for public gossip after a 30-minute marker expiry, a design trade-off a maintainer should sign off on. The latest commit addresses the two issues raised in the previous automated review. Earlier threads were all resolved by the PR author rather than independently, and the size and peer-handling complexity rule out an automated approval.

Copilot AI balanced review requested due to automatic review settings October 26, 2026 08:10
@0xrukimedo 0xrukimedo changed the title core/txpool, eth, internal/ethapi: improve transaction propagation reliability core/txpool, eth: preserve rebroadcast retries and avoid timeout jailing Oct 26, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Network-sensitive retry scheduling and peer-penalty changes require final human review and current-head CI validation.

No open issues 🎉

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 26, 2026 08:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-component retry scheduling and peer penalties affect network liveness and warrant final human review with fresh CI validation.

No open issues 🎉

Review effort: Balanced

@0xrukimedo

Copy link
Copy Markdown
Contributor Author

@claude review

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

return false
}
if !last.IsZero() {
if now.Sub(last) < pool.config.RebroadcastInterval {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Transactions whose peer-queue offers were refused are now retried about every 60s, not every RebroadcastInterval (30s) as on the base branch. The attempt timestamp is written after the tick that selected the tx, so on the next tick now.Sub(last) < pool.config.RebroadcastInterval is still true and the tx is skipped. Fix: compare the tick or selection time to the tick or selection time, for example by passing the now from identifyStuckTransactions into the attempt record, or by allowing a small slack. The acceptance timestamp in lastRebroadcast has the same pre-existing skew, so the fix should cover both.

Why this was flagged

The pool loop's ticker (legacypool.go:461) fires at T. identifyStuckTransactions runs under RLock, and the batch then goes through rebroadcastAckFeed to stuckTxBroadcastLoop. queueTransactions calls the RebroadcastAttempt callback, which sets lastRebroadcastAttempt[hash] = now at T+δ (rebroadcast_attempt.go:36,44). δ covers the scan of every pending tx plus the handler's work, so it is milliseconds. The next tick computes now at about T+30s+ε (ε is wake-up jitter, microseconds). rebroadcastEligible (rebroadcast_selection.go:61) then sees now.Sub(last) = 30s-(δ-ε) < RebroadcastInterval and skips the tx. It becomes eligible only on the tick after that. On the base branch a refused tx left nothing recorded, so the next tick retried it after 30s. Now every refused offer waits two intervals, which halves retry frequency for stuck txs when peer queues are full and the pool holds fewer than RebroadcastBatchSize eligible txs. The tests set attempt times hours in the past (rebroadcast_scheduling_test.go:58,91,146), so none of them exercises the tick boundary.

Verification: Trigger: a stuck tx whose peer-queue offer is refused. identifyStuckTransactions takes now at tick T (rebroadcast_selection.go:41), and RebroadcastAttempt writes pool.lastRebroadcastAttempt[hash] later (rebroadcast_attempt.go:44), so now.Sub(last) < pool.config.RebroadcastInterval (line 61) is true on the next tick and the tx is skipped. On base, refused txs were eligible on every 30s tick.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants