Repository navigation
Conversation
SshConnection runs state machine work on limitedParallelism(1), which only serializes code between suspension points. OpenChannel and SendChannelRequest transitions suspend while their packet is written. If the server's reply arrives during that suspension, the packet loop calls processEvent on a busy machine. KStateMachine queues the event and returns PENDING, which process() treats as rejected, so the loop throws "Unexpected SSH packet ... in the current protocol state" and closes the connection. Hold a mutex for the whole event, as SshChannelStateMachine and SftpStateMachine already do. receiveKexInit takes the same lock around its state check so the check and the event can't interleave with another event. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Sep 28, 2026
Member
|
The same problem was fixed in a different way with a change I just merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cbssh can close a working connection when the server replies quickly. This PR makes cbssh handle one message at a time properly, so that can't happen.
What goes wrong
cbssh has a rule checker (
SshClientStateMachine). Every message, whether the client sends it or the server sends it, goes through it. It checks that the message is allowed at that moment, for example "a reply to a channel open is only allowed if we asked to open one". It must check one message at a time.cbssh assumes the checker only ever handles one message at a time. But sometimes the checker pauses partway through a job, while it waits for data to go out over the network. During that pause, a second message can get in.
When that happens, the checker sees it's busy. It puts the new message in a "later" pile and answers "pending". cbssh reads "pending" as "rejected". It decides the server broke the rules and closes the connection.
Example
For cbssh, sending a packet has two parts, with a gap between them:
Part 2 can lag behind part 1. For example, the transport may write on a separate thread (
Dispatchers.IO), and cbssh has to wait for that thread to report back. On a fast network, the server can read the packet and reply inside that gap:It can happen with a single channel. It is more likely when a client runs several channels on one connection, for example a shell and SFTP.
The fix
The checker now uses a lock, like a "one at a time" sign on a door. A new message waits until the previous one is fully finished, pause included.
SshChannelStateMachineandSftpStateMachinealready work this way.SshClientStateMachinewas the only one missing it.Why not just treat "pending" as OK? cbssh would then carry on as if a message had been handled when it hadn't. That could cause other errors.
Can the lock freeze the connection?
A lock can freeze a program if two jobs each wait for the other to finish. There are four ways that could happen here, and none of them can. The reasons are under "Technical details".
Tests
StateMachineEventSerializationTestuses the pretend serverFakeSshServer. It holds the client's message mid-send, and the pretend server replies during that pause.main, the connection closes.The full
./gradlew :protocol:build :sshlib:buildpasses, including the Testcontainers suites.Technical details
Root cause
SshConnectionruns state machine work oncoroutineDispatcher.limitedParallelism(1). That only serializes code between suspension points; it is not a mutex.Some transitions suspend inside their callbacks while they write a packet (
OpenChannel→sendChannelOpen,SendChannelRequest→sendChannelRequest). If the transport write suspends after the bytes are sent, the packet loop can callprocessEventon a busy machine. KStateMachine queues the event and returnsProcessingResult.PENDING(StateMachineImpl.ktin 0.38.1).SshClientStateMachine.process()treats anything other thanPROCESSEDas rejected. Local commands hit the same race asIllegalStateException: Check failedfromdispatchCommand/openSessionChannel.In our testing, 12 concurrent exec and SFTP channel opens next to a live shell, over a blocking-socket
Transportthat writes onDispatchers.IO, closed the connection in 8 of 8 runs.KtorTcpTransporthides it most of the time because small localhost writes rarely suspend, but one suspension after a write is enough to reproduce it with a single caller.Change
SshClientStateMachine.process()holds aMutexfor the whole event.receiveKexInittakes the same lock around itsisWaitingForKexInit()check and callback, so they can't interleave with another event.Why the lock can't deadlock
SshClientStateMachine.startRekeyTimerandrekeyCompletelaunch new coroutines.OutboundPacketController'skexCompletionwhile holding the lock. The gate is only open (incomplete) fromsendKexInituntil ourSSH_MSG_NEWKEYSis written. Both happen inside transitions into or within the KEX states, and those transitions only write KEX-allowed message types (KEXINIT, KEX*_INIT, NEWKEYS, DISCONNECT). TheAuthenticatedtransitions that write connection-layer packets are not reachable until the server's NEWKEYS, which comes after the gate completes.receiveChannelOpenConfirmation/receiveChannelOpenFailure, which need a pending registry entry. A channel lock is taken before the client lock only inSessionChannel.performSendRequest, which needs an establishedSessionChannel.openSessionChannelpromotes the entry before it returns the channel, so the two paths never meet on the same channel.beforeWrite/afterWritehooks that run underwriteMutexdon't call into the state machine.Test detail
mainwith theProtocolViolationExceptionabove.pty-req) case fails onmainwithUnexpected SSH packet SSH_MSG_CHANNEL_SUCCESS, then times out.Local run notes
The build ran on a JDK 17 toolchain, with Docker for the Testcontainers suites, and with
--dependency-verification lenient.maincurrently fails dependency verification locally: the keys forsonarqube-gradle-plugin7.5.0.8588,sonar-scanner-java-library-shaded4.2.1.1698 andmockito-bom4.11.0 are not ingradle/verification-keyring.keys, and the key servers didn't return them. That is unrelated to this change.checkTlawas not run, because this change doesn't alter the state machine's structure.🤖 Generated with Claude Code