Skip to content

fix: serialize SSH client state machine events - #287

Closed
ptgamr wants to merge 1 commit into
connectbot:mainfrom
ptgamr:fix/serialize-state-machine-events
Closed

ptgamr wants to merge 1 commit into
connectbot:mainfrom
ptgamr:fix/serialize-state-machine-events

Conversation

@ptgamr

@ptgamr ptgamr commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

cbssh can close a working connection when the server replies quickly. This PR makes cbssh handle one message at a time properly, so that can't happen.

What goes wrong

cbssh has a rule checker (SshClientStateMachine). Every message, whether the client sends it or the server sends it, goes through it. It checks that the message is allowed at that moment, for example "a reply to a channel open is only allowed if we asked to open one". It must check one message at a time.

cbssh assumes the checker only ever handles one message at a time. But sometimes the checker pauses partway through a job, while it waits for data to go out over the network. During that pause, a second message can get in.

When that happens, the checker sees it's busy. It puts the new message in a "later" pile and answers "pending". cbssh reads "pending" as "rejected". It decides the server broke the rules and closes the connection.

Example

For cbssh, sending a packet has two parts, with a gap between them:

  1. The bytes leave. The packet is handed to the network and heads to the server.
  2. The send step finishes. cbssh's code gets control back, and the checker's job ends.

Part 2 can lag behind part 1. For example, the transport may write on a separate thread (Dispatchers.IO), and cbssh has to wait for that thread to report back. On a fast network, the server can read the packet and reply inside that gap:

  1. The client starts opening a channel. The checker is busy until the send step finishes.
  2. The "open a channel" request leaves the client.
  3. The server receives it and replies "OK, opened".
  4. The reply arrives before the send step has finished. The checker is still busy, so the reply goes in the "later" pile.
  5. cbssh treats it as a bad message and closes the connection with:
ProtocolViolationException: Unexpected SSH packet SSH_MSG_CHANNEL_OPEN_CONFIRMATION in the current protocol state

It can happen with a single channel. It is more likely when a client runs several channels on one connection, for example a shell and SFTP.

The fix

The checker now uses a lock, like a "one at a time" sign on a door. A new message waits until the previous one is fully finished, pause included. SshChannelStateMachine and SftpStateMachine already work this way. SshClientStateMachine was the only one missing it.

Why not just treat "pending" as OK? cbssh would then carry on as if a message had been handled when it hadn't. That could cause other errors.

Can the lock freeze the connection?

A lock can freeze a program if two jobs each wait for the other to finish. There are four ways that could happen here, and none of them can. The reasons are under "Technical details".

Tests

StateMachineEventSerializationTest uses the pretend server FakeSshServer. It holds the client's message mid-send, and the pretend server replies during that pause.

  • On main, the connection closes.
  • With this fix, it works, and a follow-up request on the same connection succeeds.

The full ./gradlew :protocol:build :sshlib:build passes, including the Testcontainers suites.

Technical details

Root cause

SshConnection runs state machine work on coroutineDispatcher.limitedParallelism(1). That only serializes code between suspension points; it is not a mutex.

Some transitions suspend inside their callbacks while they write a packet (OpenChannel → sendChannelOpen, SendChannelRequest → sendChannelRequest). If the transport write suspends after the bytes are sent, the packet loop can call processEvent on a busy machine. KStateMachine queues the event and returns ProcessingResult.PENDING (StateMachineImpl.kt in 0.38.1). SshClientStateMachine.process() treats anything other than PROCESSED as rejected. Local commands hit the same race as IllegalStateException: Check failed from dispatchCommand / openSessionChannel.

In our testing, 12 concurrent exec and SFTP channel opens next to a live shell, over a blocking-socket Transport that writes on Dispatchers.IO, closed the connection in 8 of 8 runs. KtorTcpTransport hides it most of the time because small localhost writes rarely suspend, but one suspension after a write is enough to reproduce it with a single caller.

Change

SshClientStateMachine.process() holds a Mutex for the whole event. receiveKexInit takes the same lock around its isWaitingForKexInit() check and callback, so they can't interleave with another event.

Why the lock can't deadlock

  • Re-entry: no transition callback calls back into SshClientStateMachine. startRekeyTimer and rekeyComplete launch new coroutines.
  • Key exchange gate: no callback can wait on OutboundPacketController's kexCompletion while holding the lock. The gate is only open (incomplete) from sendKexInit until our SSH_MSG_NEWKEYS is written. Both happen inside transitions into or within the KEX states, and those transitions only write KEX-allowed message types (KEXINIT, KEX*_INIT, NEWKEYS, DISCONNECT). The Authenticated transitions that write connection-layer packets are not reachable until the server's NEWKEYS, which comes after the gate completes.
  • Channel locks: the client lock is taken before a channel lock only in receiveChannelOpenConfirmation / receiveChannelOpenFailure, which need a pending registry entry. A channel lock is taken before the client lock only in SessionChannel.performSendRequest, which needs an established SessionChannel. openSessionChannel promotes the entry before it returns the channel, so the two paths never meet on the same channel.
  • Write lock: the beforeWrite / afterWrite hooks that run under writeMutex don't call into the state machine.

Test detail

  • The channel open case fails on main with the ProtocolViolationException above.
  • The channel request (pty-req) case fails on main with Unexpected SSH packet SSH_MSG_CHANNEL_SUCCESS, then times out.

Local run notes

The build ran on a JDK 17 toolchain, with Docker for the Testcontainers suites, and with --dependency-verification lenient. main currently fails dependency verification locally: the keys for sonarqube-gradle-plugin 7.5.0.8588, sonar-scanner-java-library-shaded 4.2.1.1698 and mockito-bom 4.11.0 are not in gradle/verification-keyring.keys, and the key servers didn't return them. That is unrelated to this change. checkTla was not run, because this change doesn't alter the state machine's structure.

🤖 Generated with Claude Code

SshConnection runs state machine work on limitedParallelism(1), which
only serializes code between suspension points. OpenChannel and
SendChannelRequest transitions suspend while their packet is written.
If the server's reply arrives during that suspension, the packet loop
calls processEvent on a busy machine. KStateMachine queues the event
and returns PENDING, which process() treats as rejected, so the loop
throws "Unexpected SSH packet ... in the current protocol state" and
closes the connection.

Hold a mutex for the whole event, as SshChannelStateMachine and
SftpStateMachine already do. receiveKexInit takes the same lock around
its state check so the check and the event can't interleave with
another event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kruton

kruton commented Sep 28, 2026

Copy link
Copy Markdown
Member

The same problem was fixed in a different way with a change I just merged.

@kruton kruton closed this Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants