fix(docker): read upgraded exec streams through the buffered reader

[scylla-zeus-bot]

What

A docker exec can return empty output with exit code 0, and SCT then uses that empty output as a real value. One effect is that an empty mktemp result becomes the path in container.put_archive(path=""). The daemon refuses every write with 400 Bad Request: bad parameter: path cannot be empty, the 300 s retry budget in sdcm/wait.py runs out, and the run fails with NodeSetupFailed while SCT writes scylla.yaml into the container. See artifacts-docker-fips-test build 496.

Root cause

The daemon answers an exec start with 101 UPGRADED headers and then writes the output frames on the same connection. http.client parses those headers through a BufferedReader that reads up
to 8 KiB at a time, so a frame arriving in the same recv() lands in that buffer. docker-py skips the buffer and reads from the socket, so it never sees the frame.

This is docker/docker-py#3332, also reported as #2042. The upstream fix is docker/docker-py#3333, open and unreviewed since May 2025, and docker-py 7.2.0 still carries the old code. This change mirrors it, so it can be dropped in one piece once a release includes it.

It is not a recent regression, and nothing is specific to mktemp. Any DockerCmdRunner.run() call whose output SCT parses can hit it, and so can the other exec_run() callers, such as the gcloud container in sdcm/utils/gce_utils.py. The window is small, so the failure is rare, and a busy runner widens it.

Fix

• sdcm/utils/docker_utils.py: BufferedStreamAPIClient gives docker-py's frame reader the buffer instead of the socket, and docker.utils.socket.read reads a buffer without polling
  the socket first. docker-py still does the framing, the demux and the response close. SCT's DockerClient builds its api object from the subclass, so every container SCT talks to is
  covered. The ssh and named-pipe transports keep something else in that attribute and still go through docker-py.
• Skipping that poll is part of the fix. A frame already in the buffer would otherwise wait for traffic on an idle socket, so a streamed line would arrive only when the next line does.
• sdcm/remote/remote_file.py: raise when the mktemp result is empty, instead of retrying an unusable destination for 300 s. This also covers the ssh transport, where the change above
  deliberately leaves docker-py in charge.
• sdcm/remote/docker_cmd_runner.py: build the extraction directory with PurePosixPath. A path inside a container is POSIX whatever the host runs.

Fixes SCT-952

Testing

• pr-provision-test-docker

[scylladb-promoter]

❌ Test Summary: FAILED

✅ Precommit: PASSED

Total	Passed	Failed	Skipped
44	20	0	24

❌ Tests: FAILED

Total	Passed	Failed	Errors	Skipped
6742	6685	2	0	55

Failed Tests

integration.test_events.TestSctEventsIntegration.test_default_filters_non_gce_backend_keeps_bind_race_as_error

Type: FAILURE
Message: AssertionError: assert 'Could not start Prometheus API server' in ''

Traceback

self = <unit_tests.integration.test_events.TestSctEventsIntegration object at 0x7dc4ba474c30>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7dc499d47cb0>

    @pytest.mark.integration
    def test_default_filters_non_gce_backend_keeps_bind_race_as_error(self, monkeypatch):
        """The GCE-specific first-boot bind-race filter must not apply on other backends."""
        monkeypatch.setattr(events_setup, "SkipPerIssues", lambda *args, **kwargs: True)
    
        with environment(SCT_CLUSTER_BACKEND="docker"):
            enable_default_filters(SCTConfiguration())
    
        with self.wait_for_n_events(self.get_events_logger(), count=1):
            DatabaseLogEvent.DATABASE_ERROR().add_info(
                node="A",
                line_number=22,
                line="2026-06-27T03:17:49.539Z some-non-gce-node !ERR | scylla[1669] "
                "[shard 0:strm] init - Could not start Prometheus API server on 10.128.0.47:9180: "
                "std::system_error (error system:99, posix_listen failed for address 10.128.0.47:9180: "
                "Cannot assign requested address)",
            ).publish()
    
        error_log_content = self.get_event_log_file("error.log")
>       assert "Could not start Prometheus API server" in error_log_content
E       AssertionError: assert 'Could not start Prometheus API server' in ''

unit_tests/integration/test_events.py:145: AssertionError

integration.test_external_backtrace_service.test_decode_via_external_service_returns_symbols

Type: FAILURE
Message: ValueError: External backtrace service failed: Traceback (most recent call last): File "/code/backtrace/tasks/backtrace.py", line 78, in decode_backtrace async with extract_file(build_id, task) as cache_path: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/code/backtrace/tasks/__init__.py", line 56, in extract_file cache_path.mkdir(parents=True, exist_ok=True) File "/usr/local/lib/python3.12/pathlib.py", line 1311, in mkdir os.mkdir(self, mode) OSError: [Errno 28] No space left on device: '/app/cache/42315'

Traceback

node = <unit_tests.lib.fake_cluster.DummyNode object at 0x78f5b963e120>
known_build_id = '57e6772414554e9e359c113522c308778060325c'

    def test_decode_via_external_service_returns_symbols(node, known_build_id):
        """Verify _decode_via_external_service returns decoded symbols for a known build."""
>       result = node._decode_via_external_service(known_build_id, SAMPLE_BACKTRACE_ADDRESSES)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

unit_tests/integration/test_external_backtrace_service.py:72: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <unit_tests.lib.fake_cluster.DummyNode object at 0x78f5b963e120>
build_id = '57e6772414554e9e359c113522c308778060325c'
raw_backtrace = '0x4a3c9b7\n0x11ddd33\n0x11ddcec'

    @lru_cache(maxsize=None)
    def _decode_via_external_service(self, build_id: str, raw_backtrace: str) -> str:
        """Decode backtrace using the external backtraces.scylladb.com service.
    
        Avoids loading 1+ GB DWARF debug info on the monitor node (prevents OOM).
    
        Args:
            build_id: hex build ID of the scylla binary
            raw_backtrace: raw backtrace string (newline-separated addresses)
    
        Returns:
            decoded backtrace string (stdout from the service)
    
        Raises:
            requests.RequestException: on network/HTTP error
            ValueError: if the service returns success=false
        """
        response = requests.post(
            "https://backtrace.scylladb.com/api/backtrace",
            json={"build_id": build_id, "input": "Backtrace:\n" + raw_backtrace},
            timeout=120,
        )
        response.raise_for_status()
        result = response.json()
        if not result.get("success"):
>           raise ValueError(f"External backtrace service failed: {result.get('stderr', 'unknown error')}")
E           ValueError: External backtrace service failed: Traceback (most recen

Full build log

[fruch]

@dimakr the docker and integration tests fails on known issue

I'll try getting the docker py issue fixed (with clearer PR for it)