Repository navigation
"Run tests at cursor" takes exponentially longer with more pytest tests #21151
Description
Activity
Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.77.0. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!
I just did the update to 1.77.0 and the issue persists.
Version: 1.77.0 (Universal)
Commit: 7f329fe6c66b0f86ae1574c2911b681ad5a45d63
Date: 2023-03-29T09:57:11.797Z
Electron: 19.1.11
Chromium: 102.0.5005.196
Node.js: 16.14.2
V8: 10.2.154.26-electron.0
OS: Darwin arm64 22.3.0
Sandboxed: NoThis takes a while for me, but I think this is on the Python side. Looking at the Output window, they run one process per parameterized "variant" (not sure of the correct term.) So that's 150 processes for
range(150). VS Code core is fine.- assigned and unassigned
on Apr 24, 2023 oh, actually, this is because we "include" all of them individually. Hmm. We could probably optimize that in VS Code. I'll put in a change there.
Reacted by Matthias Alphart- assigned and unassigned
on Apr 24, 2023 12 remaining items
This seems really critical to me, I can see a 10x speed slowdown in using vscode versus running pytest. Is there a workaround? x-dist?
Hi Connor Peet (@connor4312), wanted to follow up here sorry its been a minute since we spoke on this. I am attempting to implement and I am still seeing the parameterized children test items get passed as part of the runTests request: TestRunRequest. I am trying it with this example:
@pytest.mark.parametrize("number", [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]) def test_numbers(number): assert number > 0which produces 10 parameterized tests and then the function node that is the parent of those but is runnable would be test_numbers. I checked and test_numbers has the same range as the children items (the 10 parameterized tests) but both the 10 parameterized tests + the test_numbers node show in the test run request.
Are there any things other than range which are checked to determine if the children should be coalesce? And am I right assuming that the children being combined means I should expect one test run node for the parent test_numbers and not 11?
Thanks!
here you can see the 11 items in the request, the 9th being an example of a parameterized test and the 10th being the parent of the parameterized testWhich version of the Python extension are you using? On the current prerelease I'm seeing the same behavior that the parent node of parameterized tests don't have an associated range here
You can also check this by seeing if the "go to test" button in the Test Explorer view works for a given test
This was on a branch I made with edits to add a range for the parent item - Ill give the go to test button a look to confirm and if I am able to verify ill send over a link to the branch. Thanks!
Reacted by Connor PeetConnor Peet (@connor4312) here's a video of what I am seeing, the issue being that all 4 (the 3 parameterized tests and single main test) are being included in the request. Here is a link to the branch: https://github.com/eleanorjboyd/vscode-python/tree/21151-param-gutter
param-gutter-all-showing.mov
Over a call we noticed that Eleanor was testing the gutter click action where this logic hadn't been implemented. I made a PR on VS Code (linked) to do this. It's a little more complicated, but also generic.
Reacted by Eleanor Boyd🤖 Thanks for the report and detailed reproduction. This was fixed in VS Code core by microsoft/vscode@6ab3a15 via microsoft/vscode#219809: gutter-triggered test runs now coalesce parameterized children into their runnable parent instead of including every child individually. This can be closed as completed.
Type: Bug
When using Python
pytestto parametrize tests, the test explorer takes exponentially longer, while CPU is at ~100%, the more tests are created usingparametizewhen using "Run tests at cursor" to start the tests (starting from the "Run tests" button in test explorer works fine).Here is a minimal example:
With
range(10)my system (Apple M1 Pro 16 GB) takes ~1 second (measured by hand with stopwatch).With
range(100)it takes ~8 secondsWith
range(150)it takes ~50 secondsI'd expect it to take roughly as long as running it from test explorer or
pytest <file>from command line - which is about 0.1 seconds (<1 second forrange(1500)).VS Code version: Code 1.76.2 (Universal) (ee2b180d582a7f601fa6ecfdad8d9fd269ab1884, 2023-03-14T17:54:09.061Z)
OS version: Darwin arm64 22.3.0
Modes:
Sandboxed: No
System Info
canvas_oop_rasterization: disabled_off
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
metal: disabled_off
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_renderer: enabled_on
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: disabled_off
Extensions (33)
(6 theme extensions excluded)
A/B Experiments