Repository navigation
Support shuffling / randomized test runs #2425
Description
Checklist
- Feature request has a meaningful title
- I have searched the existing issues. See all issues
- I have tested using the latest version of Pester. See Installation and update guide.
Summary of the feature request
Google test has a feature where tests can be shuffled randomly per test run. This is beneficial in integration tests scenarios when one test might fail as a result of another previously ran test. Frankly it helps find obscure bugs.
How should it work?
See google test shuffle for a good description: http://google.github.io/googletest/advanced.html#shuffling-the-tests
Reacted by Miodrag Milić
Activity
nohwnd commentedon Mar 24, 2024
Interesting. That would a bit difficult (and inefficient) to provide, because Pester relies on execution that is "top-down", so if we mix tests that come from 2 different files we have to "restore" the state for each test. And run all setups and teardowns again, breaking some of the life cycles that people probably rely on. This would be specially painful for mocks I think.
I am not against it completely, I would just like a bit more data showing why doing this is worth it.
fflaten commentedon Mar 24, 2024
Yes it would be limited to shuffling tests (
It) inside the current block (Context/Describe). Is that enough to be useful?Shuffling containers/files is also an option, though that can already be achieved manually.
twhiting commentedon Mar 24, 2024
Yes google test shuffles within a "test suite" which is the behavior that @fflaten describes.
twhiting commentedon Mar 24, 2024
It is hard to explain why it's worth it without being super abstract. I for example have a powershell module that interfaces with a windows service. It is possible that test C can cause a state within the service that test A exposes. It is VERY hard to track down these cases without a reliable randomizer.
The second part of this is needing to be able to run the tests again in the same shuffled order.
Google test can shuffle based on a seed. So say in CI a test fails when shuffled with seed X. I can then manually trigger a new run with seed X and get the same exact test run order for all tests.
twhiting commentedon Mar 24, 2024
For a good description of the seed mechanism see the google test link I posted above.
nohwnd commentedon Mar 25, 2024
Makes sense. If we randomize within the same container, in a top-down way (that is we randomize on each level, but dont jump up and down), then it is hardly any change from the current way of running tests, all you need to do is shuffle the tests in the discovered tree (in a deterministic way). I think there is even an “order” list already in the discovered tree.
@twhiting do you want to make a PR for this? Even if its just a proof of concept.
twhiting commentedon Mar 31, 2024
@nohwnd i'd love to get to the feature myself but realistically it'd be weeks/months. So if this is something you are interested in feel free to jump on it!
Thanks.
majkinetor commentedon Jun 3, 2024
I came here to see if there is a flag to do this.
This is not hard to explain at all. I have a lot of experience with this with a big number of people of all types of seniority and thousands of pester tests. It's basically a norm, rather than exception, that people will depend on the tests within the same file. They may do it intentionally or by accident. Writing good tests is hard, for sure.
When tests execute in the same order, you are prone to behave like previous tests are seeding the environment or act like they are direct prerequisites of the subsequent tests. I have seen it all:
This almost always happens with the test cases within the same file, although it does happen across files too.
The one discovers this, usually by accident, when some tests are changed, skipped or removed, or simply by passage of time if tests are fuzzy in nature (my typical case). To fight some of this, I have Grafana dashboard of all previous runs, and I can select a particular test and see how often it failed during previous months.
Dashboard of screenshot with one suspicious test failure in last 30 days
This is IMO mandatory stuff to have. Randomization will either solve those problems or make them detectable quicker. I ask from devs that before commiting tests to master they provide proof of at least hundreeds of runs by using something similar to
1..100 | % { Invoke-Pester ... }, which would work much better with randomization included.fflaten commentedon Aug 7, 2026
@majkinetor @twhiting: There's an experimental shuffle feature in Pester 6.1.0-rc1. If you have time, please give it a spin for your use cases including opt-out behavior/granularity if applicable.
From release notes:
nohwnd commentedon Aug 11, 2026
Shipping in 6.1.0, implemented in #2948. That PR said "Implements #2425" instead of "Fixes", so this stayed open, it should have been closed back then.
Run.Shufflereorders test files, and the blocks and tests inside them, only ever within their own level. The seed is printed at the start of the run and returned on the result, setRun.ShuffleSeedto replay a specific order. A file that has to keep its order opts out with#pester:no-shuffle. Experimental for now.Thanks @twhiting for the request and the google test pointer.
🤖
twhiting commentedon Aug 11, 2026
Thank you! This is a sneaky valuable feature 🙂