Skip to content

Stochastic noise extensions - #2442

Merged
gavinevans merged 15 commits into
metoppv:masterfrom
gavinevans:mobt_1244_wet_stochastic_noise2
Sep 17, 2026
Merged

gavinevans merged 15 commits into
metoppv:masterfrom
gavinevans:mobt_1244_wet_stochastic_noise2

Conversation

@gavinevans

@gavinevans gavinevans commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Related to https://github.com/metoppv/mo-blue-team/issues/1268
improver_test_data PR: metoppv/improver_test_data#147

Description
This PR extends the stochastic noise implementation with two main aims:

  1. When recycling ensemble members (e.g. for increasing the number of MOGREPS-UK members from 18 to 24), the current approach would produce duplicate members that could only differ due to differences in the originally dry areas, arising from the use of stochastic noise to break ties. This, however, results in ensemble members that are very similar. In order, for the additional realizations to be useful, this PR adds the ability to add stochastic noise to positive (wet) areas, rather than just adding stochastic noise to dry areas for tie breaking. This allows ensemble members to differ in a more discernible way. The use of stochastic noise to wet areas can be limited to the additional >18 realizations.
  2. Some forecast sources e.g. MOGREPS-G or ECMWF IFS are coarser resolution models, so don't represent small scale detail, it may be useful to add some stochastic noise to these models to include some small-scale variability.

The stochastic noise values created by pysteps can sometimes be quite large (up to 5 mm/hr), so a wet-noise-amplitude multiplier is added, so that the stochastic noise can be controlled, so that it adds some small-scale variability, without markedly changing the original forecast.

Testing:

  • Ran tests and they passed OK
  • Added new tests for the new feature(s)

@maxwhitemet maxwhitemet left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Gavin. This looks good to me. I've suggested some small changes.

Comment thread improver/calibration/stochastic_noise.py
Comment thread improver/clustering/realization_clustering.py Outdated
Comment thread improver/calibration/stochastic_noise.py Outdated
Comment thread improver/calibration/stochastic_noise.py Outdated
Comment thread improver/clustering/cluster_sources_utils.py
maxwhitemet
maxwhitemet previously approved these changes Sep 3, 2026

@maxwhitemet maxwhitemet left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy with the changes made. Approved 👍

mo-jbeaver
mo-jbeaver previously approved these changes Sep 4, 2026

@mo-jbeaver mo-jbeaver left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added some comments below. The unit tests passed successfully but the acceptance tests failed:

FAILED improver_tests/acceptance/test_stochastic_noise.py::test_scale_non_positive_noise - ValueError: Checksum for stochastic_noise/scaled/kgo.nc is 45940447df6131b21f66081fcb105b8d9b2e7ffe79b3db7c97791eafab0e2d88, expected 707e828564189398fb705eeafeff94ba368714697bfe20f22dfe6a8f67cc2616 in /home/users/jasmine.beaver/impr... FAILED improver_tests/acceptance/test_stochastic_noise.py::test_dry_realizations[False] - clize.errors.UnknownOption: improver stochastic-noise: Unknown option '--wet-noise-floor'. Did you mean '--non-positive-noise-floor'? FAILED improver_tests/acceptance/test_stochastic_noise.py::test_dry_realizations[True] - clize.errors.UnknownOption: improver stochastic-noise: Unknown option '--wet-noise-floor'. Did you mean '--non-positive-noise-floor'? FAILED improver_tests/acceptance/test_stochastic_noise.py::test_positive_regions[False] - clize.errors.UnknownOption: improver stochastic-noise: Unknown option '--wet-noise-amplitude'. Did you mean '--positive-region-noise-amplitude'? FAILED improver_tests/acceptance/test_stochastic_noise.py::test_positive_regions[True] - clize.errors.UnknownOption: improver stochastic-noise: Unknown option '--wet-noise-amplitude'. Did you mean '--positive-region-noise-amplitude'?

Comment thread improver/utilities/temporal_interpolation.py Outdated
Comment thread improver/calibration/stochastic_noise.py Outdated
Comment thread improver/calibration/stochastic_noise.py Outdated
Comment thread improver/calibration/stochastic_noise.py Outdated
Comment thread improver/calibration/stochastic_noise.py Outdated
Comment thread improver/clustering/cluster_sources_utils.py Outdated
Comment thread improver_tests/calibration/stochastic_noise/test_StochasticNoise.py Outdated
Comment thread improver_tests/calibration/stochastic_noise/test_StochasticNoise.py Outdated
Comment thread improver_tests/calibration/stochastic_noise/test_StochasticNoise.py
Comment thread improver_tests/acceptance/test_stochastic_noise.py Outdated
@mo-jbeaver
mo-jbeaver dismissed their stale review September 4, 2026 10:12

Accidently approved instead of requesting changes.

@mo-jbeaver mo-jbeaver assigned gavinevans and unassigned mo-jbeaver Sep 4, 2026
@gavinevans gavinevans assigned mo-jbeaver and unassigned gavinevans Sep 15, 2026
@gavinevans
gavinevans force-pushed the mobt_1244_wet_stochastic_noise2 branch from 9a16ad0 to bb0c76e Compare September 16, 2026 07:49
mo-jbeaver
mo-jbeaver previously approved these changes Sep 16, 2026

@mo-jbeaver mo-jbeaver left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy with the updates made and all tests passed successfully.

@gavinevans
gavinevans force-pushed the mobt_1244_wet_stochastic_noise2 branch from 8106caa to 3fbc6fb Compare September 17, 2026 08:11
@gavinevans

Copy link
Copy Markdown
Contributor Author

I've updated the checksums in this commit: bbe9a2c, which fixes a forecast_trajectory_gap_filler acceptance test failure that I think was introduced by #2436.

@mo-jbeaver mo-jbeaver left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy with the changes made and all tests passed successfully.

@mo-jbeaver mo-jbeaver assigned gavinevans and unassigned mo-jbeaver Sep 17, 2026
@gavinevans
gavinevans merged commit 118cdcb into metoppv:master Sep 17, 2026
7 checks passed
@gavinevans
gavinevans deleted the mobt_1244_wet_stochastic_noise2 branch September 17, 2026 13:39
gavinevans added a commit to gavinevans/improver that referenced this pull request Sep 18, 2026
…sion

* upstream/master:
  Spatial morphing (metoppv#2441)
  Stochastic noise extensions (metoppv#2442)
  Forecast capping if calibrate forecast is too far from the uncalibrated (metoppv#2449)
  Vicinity metadata from the threshold plugin (metoppv#2439)
  Update cluster sources bookkeeping, so that forecast periods are only stored in one forecast source. (metoppv#2437)
  Correct realization numbering bug in temporal interpolation (metoppv#2436)
  Modify precedence ordering when clustering (metoppv#2435)
MoseleyS added a commit that referenced this pull request Sep 18, 2026
* master:
  Spatial Morphing - refinement of Google FILM output (#2451)
  Spatial morphing (#2441)
  Stochastic noise extensions (#2442)
  Forecast capping if calibrate forecast is too far from the uncalibrated (#2449)
  Vicinity metadata from the threshold plugin (#2439)
  Update cluster sources bookkeeping, so that forecast periods are only stored in one forecast source. (#2437)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants