don't hardcode EESSI version environments by smoors · Pull Request #362 · EESSI/test-suite

smoors · 2026-10-28T19:58:21Z

instead, query the available EESSI versions on the system with Lmod, and use those to populate the EESSI environments.

UPDATE:

also added optional param add_eessi_versions to function set_common_required_config. possible values:

True (default): add all available versions
list of EESSI versions, e.g. ['2025.06']: add only those versions
False: add no versions: the site defines its own EESSI environments and partition environs

per partition, the environs setting can change this:

no EESSI environment listed: all added versions are added
one or more EESSI environments listed, e.g. ['EESSI-2025.06']: exactly those are used, nothing is added
NO_EESSI marker listed, e.g. ['localenv', NO_EESSI]: no EESSI environments are added, the marker is removed

additionally:

keep site-defined EESSI environments instead of redefining them
add info and warning msgs
fix environ_mapping filter in find_modules, which had no effect
give each partition its own environs list, so changing one no longer affects the others
document the options in settings_example.py

casparvl

I like the idea of dynamically querying the EESSI versions available, and just testing those. I have two thoughts:

Can/should we make the set_common_required_config a bit more... configurable? I.e. add an argument set_eessi_environs = True (to add the EESSI environment to each partition), maybe a separate set_eessi_environments = True (to define the environments at the site_configuration level)? This more easily allows a site to say: don't touch them, I'll define them myself. Use case would be e.g. if a site ONLY wants to test a single EESSI environment, since it doesn't care enough to test the old ones.
Would the detection work on any system? I.e. on some systems, some action may need to be done before those EESSI modules are even visible. Shouldn't it run the eessi_prepare_cmds for example? This would allow e.g.

source /cvmfs/software.eessi.io/versions/2026.06/init/lmod/bash
module purge

On systems that don't have a module env, just so that they can get one from EESSI first. Yes, they'd need to hardcode SOME version of EESSI to initialize the Lmod environment from, but... there's no way around that.

Anyway, maybe we discuss this in this afternoons meeting, before we take a decision on how to proceed.

smoors · 2026-11-05T13:19:56Z

Can/should we make the set_common_required_config a bit more... configurable? I.e. add an argument set_eessi_environs = True (to add the EESSI environment to each partition), maybe a separate set_eessi_environments = True (to define the environments at the site_configuration level)? This more easily allows a site to say: don't touch them, I'll define them myself. Use case would be e.g. if a site ONLY wants to test a single EESSI environment, since it doesn't care enough to test the old ones.

yes, this is something i wanted to add, but wasn't sure about the best approach.

Would the detection work on any system? I.e. on some systems, some action may need to be done before those EESSI modules are even visible. Shouldn't it run the eessi_prepare_cmds for example? This would allow e.g.
source /cvmfs/software.eessi.io/versions/2026.06/init/lmod/bash
module purge
On systems that don't have a module env, just so that they can get one from EESSI first. Yes, they'd need to hardcode SOME version of EESSI to initialize the Lmod environment from, but... there's no way around that.

currently, we always have to make sure the EESSI modules are available before launching reframe, so there is no real issue. we could indeed let the test suite do it for us, for convenience.

EDIT: if we do this, we'll also have to find a way to also make them available before reframe checks the available modules. the prepare_cmds are only run in the jobs, not when "preparing" the environment (which could be seen as a reframe bug).

casparvl · 2026-11-05T14:44:39Z

currently, we always have to make sure the EESSI modules are available before launching reframe, so there is no real issue. we could indeed let the test suite do it for us, for convenience.

Hmm, you're right, I didn't think of that. Ok, we can try without first, see how that works for us. Then it's just up to the human who executes the reframe command to basically execute the prepare_cmds beforehand.

smoors · 2026-11-08T10:35:11Z

@casparvl i've implemented your first suggestion using add_eessi_versions param, see PR description. i think all use cases are covered now.

i'll leave your second suggestion for another PR if we decide to do it.

casparvl · 2026-11-09T11:32:14Z

+        if line.startswith('EESSI/')
+    }
+    if not versions:
+        getlogger().warning(


It's a bit inconsistent: if we can't detect EESSI versions because there is no LMOD_CMD, it's a hard error. If we can't detect EESSI versions because there is a module command, but your cmd has simply not returned any EESS modules (e.g. because the MODULEPATH wasn't set correctly) it's a warning. We should make this consistent, but... with what?

You could say: it's an error, because the user specified add_eessi_versions=True in their config (well, implicitely, since it is the default). If you can't detect EESSI environments, we cannot add them, and since that was requested, it is an error.

One could also argue that this is a breaking change, and that raising an error would be 'too breaking'. I.e. without changing any of the defaults, if my MODULEPATH doesn't contain EESSI (or if I don't have a modulecommand), I now suddenly get a hard error.

I'm not sure which way to go...

there's probably no clear answer, but the way i see it:

if there is not module system available (assuming we'll generalize LMOD_CMD to other module systems), then we can't test any modules at all, so should clearly error.

if there are no EESSI versions detected, we can still test local modules if there is a local environ defined, so it's not as bad.

a solution could be to add a ReFram variable, say no_eessi_modules_found_policy, similar to required_mem_per_node_undefined_policy and readonly_files_undefined_policy. then each user can decide if they want an error or a warning.

what do you think?

Hmmm, maybe I'm just bikeshedding. I was really thinking about it from a single function perspective: if you ahve a function that is supposed to 'get' the EESSI versions, and it didn't find any, I thought that was a (hard) failure as well. I.e. function didn't return what it was supposed to return. But I guess that's not the case: it did return what it was supposed to return, namely 'there are no EESSI modules on this system' (but we can still test the local ones).

Fine, let's just keep it like this - maybe generalize the LMOD_CMD to other module systems. I do still think that a hard error on a system that has Environment Modules instead of Lmod is... too harsh.

Anyway, this conversation is resolve,d we'll pick up the LMOD_CMD thing at that line :)

changed it to use warning for the missing Lmod, since other module systems may stil be available and so local modules can still be tested. so both are now a warning. also using the ReFrame machinery for the modules sytem now.

7bd05a3

casparvl · 2026-11-09T11:32:58Z

+    The EESSI modules must already be available (in $MODULEPATH) before running ReFrame.
+    :return: sorted list of available EESSI versions, e.g. ['2023.06', '2025.06']
+    """
+    lmod_cmd = os.getenv('LMOD_CMD')


What if you're running on a system with Tcl / EnvironmentModules? I'm not sure how EESSI it would be to generalize this for other module systems?

Should we / can we leverage ReFrame's module system? I think that already supports different module implementations?

should indeed generalize it, i'll see what we can do.

hm no, all modules in EESSI are in Lua format, so EESSI requires Lmod.

casparvl

Haven't tested yet. Changes look good in general. Some small/medium sized concerns.

casparvl · 2026-11-09T11:38:56Z

Unrelated thought: we don't really have unit tests for the test suite. This makes me wonder if we should. I.e. something that calls the common_config with various site_configurations set, and see that hte output is 'as intended'... It's quite complex logic, that'll break easily in future changes if we don't test it...

smoors · 2026-11-09T14:24:19Z

Unrelated thought: we don't really have unit tests for the test suite. This makes me wonder if we should. I.e. something that calls the common_config with various site_configurations set, and see that hte output is 'as intended'... It's quite complex logic, that'll break easily in future changes if we don't test it...

i agree we should have a test suite for the test suite :)
i already have a python script that tests various things in this PR, but let's do that in a follow-up?

casparvl · 2026-11-09T15:11:20Z

I mean, we can think about what it should cover. But I'm mainly thinking about the machinery around the actual tests. I don't think we can (should) test the test classes themselves (other than maybe a --dry-run based test).

That being said, https://github.com/EESSI/test-suite/actions/runs/42527362655/job/128873824331?pr=362 essenitally does a dry-run, but with VERY old EESSI version. I guess we should update that...

…lmod; update info msg

Samuel Moors added 3 commits October 28, 2026 20:55

don't hardcode EESSI version environments

61602b3

warn instead of error if no EESSI versions are found

09be518

fix gh workflow

075dba8

casparvl reviewed Nov 5, 2026

View reviewed changes

add add_eessi_versions param

68c4669

smoors requested a review from casparvl November 8, 2026 10:35

Samuel Moors added 5 commits November 8, 2026 15:46

update

2a16faf

add info and warning messages

266486c

fixes

4eed108

add NO_EESSI marker

1e9e5e3

fix

a7177ba

casparvl reviewed Nov 9, 2026

View reviewed changes

Comment thread eessi/testsuite/common_config.py Resolved

casparvl reviewed Nov 9, 2026

View reviewed changes

casparvl requested changes Nov 9, 2026

View reviewed changes

casparvl reviewed Nov 9, 2026

View reviewed changes

Comment thread eessi/testsuite/utils.py Resolved

use reframe machinery for lmod; convert error to warning for missing …

7bd05a3

…lmod; update info msg

smoors requested a review from casparvl November 9, 2026 16:15

Merge branch 'main' into eessi-vers

a2dfbef

Conversation

smoors commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

casparvl left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smoors commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

casparvl commented Nov 5, 2026

Uh oh!

smoors commented Nov 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

casparvl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

casparvl commented Nov 9, 2026

Uh oh!

smoors commented Nov 9, 2026

Uh oh!

casparvl commented Nov 9, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

smoors commented Oct 28, 2026 •

edited

Loading

casparvl left a comment •

edited

Loading

smoors commented Nov 5, 2026 •

edited

Loading

smoors commented Nov 8, 2026 •

edited

Loading