(Closes #3461) duplicated script names

changelog

@@ -1,3 +1,6 @@
+   42) PR #3468 for #3461. Cleans up the use of sys.path so that
+   different transformation scripts with the same name can be imported.
+
    41) PR #3573. Improves Developer Guide description of coding style
    and fixes a few incorrect type-hints.
 

doc/user_guide/user_scripts.rst

@@ -13,23 +13,22 @@ PSyclone User Scripts
 The standard way to transform a codebase using psyclone is through the
 :ref:`psyclone_command` tool, which has an optional ``-s <SCRIPT_NAME>``
 flag that allows users to specify a transformation user script to
-programmatically modify the input code::
+programmatically modify the input code using a relative or absolute
+path::
 
     > psyclone -s optimise.py input_source.f90
-
-In this case, the current directory is prepended to the Python search path
-``PYTHONPATH`` which will then be used to try to find the script file. Thus,
-the search begins in the current directory and continues over any pre-existing
-directories in the search path, failing if the file cannot be found.
-
-Alternatively, script files may be specified with a path. In this case
-the file must exist in the specified location. This location is then added to
-the Python search path ``PYTHONPATH`` as before. For example::
-
     > psyclone -s ./optimise.py input_source.f90
     > psyclone -s ../scripts/optimise.py input_source.f90
     > psyclone -s /home/me/PSyclone/scripts/optimise.py input_source.f90
 
+PSyclone will not take ``PYTHONPATH`` into account when importing the script,
+so the user must ensure that the script is found: for this, it must either be
+in the current working directory (if no path is specified at all), or be
+specified with a valid (relative or absolute) path. PSyclone will add the
+directory of the script to Python's ``sys.path``, meaning the transformation
+script can import any helper script in the same directory without additional
+setup.
+
 A valid PSyclone user script file must contain a ``trans`` function which accepts
 a :ref:`PSyIR node<psyir-ug>` representing the root of the psy-layer
 code (as a FileContainer):

src/psyclone/generator.py

@@ -18,6 +18,7 @@
 import importlib
 import logging
 import os
+import pathlib
 import shutil
 import sys
 import traceback
@@ -90,9 +91,16 @@ def load_script(
            List[str],
            Union[bool, List[str]],
            dict[str, str]]:
-    ''' Loads the specified script containing a psyclone recipe. We also
-    prepend the script path to the sys.path, so that the script itself and
-    any imports that it has from the same directory can be found.
+    ''' Loads the specified script containing a PSyclone recipe. This is done
+    without adding the imported symbol to the system list of all Python
+    modules. This will allow later to import a potentially different script
+    with the same name. The script path is also prepended to sys.path, so that
+    the script can import helper scripts from its own directory.
+
+    TODO #3514
+    To avoid that sys.path keeps growing (if PSyclone is called more than
+    once), the caller must remove the first entry of sys.path after executing
+    the script.
 
     :param script_name: name of the script to load.
     :param kwargs_str: the kwargs argument from the command line.
@@ -131,12 +139,22 @@ def load_script(
         raise GenerationError(
             f"generator: expected the script file '{filename}' to have "
             f"the '.py' extension")
-    # prepend file path - if none, the empty string equates to the current
-    # working directory - to the system path to guarantee we find the user
-    # provided module instead of a similarly named module that might
-    # already exist elsewhere in the system path
+
+    # Add the script directory to sys.path, so scripts can easily import
+    # helper scripts in the same directory (this step is not needed to
+    # import the script itself, but it maintains backwards compatibility).
     sys.path.insert(0, filepath)
-    recipe_module = importlib.import_module(module_name)
+
+    # This will import the module, but not make it part of the
+    # system list of all modules, i.e. a module of this name can be
+    # imported and used elsewhere. This will allow to import a module
+    # with the same name, which is required if PSyclone is called
+    # repeatedly from the same Python process (and each call might
+    # have different scripts with the same name, e.g. local.py in LFRic).
+    script_path = pathlib.Path(script_name)
+    spec = importlib.util.spec_from_file_location(module_name, script_path)
+    recipe_module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(recipe_module)
 
     files_to_skip: list[str]
     if hasattr(recipe_module, "FILES_TO_SKIP"):
@@ -266,6 +284,8 @@ def generate(filename: str,
             # defined, otherwise an exception is raised.
             trans_func, _, _, kwargs = load_script(script_name, kwargs_str)
             trans_func(psy.container.root, **kwargs)
+            # TODO #3514: proper cleanup using with
+            del sys.path[0]
         alg_gen = None
 
     elif api in GOCEAN_API_NAMES or (api in LFRIC_API_NAMES and LFRIC_TESTING):
@@ -319,6 +339,8 @@ def generate(filename: str,
                                                "trans_alg", is_optional=True)
             if recipe:
                 recipe(psyir, **kwargs)
+            # TODO #3514: proper cleanup using with
+            del sys.path[0]
 
         # For each kernel called from the algorithm layer
         kernels: dict[int, dict[int, Node]] = {}
@@ -411,6 +433,8 @@ def generate(filename: str,
             # raised.
             trans_func, _, _, kwargs = load_script(script_name, kwargs_str)
             trans_func(psy.container.root, **kwargs)
+            # TODO #3514: proper cleanup using with
+            del sys.path[0]
 
     # TODO issue #1618 remove Alg class and tests from PSyclone
     if api in LFRIC_API_NAMES and not LFRIC_TESTING:
@@ -980,3 +1004,6 @@ def code_transformation_mode(input_file: str,
         else:
             print(f"File '{input_file}' skipped because it is listed in "
                   "FILES_TO_SKIP.", file=sys.stdout)
+
+    # TODO #3514: proper cleanup using with
+    del sys.path[0]