Auto3Dseg Overly Slow & Raises Error

[bilalcodehub]

Dear All,

I am encountering an issue where Auto3Dseg appears to be running very slow. I think it is not utilising the GPU on our server. I've been working with the introductory tutorials code using some .mha files for multi-organ segmentation, and while everything else functions correctly, I run into problems when executing the following segment:

history = import_bundle_algo_history(work_dir, only_trained=False)
for algo_dict in history:
    algo = algo_dict[AlgoKeys.ALGO]
    algo.train(train_param)  # can use default params by `algo.train()`
    acc = algo.get_score()
    algo_to_pickle(algo, template_path=algo.template_path, best_metric=acc)

This results in an error after a prolonged period of running. Notably, I am using PyTorch version 2.3.0+cu118 and torch.cuda.is_available() returns True. However, the nvidia-smi command does not show any GPU activity despite the system having 2 NVIDIA RTX 2090 Ti GPUs.

2024-05-11 20:44:34,769 - INFO - The keys num_warmup_epochs cannot be found in the work_dir/dints_0/configs/hyper_parameters.yaml for training. Skipped overriding key num_warmup_epochs.
2024-05-11 20:44:34,771 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '2', 'work_dir/dints_0/scripts/train.py', 'run', "--config_file='work_dir/dints_0/configs/hyper_parameters.yaml,work_dir/dints_0/configs/hyper_parameters_search.yaml,work_dir/dints_0/configs/network.yaml,work_dir/dints_0/configs/network_search.yaml,work_dir/dints_0/configs/transforms_infer.yaml,work_dir/dints_0/configs/transforms_train.yaml,work_dir/dints_0/configs/transforms_validate.yaml'", '--training#num_epochs_per_validation=1', '--training#num_images_per_batch=2']
W0511 20:44:35.604000 140672643610432 torch/distributed/run.py:757] 
W0511 20:44:35.604000 140672643610432 torch/distributed/run.py:757] *****************************************
W0511 20:44:35.604000 140672643610432 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0511 20:44:35.604000 140672643610432 torch/distributed/run.py:757] *****************************************
torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
monai.transforms.croppad.dictionary CropForegroundd.__init__:allow_smaller: Current default value of argument `allow_smaller=True` has been deprecated since version 1.2. It will be changed to `allow_smaller=False` in version 1.5.
monai.transforms.croppad.dictionary CropForegroundd.__init__:allow_smaller: Current default value of argument `allow_smaller=True` has been deprecated since version 1.2. It will be changed to `allow_smaller=False` in version 1.5.
dints_0 - training ...:   1%|          | 9/1128 [33:48<70:03:20, 225.38s/round]
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/exsdatalab/mlworks/aortaseg24/work_dir/dints_0/scripts/train.py", line 984, in <module>
[rank0]:     fire.Fire()
[rank0]:   File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
[rank0]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank0]:   File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
[rank0]:     component, remaining_args = _CallAndUpdateTrace(
[rank0]:   File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
[rank0]:     component = fn(*varargs, **kwargs)
[rank0]:   File "/home/exsdatalab/mlworks/aortaseg24/work_dir/dints_0/scripts/train.py", line 715, in run
[rank0]:     val_outputs = val_outputs.cpu()
[rank0]: UnboundLocalError: local variable 'val_outputs' referenced before assignment
W0511 21:19:17.632000 140672643610432 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 10001 closing signal SIGTERM
E0511 21:19:18.749000 140672643610432 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 10000) of binary: /home/exsdatalab/mambaforge/bin/python3.10
Traceback (most recent call last):
  File "/home/exsdatalab/mambaforge/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
work_dir/dints_0/scripts/train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-05-11_21:19:17
  host      : exsdatalab
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 10000)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
Cell In[12], line 4
      2 for algo_dict in history:
      3     algo = algo_dict[AlgoKeys.ALGO]
----> 4     algo.train(train_param)  # can use default params by `algo.train()`
      5     acc = algo.get_score()
      6     algo_to_pickle(algo, template_path=algo.template_path, best_metric=acc)

File ~/mlworks/aortaseg24/work_dir/algorithm_templates/dints/scripts/algo.py:490, in DintsAlgo.train(self, train_params, device_setting, search)
    488 cmd, devices_info = self._create_cmd(dints_train_params)
    489 cmd = "OMP_NUM_THREADS=1 " + cmd
--> 490 return self._run_cmd(cmd, devices_info)

File ~/mambaforge/lib/python3.10/site-packages/monai/apps/auto3dseg/bundle_gen.py:251, in BundleAlgo._run_cmd(self, cmd, devices_info)
    249     return _run_cmd_bcprun(cmd, n=self.device_setting["NUM_NODES"], p=self.device_setting["n_devices"])
    250 elif int(self.device_setting["n_devices"]) > 1:
--> 251     return _run_cmd_torchrun(
    252         cmd, nnodes=1, nproc_per_node=self.device_setting["n_devices"], env=ps_environ, check=True
    253     )
    254 else:
    255     return run_cmd(cmd.split(), run_cmd_verbose=True, env=ps_environ, check=True)

File ~/mambaforge/lib/python3.10/site-packages/monai/auto3dseg/utils.py:502, in _run_cmd_torchrun(cmd, **kwargs)
    500     torchrun_list += [f"--{arg}", str(params.pop(arg))]
    501 torchrun_list += cmd_list
--> 502 return run_cmd(torchrun_list, run_cmd_verbose=True, **params)

File ~/mambaforge/lib/python3.10/site-packages/monai/utils/misc.py:874, in run_cmd(cmd_list, **kwargs)
    872     monai.apps.utils.get_logger("run_cmd").info(f"{cmd_list}")
    873 try:
--> 874     return subprocess.run(cmd_list, **kwargs)
    875 except subprocess.CalledProcessError as e:
    876     if not debug:

File ~/mambaforge/lib/python3.10/subprocess.py:526, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    524     retcode = process.poll()
    525     if check and retcode:
--> 526         raise CalledProcessError(retcode, process.args,
    527                                  output=stdout, stderr=stderr)
    528 return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['torchrun', '--nnodes', '1', '--nproc_per_node', '2', 'work_dir/dints_0/scripts/train.py', 'run', "--config_file='work_dir/dints_0/configs/hyper_parameters.yaml,work_dir/dints_0/configs/hyper_parameters_search.yaml,work_dir/dints_0/configs/network.yaml,work_dir/dints_0/configs/network_search.yaml,work_dir/dints_0/configs/transforms_infer.yaml,work_dir/dints_0/configs/transforms_train.yaml,work_dir/dints_0/configs/transforms_validate.yaml'", '--training#num_epochs_per_validation=1', '--training#num_images_per_batch=2']' returned non-zero exit status 1.

Could anyone provide insights or suggestions on how to resolve this issue?

Thank you for your assistance.

Best regards,
Bilal

[KumoLiu]

Hi @dongyang0122, could you please help take a look at this issue, thanks.

[diazandr3s]

Dear All,

I am encountering an issue where Auto3Dseg appears to be running very slow. I think it is not utilising the GPU on our server. I've been working with the introductory tutorials code using some .mha files for multi-organ segmentation, and while everything else functions correctly, I run into problems when executing the following segment:

history = import_bundle_algo_history(work_dir, only_trained=False)
for algo_dict in history:
    algo = algo_dict[AlgoKeys.ALGO]
    algo.train(train_param)  # can use default params by `algo.train()`
    acc = algo.get_score()
    algo_to_pickle(algo, template_path=algo.template_path, best_metric=acc)

This results in an error after a prolonged period of running. Notably, I am using PyTorch version 2.3.0+cu118 and torch.cuda.is_available() returns True. However, the nvidia-smi command does not show any GPU activity despite the system having 2 NVIDIA RTX 2090 Ti GPUs.

2024-05-11 20:44:34,769 - INFO - The keys num_warmup_epochs cannot be found in the work_dir/dints_0/configs/hyper_parameters.yaml for training. Skipped overriding key num_warmup_epochs.
2024-05-11 20:44:34,771 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '2', 'work_dir/dints_0/scripts/train.py', 'run', "--config_file='work_dir/dints_0/configs/hyper_parameters.yaml,work_dir/dints_0/configs/hyper_parameters_search.yaml,work_dir/dints_0/configs/network.yaml,work_dir/dints_0/configs/network_search.yaml,work_dir/dints_0/configs/transforms_infer.yaml,work_dir/dints_0/configs/transforms_train.yaml,work_dir/dints_0/configs/transforms_validate.yaml'", '--training#num_epochs_per_validation=1', '--training#num_images_per_batch=2']
W0511 20:44:35.604000 140672643610432 torch/distributed/run.py:757] 
W0511 20:44:35.604000 140672643610432 torch/distributed/run.py:757] *****************************************
W0511 20:44:35.604000 140672643610432 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0511 20:44:35.604000 140672643610432 torch/distributed/run.py:757] *****************************************
torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
monai.transforms.croppad.dictionary CropForegroundd.__init__:allow_smaller: Current default value of argument `allow_smaller=True` has been deprecated since version 1.2. It will be changed to `allow_smaller=False` in version 1.5.
monai.transforms.croppad.dictionary CropForegroundd.__init__:allow_smaller: Current default value of argument `allow_smaller=True` has been deprecated since version 1.2. It will be changed to `allow_smaller=False` in version 1.5.
dints_0 - training ...:   1%|          | 9/1128 [33:48<70:03:20, 225.38s/round]
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/exsdatalab/mlworks/aortaseg24/work_dir/dints_0/scripts/train.py", line 984, in <module>
[rank0]:     fire.Fire()
[rank0]:   File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
[rank0]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank0]:   File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
[rank0]:     component, remaining_args = _CallAndUpdateTrace(
[rank0]:   File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
[rank0]:     component = fn(*varargs, **kwargs)
[rank0]:   File "/home/exsdatalab/mlworks/aortaseg24/work_dir/dints_0/scripts/train.py", line 715, in run
[rank0]:     val_outputs = val_outputs.cpu()
[rank0]: UnboundLocalError: local variable 'val_outputs' referenced before assignment
W0511 21:19:17.632000 140672643610432 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 10001 closing signal SIGTERM
E0511 21:19:18.749000 140672643610432 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 10000) of binary: /home/exsdatalab/mambaforge/bin/python3.10
Traceback (most recent call last):
  File "/home/exsdatalab/mambaforge/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
work_dir/dints_0/scripts/train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-05-11_21:19:17
  host      : exsdatalab
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 10000)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
Cell In[12], line 4
      2 for algo_dict in history:
      3     algo = algo_dict[AlgoKeys.ALGO]
----> 4     algo.train(train_param)  # can use default params by `algo.train()`
      5     acc = algo.get_score()
      6     algo_to_pickle(algo, template_path=algo.template_path, best_metric=acc)

File ~/mlworks/aortaseg24/work_dir/algorithm_templates/dints/scripts/algo.py:490, in DintsAlgo.train(self, train_params, device_setting, search)
    488 cmd, devices_info = self._create_cmd(dints_train_params)
    489 cmd = "OMP_NUM_THREADS=1 " + cmd
--> 490 return self._run_cmd(cmd, devices_info)

File ~/mambaforge/lib/python3.10/site-packages/monai/apps/auto3dseg/bundle_gen.py:251, in BundleAlgo._run_cmd(self, cmd, devices_info)
    249     return _run_cmd_bcprun(cmd, n=self.device_setting["NUM_NODES"], p=self.device_setting["n_devices"])
    250 elif int(self.device_setting["n_devices"]) > 1:
--> 251     return _run_cmd_torchrun(
    252         cmd, nnodes=1, nproc_per_node=self.device_setting["n_devices"], env=ps_environ, check=True
    253     )
    254 else:
    255     return run_cmd(cmd.split(), run_cmd_verbose=True, env=ps_environ, check=True)

File ~/mambaforge/lib/python3.10/site-packages/monai/auto3dseg/utils.py:502, in _run_cmd_torchrun(cmd, **kwargs)
    500     torchrun_list += [f"--{arg}", str(params.pop(arg))]
    501 torchrun_list += cmd_list
--> 502 return run_cmd(torchrun_list, run_cmd_verbose=True, **params)

File ~/mambaforge/lib/python3.10/site-packages/monai/utils/misc.py:874, in run_cmd(cmd_list, **kwargs)
    872     monai.apps.utils.get_logger("run_cmd").info(f"{cmd_list}")
    873 try:
--> 874     return subprocess.run(cmd_list, **kwargs)
    875 except subprocess.CalledProcessError as e:
    876     if not debug:

File ~/mambaforge/lib/python3.10/subprocess.py:526, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    524     retcode = process.poll()
    525     if check and retcode:
--> 526         raise CalledProcessError(retcode, process.args,
    527                                  output=stdout, stderr=stderr)
    528 return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['torchrun', '--nnodes', '1', '--nproc_per_node', '2', 'work_dir/dints_0/scripts/train.py', 'run', "--config_file='work_dir/dints_0/configs/hyper_parameters.yaml,work_dir/dints_0/configs/hyper_parameters_search.yaml,work_dir/dints_0/configs/network.yaml,work_dir/dints_0/configs/network_search.yaml,work_dir/dints_0/configs/transforms_infer.yaml,work_dir/dints_0/configs/transforms_train.yaml,work_dir/dints_0/configs/transforms_validate.yaml'", '--training#num_epochs_per_validation=1', '--training#num_images_per_batch=2']' returned non-zero exit status 1.

Could anyone provide insights or suggestions on how to resolve this issue?

Thank you for your assistance.

Best regards, Bilal

Hi @bilalcodehub,

I'd suggest you first try the SegResNet architecture.

python -m monai.apps.auto3dseg AutoRunner run --input ./my_task.yaml --algos segresnet --work_dir ./outputs/output_my_task

Here is more information: https://github.com/lassoan/SlicerMONAIAuto3DSeg/?tab=readme-ov-file#how-to-train-a-new-model-from-scratch

[bilalcodehub]

Hi @diazandr3s,

Thank you for your guidance. I've successfully started training the models on the server. The first model trains without any issues. However, I encounter an error with the second model. Here is the error message I receive:

base) exsdatalab@exsdatalab:~/mlworks/aortaseg24$ python -m monai.apps.auto3dseg AutoRunner run --input ./work_dir/input.yaml --algos segresnet --work_dir ./work_dir --num_workers 2 --batch_size 4
2024-05-17 10:42:45,124 - INFO - AutoRunner using work directory ./work_dir
2024-05-17 10:42:45,125 - INFO - Loading input config ./work_dir/input.yaml
2024-05-17 10:42:45,125 - INFO - Setting num_fold 2 based on the input datalist /home/exsdatalab/mlworks/aortaseg24/work_dir/datalist.json.
2024-05-17 10:42:45,143 - INFO - Using user defined command running prefix , will override other settings
2024-05-17 10:42:45,143 - INFO - Skipping data analysis...
2024-05-17 10:42:45,143 - INFO - Skipping algorithm generation...
2024-05-17 10:42:45,158 - INFO - Skipping already trained algos ['segresnet_0'].Set option train=True to always retrain all algos.
2024-05-17 10:42:45,158 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '2', '/home/exsdatalab/mlworks/aortaseg24/work_dir/segresnet_1/scripts/train.py', 'run', "--config_file='/home/exsdatalab/mlworks/aortaseg24/work_dir/segresnet_1/configs/hyper_parameters.yaml'"]
W0517 10:42:46.019000 140101041149760 torch/distributed/run.py:757] 
W0517 10:42:46.019000 140101041149760 torch/distributed/run.py:757] *****************************************
W0517 10:42:46.019000 140101041149760 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0517 10:42:46.019000 140101041149760 torch/distributed/run.py:757] *****************************************
segresnet_1 - training:   1%|▉                                                                                                                                 | 9/1250 [01:39<2:41:02,  7.79s/epoch]2024-05-17 10:45:11,068 - INFO - CUDA out of memory. Tried to allocate 11.78 GiB. GPU  has a total capacity of 23.70 GiB of which 10.04 GiB is free. Including non-PyTorch memory, this process has 13.64 GiB memory in use. Of the allocated memory 10.40 GiB is allocated by PyTorch, and 1.52 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-05-17 10:45:11,068 - WARNING - GPU stitching failed, buffer 1 dim -1, image dim torch.Size([1, 1, 453, 416, 699]).
2024-05-17 10:45:36,097 - INFO - CUDA out of memory. Tried to allocate 7.98 GiB. GPU 
2024-05-17 10:45:36,097 - WARNING - GPU stitching failed, buffer 1 dim -1, image dim torch.Size([1, 1, 389, 329, 697]).
2024-05-17 10:49:03,757 - INFO - CUDA out of memory. Tried to allocate 442.00 MiB. GPU  has a total capacity of 23.70 GiB of which 330.81 MiB is free. Including non-PyTorch memory, this process has 23.35 GiB memory in use. Of the allocated memory 21.04 GiB is allocated by PyTorch, and 603.65 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-05-17 10:49:03,757 - WARNING - GPU stitching failed, buffer 1 dim -1, image dim torch.Size([1, 1, 449, 351, 613]).
segresnet_1 - training: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1250/1250 [28:21:36<00:00, 81.68s/epoch]
2024-05-18 15:04:33,565 - INFO - Ensembling using 2 GPU!
2024-05-18 15:04:33,565 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '2', '-m', 'monai.apps.auto3dseg', 'EnsembleRunner', 'ensemble', '--data_src_cfg_name', '/home/exsdatalab/mlworks/aortaseg24/work_dir/input.yaml', '--work_dir', '/home/exsdatalab/mlworks/aortaseg24/work_dir', '--num_fold', '2', '--ensemble_method_name', 'AlgoEnsembleBestByFold', '--mgpu', 'True', '--num_workers=2', '--batch_size=4']
W0518 15:04:34.423000 140082965288768 torch/distributed/run.py:757] 
W0518 15:04:34.423000 140082965288768 torch/distributed/run.py:757] *****************************************
W0518 15:04:34.423000 140082965288768 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0518 15:04:34.423000 140082965288768 torch/distributed/run.py:757] *****************************************
[W CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
[W CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
2024-05-18 15:04:37,707 - INFO - The output_dir is not specified. /home/exsdatalab/mlworks/aortaseg24/work_dir/ensemble_output will be used to save ensemble predictions.
2024-05-18 15:04:37,707 - INFO - Directory /home/exsdatalab/mlworks/aortaseg24/work_dir/ensemble_output is created to save ensemble predictions
2024-05-18 15:04:37,760 - INFO - Auto3Dseg picked the following networks to ensemble:
2024-05-18 15:04:37,760 - INFO - segresnet_0
2024-05-18 15:04:37,760 - INFO - segresnet_1
2024-05-18 15:04:37,760 - INFO - Auto3Dseg ensemble prediction outputs will be saved in /home/exsdatalab/mlworks/aortaseg24/work_dir/ensemble_output.
Ensembling (rank 0)...:  20%|█████████████████████████████████▏                                                                                                                                    | 1/5 [01:47<07:11, 107.76s/it]W0518 15:09:49.824000 140082965288768 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 153697 closing signal SIGTERM
E0518 15:09:51.473000 140082965288768 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: -9) local_rank: 1 (pid: 153698) of binary: /home/exsdatalab/mambaforge/bin/python3.10
Traceback (most recent call last):
  File "/home/exsdatalab/mambaforge/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
=======================================================
monai.apps.auto3dseg FAILED
-------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
-------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-05-18_15:09:49
  host      : exsdatalab
  rank      : 1 (local_rank: 1)
  exitcode  : -9 (pid: 153698)
  error_file: <N/A>
  traceback : Signal 9 (SIGKILL) received by PID 153698
=======================================================
Traceback (most recent call last):
  File "/home/exsdatalab/mambaforge/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/exsdatalab/mambaforge/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/apps/auto3dseg/__main__.py", line 24, in <module>
    fire.Fire(
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/apps/auto3dseg/auto_runner.py", line 825, in run
    ensemble_runner.run(self.device_setting)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/apps/auto3dseg/ensemble_builder.py", line 621, in run
    self._create_cmd()
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/apps/auto3dseg/ensemble_builder.py", line 657, in _create_cmd
    _run_cmd_torchrun(
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/auto3dseg/utils.py", line 502, in _run_cmd_torchrun
    return run_cmd(torchrun_list, run_cmd_verbose=True, **params)
  File "/home/exsdatalab/mambaforge/lib/python3.10/site-packages/monai/utils/misc.py", line 874, in run_cmd
    return subprocess.run(cmd_list, **kwargs)
  File "/home/exsdatalab/mambaforge/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['torchrun', '--nnodes', '1', '--nproc_per_node', '2', '-m', 'monai.apps.auto3dseg', 'EnsembleRunner', 'ensemble', '--data_src_cfg_name', '/home/exsdatalab/mlworks/aortaseg24/work_dir/input.yaml', '--work_dir', '/home/exsdatalab/mlworks/aortaseg24/work_dir', '--num_fold', '2', '--ensemble_method_name', 'AlgoEnsembleBestByFold', '--mgpu', 'True', '--num_workers=2', '--batch_size=4']' returned non-zero exit status 1.

Could you please help me understand what might be causing this issue?

Thank you for your time and assistance.
Kind regards,
Bilal

[diazandr3s]

Hi @bilalcodehub,

This is strange. From the logs, I see that the first model (segresnet_0) was skipped and not trained.
The Auto3DSeg only trained the second model segresnet_1
The good news is that you have one model. Which most of the times is good enough. You could use that model to run inference on your unlabelled dataset.

Using the segresnet_1/model/model.pt file, you can follow these instructions to test the model.

Let us know how that goes

[soumen02]

Hi all,

Just wanted to add to the thread.
Facing the same error, when I try to run the auto_runner tutorial. Mine throws an error when training swinunetr.

2024-06-02 11:35:23,491 - INFO - AutoRunner using work directory ./work_dir
2024-06-02 11:35:23,492 - INFO - Loading input config ./input.yaml
2024-06-02 11:35:23,493 - INFO - Datalist was copied to work_dir: /home/mri/laminate/auto3dseg/notebooks/work_dir/msd_task04_hippocampus_folds.json
2024-06-02 11:35:23,493 - INFO - Setting num_fold 5 based on the input datalist /home/mri/laminate/auto3dseg/notebooks/work_dir/msd_task04_hippocampus_folds.json.
2024-06-02 11:35:23,495 - INFO - Using user defined command running prefix , will override other settings
2024-06-02 11:35:23,495 - INFO - Running data analysis...
2024-06-02 11:35:23,495 - INFO - Found 4 GPUs for data analyzing!
100%|██████████| 8/8 [00:04<00:00,  1.73it/s]
2024-06-02 11:35:37,059 - INFO - Writing data stats to /home/mri/laminate/auto3dseg/notebooks/work_dir/datastats.yaml.
2024-06-02 11:35:37,071 - INFO - Writing by-case data stats to /home/mri/laminate/auto3dseg/notebooks/work_dir/datastats_by_case.yaml, this may take a while.
2024-06-02 11:35:37,339 - INFO - BundleGen from https://github.com/Project-MONAI/research-contributions/releases/download/algo_templates/249bf4b.tar.gz
algo_templates.tar.gz: 104kB [00:00, 128kB/s]                              2024-06-02 11:35:38,173 - INFO - Downloaded: /tmp/tmpelio0vzb/algo_templates.tar.gz
2024-06-02 11:35:38,173 - INFO - Expected md5 is None, skip md5 check for file /tmp/tmpelio0vzb/algo_templates.tar.gz.
2024-06-02 11:35:38,173 - INFO - Writing into directory: /home/mri/laminate/auto3dseg/notebooks/work_dir.
2024-06-02 11:35:38,316 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_0
2024-06-02 11:35:38,381 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_1
2024-06-02 11:35:38,446 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_2
2024-06-02 11:35:38,511 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_3
2024-06-02 11:35:38,576 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_4
2024-06-02 11:35:39,277 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_0
2024-06-02 11:35:39,315 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_1
2024-06-02 11:35:39,352 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_2
2024-06-02 11:35:39,389 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_3
2024-06-02 11:35:39,426 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_4
2024-06-02 11:35:39,450 - INFO - segresnet2d_0 is skipped! SegresNet2D is skipped due to median spacing of [1.0, 1.0, 1.0],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-06-02 11:35:39,468 - INFO - segresnet2d_1 is skipped! SegresNet2D is skipped due to median spacing of [1.0, 1.0, 1.0],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-06-02 11:35:39,485 - INFO - segresnet2d_2 is skipped! SegresNet2D is skipped due to median spacing of [1.0, 1.0, 1.0],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-06-02 11:35:39,502 - INFO - segresnet2d_3 is skipped! SegresNet2D is skipped due to median spacing of [1.0, 1.0, 1.0],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-06-02 11:35:39,519 - INFO - segresnet2d_4 is skipped! SegresNet2D is skipped due to median spacing of [1.0, 1.0, 1.0],  which means the dataset is not highly anisotropic, e.g. spacing[2] < 3*(spacing[0] + spacing[1])/2) .
2024-06-02 11:35:40,305 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0
2024-06-02 11:35:40,354 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_1
2024-06-02 11:35:40,403 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_2
2024-06-02 11:35:40,452 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_3
2024-06-02 11:35:40,500 - INFO - Generated:/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_4
2024-06-02 11:35:40,623 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '4', '/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_0/scripts/train.py', 'run', "--config_file='/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_0/configs/hyper_parameters.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_0/configs/hyper_parameters_search.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_0/configs/network.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_0/configs/network_search.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_0/configs/transforms_infer.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_0/configs/transforms_train.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_0/configs/transforms_validate.yaml'"]
W0602 11:35:41.560926 139642697933824 torch/distributed/run.py:757] 
W0602 11:35:41.560926 139642697933824 torch/distributed/run.py:757] *****************************************
W0602 11:35:41.560926 139642697933824 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0602 11:35:41.560926 139642697933824 torch/distributed/run.py:757] *****************************************
dints_0 - training ...: 100%|██████████| 7/7 [03:26<00:00, 29.49s/round]
dints_0 - validation at original spacing/resolution
2024-06-02 11:39:18,066 - WARNING - dints_0 - training: finished
2024-06-02 11:39:22,017 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '4', '/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_1/scripts/train.py', 'run', "--config_file='/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_1/configs/hyper_parameters.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_1/configs/hyper_parameters_search.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_1/configs/network.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_1/configs/network_search.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_1/configs/transforms_infer.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_1/configs/transforms_train.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_1/configs/transforms_validate.yaml'"]
W0602 11:39:22.932046 140273598444544 torch/distributed/run.py:757] 
W0602 11:39:22.932046 140273598444544 torch/distributed/run.py:757] *****************************************
W0602 11:39:22.932046 140273598444544 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0602 11:39:22.932046 140273598444544 torch/distributed/run.py:757] *****************************************
dints_1 - training ...: 100%|██████████| 7/7 [03:27<00:00, 29.59s/round]
dints_1 - validation at original spacing/resolution
2024-06-02 11:43:00,261 - WARNING - dints_1 - training: finished
2024-06-02 11:43:03,428 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '4', '/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_2/scripts/train.py', 'run', "--config_file='/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_2/configs/hyper_parameters.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_2/configs/hyper_parameters_search.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_2/configs/network.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_2/configs/network_search.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_2/configs/transforms_infer.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_2/configs/transforms_train.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_2/configs/transforms_validate.yaml'"]
W0602 11:43:04.338885 140082210231296 torch/distributed/run.py:757] 
W0602 11:43:04.338885 140082210231296 torch/distributed/run.py:757] *****************************************
W0602 11:43:04.338885 140082210231296 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0602 11:43:04.338885 140082210231296 torch/distributed/run.py:757] *****************************************
dints_2 - training ...: 100%|██████████| 7/7 [03:27<00:00, 29.62s/round]
dints_2 - validation at original spacing/resolution
2024-06-02 11:46:41,806 - WARNING - dints_2 - training: finished
2024-06-02 11:46:44,839 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '4', '/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_3/scripts/train.py', 'run', "--config_file='/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_3/configs/hyper_parameters.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_3/configs/hyper_parameters_search.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_3/configs/network.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_3/configs/network_search.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_3/configs/transforms_infer.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_3/configs/transforms_train.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_3/configs/transforms_validate.yaml'"]
W0602 11:46:45.755149 139753312572416 torch/distributed/run.py:757] 
W0602 11:46:45.755149 139753312572416 torch/distributed/run.py:757] *****************************************
W0602 11:46:45.755149 139753312572416 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0602 11:46:45.755149 139753312572416 torch/distributed/run.py:757] *****************************************
dints_3 - training ...: 100%|██████████| 7/7 [03:26<00:00, 29.48s/round]
dints_3 - validation at original spacing/resolution
2024-06-02 11:50:22,302 - WARNING - dints_3 - training: finished
2024-06-02 11:50:26,254 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '4', '/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_4/scripts/train.py', 'run', "--config_file='/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_4/configs/hyper_parameters.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_4/configs/hyper_parameters_search.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_4/configs/network.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_4/configs/network_search.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_4/configs/transforms_infer.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_4/configs/transforms_train.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/dints_4/configs/transforms_validate.yaml'"]
W0602 11:50:27.172852 140150688244736 torch/distributed/run.py:757] 
W0602 11:50:27.172852 140150688244736 torch/distributed/run.py:757] *****************************************
W0602 11:50:27.172852 140150688244736 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0602 11:50:27.172852 140150688244736 torch/distributed/run.py:757] *****************************************
dints_4 - training ...: 100%|██████████| 7/7 [03:27<00:00, 29.58s/round]
dints_4 - validation at original spacing/resolution
2024-06-02 11:54:04,227 - WARNING - dints_4 - training: finished
2024-06-02 11:54:07,669 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '4', '/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_0/scripts/train.py', 'run', "--config_file='/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_0/configs/hyper_parameters.yaml'"]
W0602 11:54:08.589611 139806660957184 torch/distributed/run.py:757] 
W0602 11:54:08.589611 139806660957184 torch/distributed/run.py:757] *****************************************
W0602 11:54:08.589611 139806660957184 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0602 11:54:08.589611 139806660957184 torch/distributed/run.py:757] *****************************************
segresnet_0 - training: 100%|██████████| 1250/1250 [10:05<00:00,  2.06epoch/s]
2024-06-02 12:04:24,477 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '4', '/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_1/scripts/train.py', 'run', "--config_file='/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_1/configs/hyper_parameters.yaml'"]
W0602 12:04:25.388635 140487885845504 torch/distributed/run.py:757] 
W0602 12:04:25.388635 140487885845504 torch/distributed/run.py:757] *****************************************
W0602 12:04:25.388635 140487885845504 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0602 12:04:25.388635 140487885845504 torch/distributed/run.py:757] *****************************************
segresnet_1 - training: 100%|██████████| 1250/1250 [09:47<00:00,  2.13epoch/s]
2024-06-02 12:14:26,271 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '4', '/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_2/scripts/train.py', 'run', "--config_file='/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_2/configs/hyper_parameters.yaml'"]
W0602 12:14:27.192992 139812239975424 torch/distributed/run.py:757] 
W0602 12:14:27.192992 139812239975424 torch/distributed/run.py:757] *****************************************
W0602 12:14:27.192992 139812239975424 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0602 12:14:27.192992 139812239975424 torch/distributed/run.py:757] *****************************************
segresnet_2 - training: 100%|██████████| 1250/1250 [09:51<00:00,  2.11epoch/s]
2024-06-02 12:24:28,058 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '4', '/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_3/scripts/train.py', 'run', "--config_file='/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_3/configs/hyper_parameters.yaml'"]
W0602 12:24:28.975201 140445166011392 torch/distributed/run.py:757] 
W0602 12:24:28.975201 140445166011392 torch/distributed/run.py:757] *****************************************
W0602 12:24:28.975201 140445166011392 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0602 12:24:28.975201 140445166011392 torch/distributed/run.py:757] *****************************************
segresnet_3 - training: 100%|██████████| 1250/1250 [09:43<00:00,  2.14epoch/s]
2024-06-02 12:34:24,844 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '4', '/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_4/scripts/train.py', 'run', "--config_file='/home/mri/laminate/auto3dseg/notebooks/work_dir/segresnet_4/configs/hyper_parameters.yaml'"]
W0602 12:34:25.756102 139943283799040 torch/distributed/run.py:757] 
W0602 12:34:25.756102 139943283799040 torch/distributed/run.py:757] *****************************************
W0602 12:34:25.756102 139943283799040 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0602 12:34:25.756102 139943283799040 torch/distributed/run.py:757] *****************************************
segresnet_4 - training: 100%|██████████| 1250/1250 [09:51<00:00,  2.11epoch/s]
2024-06-02 12:44:26,639 - INFO - ['torchrun', '--nnodes', '1', '--nproc_per_node', '4', '/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/scripts/train.py', 'run', "--config_file='/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/configs/hyper_parameters.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/configs/network.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/configs/transforms_infer.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/configs/transforms_train.yaml,/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/configs/transforms_validate.yaml'"]
W0602 12:44:27.578003 140421286454272 torch/distributed/run.py:757] 
W0602 12:44:27.578003 140421286454272 torch/distributed/run.py:757] *****************************************
W0602 12:44:27.578003 140421286454272 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0602 12:44:27.578003 140421286454272 torch/distributed/run.py:757] *****************************************
[rank3]: Traceback (most recent call last):
[rank3]:   File "/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/scripts/train.py", line 813, in <module>
[rank3]:     fire.Fire()
[rank3]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/fire/core.py", line 143, in Fire
[rank3]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank3]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/fire/core.py", line 477, in _Fire
[rank3]:     component, remaining_args = _CallAndUpdateTrace(
[rank3]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
[rank3]:     component = fn(*varargs, **kwargs)
[rank3]:   File "/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/scripts/train.py", line 318, in run
[rank3]:     model_dict = torch.load(pretrained_path, map_location=device)["state_dict"]
[rank3]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/serialization.py", line 997, in load
[rank3]:     with _open_file_like(f, 'rb') as opened_file:
[rank3]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/serialization.py", line 444, in _open_file_like
[rank3]:     return _open_file(name_or_buffer, mode)
[rank3]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/serialization.py", line 425, in __init__
[rank3]:     super().__init__(open(name, mode))
[rank3]: FileNotFoundError: [Errno 2] No such file or directory: '/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt'
[rank1]: Traceback (most recent call last):
[rank1]:   File "/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/scripts/train.py", line 813, in <module>
[rank1]:     fire.Fire()
[rank1]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/fire/core.py", line 143, in Fire
[rank1]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank1]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/fire/core.py", line 477, in _Fire
[rank1]:     component, remaining_args = _CallAndUpdateTrace(
[rank1]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
[rank1]:     component = fn(*varargs, **kwargs)
[rank1]:   File "/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/scripts/train.py", line 318, in run
[rank1]:     model_dict = torch.load(pretrained_path, map_location=device)["state_dict"]
[rank1]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/serialization.py", line 997, in load
[rank1]:     with _open_file_like(f, 'rb') as opened_file:
[rank1]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/serialization.py", line 444, in _open_file_like
[rank1]:     return _open_file(name_or_buffer, mode)
[rank1]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/serialization.py", line 425, in __init__
[rank1]:     super().__init__(open(name, mode))
[rank1]: FileNotFoundError: [Errno 2] No such file or directory: '/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt'
[rank2]: Traceback (most recent call last):
[rank2]:   File "/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/scripts/train.py", line 813, in <module>
[rank2]:     fire.Fire()
[rank2]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/fire/core.py", line 143, in Fire
[rank2]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank2]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/fire/core.py", line 477, in _Fire
[rank2]:     component, remaining_args = _CallAndUpdateTrace(
[rank2]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
[rank2]:     component = fn(*varargs, **kwargs)
[rank2]:   File "/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/scripts/train.py", line 318, in run
[rank2]:     model_dict = torch.load(pretrained_path, map_location=device)["state_dict"]
[rank2]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/serialization.py", line 997, in load
[rank2]:     with _open_file_like(f, 'rb') as opened_file:
[rank2]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/serialization.py", line 444, in _open_file_like
[rank2]:     return _open_file(name_or_buffer, mode)
[rank2]:   File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/serialization.py", line 425, in __init__
[rank2]:     super().__init__(open(name, mode))
[rank2]: FileNotFoundError: [Errno 2] No such file or directory: '/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt'
W0602 12:44:37.615798 140421286454272 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 47260 closing signal SIGTERM
E0602 12:44:38.180283 140421286454272 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 1 (pid: 47261) of binary: /opt/miniconda3/envs/monailabel/bin/python
Traceback (most recent call last):
  File "/opt/miniconda3/envs/monailabel/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/miniconda3/envs/monailabel/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/home/mri/laminate/auto3dseg/notebooks/work_dir/swinunetr_0/scripts/train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-06-02_12:44:37
  host      : b141-l-oa22-monai
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 47262)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-06-02_12:44:37
  host      : b141-l-oa22-monai
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 47263)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-06-02_12:44:37
  host      : b141-l-oa22-monai
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 47261)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

[diazandr3s]

Hi @soumen02,

Not sure what's causing this error. But, from the logs, I see you managed to train dints and segresnet. They should be more than enough for performance.

I'd suggest you test the performance of segresnet (i.e. using the model.pt file located at segresnet_0/model) and these instructions: lassoan/SlicerMONAIAuto3DSeg#52 (comment)

Let us know,

[soumen02]

Hey @diazandr3s ,

Thanks for the reply, I'll try that.
Does the above method allow for the ensemble feature that Auto3DSeg provides? If not, is it better to just train using Monailabel?

Thanks a lot,

[diazandr3s]

No, that script is only for single-model inference. For ensemble, please use the Auto3DSeg API directly.
You could also make one of these models to run in MONAI Label.