The NVIDIA Resiliency Extension (NVRx) integrates multiple resiliency-focused solutions for PyTorch-based workloads. Users can modularly integrate NVRx capabilities into their own infrastructure to maximize AI training productivity at scale. NVRx maximizes goodput by enabling system-wide health checks, quickly detecting faults at runtime and resuming training automatically. NVRx minimizes loss of work by enabling fast and frequent checkpointing.
For detailed documentation and usage information about each component, please refer to https://nvidia.github.io/nvidia-resiliency-ext/.
⚠️ NOTE: This project is still experimental and under active development. The code, features, and documentation are evolving rapidly. Please expect frequent updates and breaking changes. Contributions are welcome and we encourage you to watch for updates.
-
- Section-based hang detection and coordinated worker restarts within a running job.
- Megatron GPT recovery walkthrough with fixed CLI timeouts and NVRx-backed async checkpoints.
- Small DDP example with section monitoring and synchronous PyTorch checkpoints.
-
- Providing an efficient framework for asynchronous checkpointing.
-
- Providing an efficient framework for local checkpointing.
-
- Monitoring GPU and CPU performance of ranks.
- Identifying slower ranks that may impede overall training efficiency.
git clone https://github.com/NVIDIA/nvidia-resiliency-extcd nvidia-resiliency-extpip install .
pip install nvidia-resiliency-ext
| Category | Supported Versions / Requirements |
|---|---|
| Architecture | x86_64, arm64 |
| Operating System | Ubuntu 22.04, 24.04 |
| Python Version | Release wheels/CI: 3.12, 3.14 |
| PyTorch Version | >= 2.8.0 |
| CUDA & CUDA Toolkit | >= 12.8 |
| NVML Driver | >= 535 (570 required for GPU health check) |
| TE Version | >= 2.5 |