Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions accelerator/cpu_accelerator.py
Original file line number Diff line number Diff line change
Expand Up @@ -316,9 +316,9 @@ def get_op_builder(self, class_name):
# is op_builder from deepspeed or a 3p version? this should only succeed if it's deepspeed
# if successful this also means we're doing a local install and not JIT compile path
from op_builder import __deepspeed__ # noqa: F401 # type: ignore
from op_builder.cpu import AsyncIOBuilder, CCLCommBuilder, ShareMemCommBuilder, FusedAdamBuilder, CPUAdamBuilder, NotImplementedBuilder
from op_builder.cpu import AsyncIOBuilder, CCLCommBuilder, ShareMemCommBuilder, FusedAdamBuilder, CPUAdamBuilder, PinMemoryBuilder, NotImplementedBuilder
except ImportError:
from deepspeed.ops.op_builder.cpu import AsyncIOBuilder, CCLCommBuilder, ShareMemCommBuilder, FusedAdamBuilder, CPUAdamBuilder, NotImplementedBuilder
from deepspeed.ops.op_builder.cpu import AsyncIOBuilder, CCLCommBuilder, ShareMemCommBuilder, FusedAdamBuilder, CPUAdamBuilder, PinMemoryBuilder, NotImplementedBuilder

if class_name == "CCLCommBuilder":
return CCLCommBuilder
Expand All @@ -330,6 +330,8 @@ def get_op_builder(self, class_name):
return CPUAdamBuilder
elif class_name == "AsyncIOBuilder":
return AsyncIOBuilder
elif class_name == "PinMemoryBuilder":
return PinMemoryBuilder
else:
# return a NotImplementedBuilder to avoid get NoneType[Name] in unit tests
return NotImplementedBuilder
Expand Down
5 changes: 3 additions & 2 deletions accelerator/xpu_accelerator.py
Original file line number Diff line number Diff line change
Expand Up @@ -233,8 +233,9 @@ def _pin_memory(self, tensor, align_bytes=1):
if align_bytes == 1:
return tensor.pin_memory(device=self.current_device_name())
elif align_bytes == 0:
from deepspeed.ops.op_builder.xpu import AsyncIOBuilder
self.aio_handle = AsyncIOBuilder().load().aio_handle(128 * 1024, 8, False, False, False)
# Page-locked host buffer without starting AIO worker threads.
from deepspeed.ops.op_builder import PinMemoryBuilder
self.aio_handle = PinMemoryBuilder().load().pin_handle()
aligned_t = self.aio_handle.new_cpu_locked_tensor(tensor.numel(), tensor)
aligned_t = aligned_t[:tensor.numel()].copy_(tensor)
self.aligned_tensors.append([aligned_t.data_ptr(), aligned_t[-1].data_ptr()])
Expand Down
22 changes: 0 additions & 22 deletions csrc/aio/common/deepspeed_aio_utils.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -116,25 +116,3 @@ int64_t get_fd_file_size(const int fd, int64_t& size)
size = st.st_size;
return 0;
}

void* ds_page_aligned_alloc(const int64_t size, const bool lock)
{
void* ptr;
int retval;

retval = posix_memalign(&ptr, (size_t)sysconf(_SC_PAGESIZE), size);
if (retval) { return nullptr; }

if (lock == false) { return ptr; }

auto mlock_ret = mlock(ptr, size);
if (mlock_ret != 0) {
auto mlock_error = errno;
std::cerr << "mlock failed to allocate " << size << " bytes with error no " << mlock_error
<< " msg " << strerror(mlock_error) << std::endl;
free(ptr);
return nullptr;
}

return ptr;
}
2 changes: 0 additions & 2 deletions csrc/aio/common/deepspeed_aio_utils.h
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,5 @@ struct io_prep_generator {
int prep_iocbs(const int n_iocbs, std::vector<struct iocb*>* iocbs);
};

void* ds_page_aligned_alloc(const int64_t size, const bool lock = false);

int64_t get_file_size(const char* filename, int64_t& size);
int64_t get_fd_file_size(const int fd, int64_t& size);
32 changes: 3 additions & 29 deletions csrc/aio/py_lib/deepspeed_pin_tensor.h
Original file line number Diff line number Diff line change
@@ -1,37 +1,11 @@
// Copyright (c) Microsoft Corporation.
// SPDX-License-Identifier: Apache-2.0

// DeepSpeed Team

/*
Functionality for managing CPU tensors occupying page-locked memory.
TODO: Implement a full-featured manager that
1. Avoid page-locked memory leaks
2. Minimize page-locked memory usage by reducing internal fragmentation
Functionality for managing CPU tensors occupying page-locked memory.
Compatibility include: pin-tensor manager lives under csrc/pin_memory/.
*/

#include <map>
#include <memory>
#include <mutex>
#include "deepspeed_py_aio.h"
#pragma once

struct deepspeed_pin_tensor_t {
std::map<void*, int64_t> _locked_tensors;
std::mutex _mutex;

deepspeed_pin_tensor_t() = default;

~deepspeed_pin_tensor_t();

// Process-wide shared manager so that pinned-buffer recognition is consistent
// across every io handle (each handle references this single instance).
static std::shared_ptr<deepspeed_pin_tensor_t> shared();

torch::Tensor alloc(const int64_t num_elem, const at::ScalarType& elem_type);
torch::Tensor alloc(const int64_t num_elem, const torch::TensorOptions& options);

bool free(torch::Tensor& locked_tensor);

bool is_managed(const torch::Tensor& buffer);
};
#include "../../pin_memory/deepspeed_pin_tensor.h"
28 changes: 28 additions & 0 deletions csrc/aio/py_lib/deepspeed_pin_tensor_client.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
// SPDX-License-Identifier: Apache-2.0

// DeepSpeed Team

/*
Resolve the process-wide pin-tensor manager exported by the pin_memory op.
The manager is compiled only into pin_memory; async_io/gds must load that op
first (see AsyncIOBuilder.load) so this symbol is visible via RTLD_GLOBAL.
*/

#include "deepspeed_pin_tensor.h"

#include <dlfcn.h>
#include <stdexcept>

std::shared_ptr<deepspeed_pin_tensor_t> deepspeed_pin_tensor_t::shared()
{
using holder_fn_t = void* (*)();
auto* fn =
reinterpret_cast<holder_fn_t>(dlsym(RTLD_DEFAULT, "deepspeed_pin_tensor_mgr_holder"));
if (fn == nullptr) {
throw std::runtime_error(
"DeepSpeed pin_memory op must be loaded before async_io/gds (missing "
"deepspeed_pin_tensor_mgr_holder). Load PinMemoryBuilder first.");
}
auto* holder = static_cast<std::shared_ptr<deepspeed_pin_tensor_t>*>(fn());
return *holder;
}
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
// Copyright (c) Microsoft Corporation.
// SPDX-License-Identifier: Apache-2.0

// DeepSpeed Team
Expand All @@ -8,6 +7,11 @@ Functionality for managing CPU tensors occupying page-locked memory.
*/

#include "deepspeed_pin_tensor.h"
#include "page_alloc.h"

#include <cassert>

#include <sys/mman.h>

using namespace std;

Expand All @@ -26,6 +30,15 @@ std::shared_ptr<deepspeed_pin_tensor_t> deepspeed_pin_tensor_t::shared()
return mgr;
}

extern "C" void* deepspeed_pin_tensor_mgr_holder()
{
// Heap-allocate the shared_ptr so its control block outlives any transient
// copies made by other extensions that resolve this symbol via dlsym.
static auto* holder =
new std::shared_ptr<deepspeed_pin_tensor_t>(deepspeed_pin_tensor_t::shared());
return static_cast<void*>(holder);
}

torch::Tensor deepspeed_pin_tensor_t::alloc(const int64_t num_elem,
const torch::TensorOptions& options)
{
Expand Down
42 changes: 42 additions & 0 deletions csrc/pin_memory/deepspeed_pin_tensor.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
// SPDX-License-Identifier: Apache-2.0

// DeepSpeed Team

/*
Functionality for managing CPU tensors occupying page-locked memory.
TODO: Implement a full-featured manager that
1. Avoid page-locked memory leaks
2. Minimize page-locked memory usage by reducing internal fragmentation
*/

#pragma once

#include <torch/extension.h>
#include <map>
#include <memory>
#include <mutex>

struct deepspeed_pin_tensor_t {
std::map<void*, int64_t> _locked_tensors;
std::mutex _mutex;

deepspeed_pin_tensor_t() = default;

~deepspeed_pin_tensor_t();

// Process-wide shared manager so that pinned-buffer recognition is consistent
// across every io handle (each handle references this single instance).
// Canonical instance lives in the pin_memory extension; other ops resolve it
// via deepspeed_pin_tensor_mgr_holder() after that extension is loaded.
static std::shared_ptr<deepspeed_pin_tensor_t> shared();

torch::Tensor alloc(const int64_t num_elem, const at::ScalarType& elem_type);
torch::Tensor alloc(const int64_t num_elem, const torch::TensorOptions& options);

bool free(torch::Tensor& locked_tensor);

bool is_managed(const torch::Tensor& buffer);
};

// Exported so async_io/gds can resolve the same manager across .so boundaries.
extern "C" void* deepspeed_pin_tensor_mgr_holder();
34 changes: 34 additions & 0 deletions csrc/pin_memory/page_alloc.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
// SPDX-License-Identifier: Apache-2.0

// DeepSpeed Team

#include "page_alloc.h"

#include <sys/mman.h>
#include <unistd.h>
Comment on lines +7 to +8

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Guard the POSIX pin_memory op on Windows

When building wheels on Windows, setup.py defaults to prebuilding compatible ops, but this new op inherits the default compatibility and its source uses POSIX-only headers/APIs such as sys/mman.h, unistd.h, posix_memalign, sysconf, and mlock. A Windows build will now try to compile this source and fail during installation, so add a Windows implementation or make PinMemoryBuilder.is_compatible() return false on Windows.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 0dfc557: PinMemoryBuilder.is_compatible() returns False on win32 because the allocator uses POSIX mlock/posix_memalign.

#include <cerrno>
#include <cstdlib>
#include <cstring>
#include <iostream>

void* ds_page_aligned_alloc(const int64_t size, const bool lock)
{
void* ptr;
int retval;

retval = posix_memalign(&ptr, (size_t)sysconf(_SC_PAGESIZE), size);
if (retval) { return nullptr; }

if (lock == false) { return ptr; }

auto mlock_ret = mlock(ptr, size);
if (mlock_ret != 0) {
auto mlock_error = errno;
std::cerr << "mlock failed to allocate " << size << " bytes with error no " << mlock_error
<< " msg " << strerror(mlock_error) << std::endl;
free(ptr);
return nullptr;
}

return ptr;
}
13 changes: 13 additions & 0 deletions csrc/pin_memory/page_alloc.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
// SPDX-License-Identifier: Apache-2.0

// DeepSpeed Team

/*
Page-aligned host allocation with optional mlock, independent of libaio.
*/

#pragma once

#include <cstdint>

void* ds_page_aligned_alloc(const int64_t size, const bool lock = false);
48 changes: 48 additions & 0 deletions csrc/pin_memory/py_ds_pin_memory.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
// SPDX-License-Identifier: Apache-2.0

// DeepSpeed Team

#include <torch/extension.h>
#include "deepspeed_pin_tensor.h"

using namespace pybind11::literals;

struct deepspeed_pin_handle_t {
std::shared_ptr<deepspeed_pin_tensor_t> _pinned_tensor_mgr;

deepspeed_pin_handle_t() : _pinned_tensor_mgr(deepspeed_pin_tensor_t::shared()) {}

torch::Tensor new_cpu_locked_tensor(const int64_t num_elem, const torch::Tensor& example_tensor)
{
return _pinned_tensor_mgr->alloc(num_elem, example_tensor.options());
}

bool free_cpu_locked_tensor(torch::Tensor& locked_tensor)
{
return _pinned_tensor_mgr->free(locked_tensor);
}

bool is_pinned(const torch::Tensor& buffer)
{
return buffer.is_pinned() || _pinned_tensor_mgr->is_managed(buffer);
}
};

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m)
{
py::class_<deepspeed_pin_handle_t>(m, "pin_handle")
.def(py::init<>(), "Pin-memory handle constructor")
.def("new_cpu_locked_tensor",
&deepspeed_pin_handle_t::new_cpu_locked_tensor,
"Allocate a page-locked CPU tensor",
"num_elem"_a,
"example_tensor"_a)
.def("free_cpu_locked_tensor",
&deepspeed_pin_handle_t::free_cpu_locked_tensor,
"Free a page-locked CPU tensor",
"locked_tensor"_a)
.def("is_pinned",
&deepspeed_pin_handle_t::is_pinned,
"Return whether buffer is torch-pinned or managed by DeepSpeed pin_memory",
"buffer"_a);
}
5 changes: 5 additions & 0 deletions deepspeed/ops/pin_memory/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# SPDX-License-Identifier: Apache-2.0

# DeepSpeed Team

from ..op_builder.pin_memory import PinMemoryBuilder

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Document the new pin_memory API

This commit exposes a new public deepspeed.ops.pin_memory / PinMemoryBuilder / pin_handle path and changes how DeepNVMe pinning is shared, but the changed files include no documentation or tutorial updates explaining how users should call it or how it differs from the existing PyTorch/accelerator pinning paths. The repository instructions require documentation updates for new features.

AGENTS.md reference: AGENTS.md:L26-L26

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 0dfc557 (DeepNVMe tutorial) and follow-ups in docs/code-docs/source/memory.rst (Host Memory Pinning): documents PinMemoryBuilder / pin_handle, shared process-wide manager with AIO/GDS, and how it differs from torch/accelerator pinning.

15 changes: 15 additions & 0 deletions docs/_tutorials/deepnvme.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,21 @@ True
1073741824
```

You can also allocate page-locked host tensors directly through the standalone `pin_memory` operator, which pins CPU memory (`posix_memalign` + `mlock`). This is useful on CPU-only hosts or when you want a pinned buffer without starting an I/O handle. Buffers allocated this way are recognized by DeepNVMe I/O handles (they share one process-wide pin manager), so DeepNVMe can still skip bounce buffers for them.

```bash
>>> import torch
>>> from deepspeed.ops.op_builder import PinMemoryBuilder
>>> pin = PinMemoryBuilder().load().pin_handle()
>>> t = pin.new_cpu_locked_tensor(1024**3, torch.empty(0, dtype=torch.uint8))
>>> pin.is_pinned(t)
True
>>> pin.free_cpu_locked_tensor(t)
True
```

The `pin_handle` exposes `new_cpu_locked_tensor(num_elem, example_tensor)`, `free_cpu_locked_tensor(tensor)`, and `is_pinned(tensor)`. The same methods remain available on DeepNVMe I/O handles as thin wrappers, so existing code continues to work unchanged.

On the other hand,`gds_handle` provides `new_pinned_device_tensor()` and `pin_device_tensor()` functions for pinning CUDA tensors. The following example illustrates writing a pinned CUDA tensor to a local NVMe file.

```bash
Expand Down
Loading
Loading