A C++ recreation of Hugging Face's LeRobot for robotics AI learning.
This project recreates key components like dataset loading (Apache Arrow/Parquet), normalization, and a basic ACT (Action Chunking Transformer) policy in pure C++ with LibTorch. It supports training on datasets like Push-T, with plans for full ACT chunking, simulation testing, and hardware deployment.
| Dep | Why | Resources |
|---|---|---|
| LibTorch | Tensors for states, actions, policies (e.g., ACT transformer) | PyTorch C++ Docs |
| Apache Arrow + Parquet | Parquet reads for hf_dataset (frames, episodes) | Arrow C++ Docs, Quickstart |
| OpenCV | Image/video for obs.images, viz | OpenCV Docs |
| nlohmann/json | Parse meta.json (fps, episodes) | JSON for Modern C++ |
source ./setup.sh# Downloads LibTorch, installs apt deps (Arrow, OpenCV, JSON, GTest). Note: Source to set env../build.sh# CMake + make.LD_LIBRARY_PATH=/path/to/libtorch/lib build/train# Runs training on Push-T (or other datasets).
- Dataset Loading: Supports LeRobot-style datasets (e.g., Push-T). Normalization stats (mean/std) computed/cached.
- Policy: Basic ACT policy implemented with CNN (image feat), state projection, Transformer encoder, and linear head. Predicts single actions; trained via MSE on normalized data.
- Training: Single-sample SGD with Adam, grad clipping. Logs loss every 1k steps, avg every 10k. Tested on Push-T (25k frames, state/action dim=2).
- Tweaks: Configurable hidden_dim (64/256), lower LR for stability.
Work in progress. train.cpp trains the policy; add inference.cpp for deployment. Tests in main.cpp verify deps (tensor, array, image, JSON).
- Implement full ACT chunking (predict multiple future actions for planning).
- Add simulation (e.g., simple 2D Push-T in C++)
- Hardware support (e.g., SO-101 arm via serial SDK).
- Multi-dataset training (e.g., ALOHA, xArm).