AI Infra / Engineer - Large-scale Training

Your Mission As AI Infra / Engineer — Large-scale Training, you will build the systems that make UMA’s humanoid robot learning fast, scalable, and reliable. You will turn experimental training code into robust execution pipelines that scale across thousands of GPUs and support rapid iteration. You will work directly with AI researchers, shaping the software used to train our robotic foundation models across pretraining, mid-training, and post-training on large-scale multimodal data. You will own the engineering challenges of distributed execution: choosing parallelism strategies, optimizing memory and communication, accelerating critical operations, and keeping long-running jobs reliable.. We already have a working training stack and a strong team behind it. You will build on that foundation, with room to rethink components where needed. Working closely with AI researchers, you will scale pretraining across GPUs and nodes, optimize compute, memory, and communication, and make long-running training jobs reliable. The goal is to reduce training time and increase experiment throughput, enabling larger models and faster research iteration within our compute budget. Debugging here goes beyond keeping jobs running. You will understand enough of the learning process to recognize when a numerical issue or distributed implementation bug appears in a loss curve, and work with researchers to resolve it. The systems you build will also support mid-training and post-training as our models evolve. Key responsibilities: - Build distributed pretraining systems across multiple GPUs and nodes, selecting appropriate strategies such as DDP, FSDP, and tensor, pipeline, or expert parallelism as workloads require. - Develop RL and post-training execution infrastructure, including distributed rollouts, actor–learner coordination, policy synchronization, replay buffers, and efficient batching. - Maximize MFU and eliminate bottlenecks: Profile the complete training loop—data loading, decoding, CPU-GPU memory transfers, and compute kernels—eliminating bottlenecks to keep MFU high and ensure accelerators operate at peak efficiency. - Integrate and benchmark accelerated kernels and execution techniques, such as FlashAttention, fused operations, mixed precision, and compilation; develop custom optimizations where they provide clear value. - Build efficient training-time data movement and buffering, including out-of-core storage, caching, prefetching, and asynchronous transfers, in partnership with the Data Infrastructure team. - Turn research prototypes into maintainable training systems, with reusable components that support evolving architectures and experimental workflows. - Diagnose distributed and numerical failures, including deadlocks, memory leaks, gradient anomalies, precision issues, and inconsistencies between single-device and distributed runs. - Deep numerical debugging and run determinism: Diagnose complex issues across the stack—from distributed deadlocks and memory leaks to training instabilities and gradient anomalies—while enforcing numerical determinism so that training runs, evaluations, and experiments are strictly reproducible. - Build reliable checkpointing and fault-tolerant resumption, preserving model, optimizer, and relevant execution state so interrupted jobs can recover correctly. - Strengthen observability and reproducibility through configuration tracking, profiling, run diagnostics, and deterministic execution where practical. What You Bring To The Table - Solid experience building ML training infrastructure, distributed systems, or high-performance ML software. - Strong Python and PyTorch, with hands-on knowledge of distributed training tools such as DDP, FSDP, or Megatron. - Experience with multiprocessing, asynchronous execution, and GPU memory management, and an understanding of how communication and synchronization affect scaling. - Practical knowledge of losses, optimizers, gradients, and numerical precision, sufficient to debug training behavior alongside researchers. - Experience profiling accelerator workloads with tools such as PyTorch Profiler or Nsight Systems, and translating findings into measurable improvements. - A pragmatic engineering mindset: you can turn experimental code into reliable, scalable systems while keeping research iteration fast. - Clear communication and a collaborative approach to working with researchers whose requirements evolve quickly. - Bonus: RL infrastructure, distributed rollouts, replay buffers, or actor–learner architectures. - Bonus: C++, CUDA, Triton, custom kernels, or compiler-based optimization. - Bonus: Slurm or Kubernetes, GPU networking such as InfiniBand/RoCE, or storage caching. - Bonus: contributions to open-source training tools or technical writing. UMA is an inclusive workplace that values exceptional builders over perfect pedigrees. Whatever your background, identity, or journey, if you don’t meet every criterion but believe you can have an outsized impact here, we strongly encourage you to apply.