Software Engineer, ML Training Optimization
At a glance
Mid-level Software Engineering role at Zoox. Foster City · full-time · $177,000–$270,000 base.
techroles summary, based on the employer's posting.
What you'll do
- Improve training speed and scale for VLM, VLA and foundation models using parallelism, mixed precision, checkpointing and GPU kernels
- Tune reinforcement-learning frameworks and their performance for driving foundation models
- Trace slow training across compute, compilers, frameworks and cluster networking, then resolve the bottlenecks
- Work with research, data and hardware teams to shape requirements and architecture for large-scale training systems
- Build custom kernels and benchmarks to check whether training optimizations deliver
What you bring
- At least four years of overall experience, including two years on large-scale model-training platforms
- Hands-on experience with distributed training frameworks and large GPU or accelerator clusters
- Experience with reinforcement-learning frameworks and algorithms such as PPO or GRPO
- Proficiency in Python or C++
Who this fits
This mid-level role suits a software engineer focused on ML training performance, distributed systems and reinforcement learning. It is full-time and onsite in Foster City.
From the employer
In this role, you will:
-
Design, implement, and operate cutting-edge ML training performance optimization techniques — distributed training (data/tensor/pipeline parallelism), mixed-precision, gradient checkpointing, and GPU kernels — to scale our VLM, VLA, and Foundational models training efficiency.
-
Work on RL framework and RL performance optimization for driving foundation model
-
Profile and debug training throughput bottlenecks across the compute, compiler, framework, and cluster-networking stack.
-
Collaborate closely with ML researchers, data engineers, and hardware engineers to define requirements and align on architectural decisions for large-scale training infrastructure.
-
Implement custom kernels and build benchmarks to track and validate training-performance optimizations across the stack.
Qualifications
-
4+ years of total experience, including 2+ years working on large-scale model training platforms.
-
Experience with distributed training frameworks (Megatron-LM, TorchTitan), efficiently utilizing GPU/accelerator clusters at scale.
-
Experience with RL framework such as Verl/MILES and RL algorithms(GRPO, PPO)
-
Experience with collective communication libraries (e.g., NCCL) and optimizing communication/compute overlap in multi-node training.
-
Experience using profiling tools (NVIDIA Nsight, PyTorch Profiler, or XLA/JAX profiling) to identify and resolve training bottlenecks.
-
Proficient in Python or C++.
-
Experience with ML compilers, performance tooling, or hardware/software co-design (e.g., GPU/TPU/accelerator architectures).
Note: You do not have to meet all the requirements below to be considered for this position:
Follow us on LinkedIn
AccommodationsIf you need an accommodation to participate in the application or interview process please reach out to accommodations@zoox.com or your assigned recruiter.
A Final Note:You do not need to match every listed expectation to apply for this position. Here at Zoox, we know that diverse perspectives foster the innovation we need to be successful, and we are committed to building a team that encompasses a variety of backgrounds, experiences, and skills.