Experiments with Success Guided Sampling ran up to 2^20 parallel environments, over one million, and solved multi-terrain quadruped locomotion and contact-rich assembly tasks that prior methods fail to solve.
Diverse simulator resets, combined with massively parallel simulation, can ease much of the engineering burden on several manipulation problems. Naively scaling that approach to more precise or dynamic problems remains non-trivial.
Robots waste a growing share of their practice time. A new sampler targets configurations around the frontier of a policy's capabilities.
Uniform sampling wastes experience. Sampling uniformly from the reset distribution wastes a growing fraction of learning experience on configurations the policy has mastered or cannot yet attempt.
Hard to see the benefit of more parallel environments. Because so much of a batch's learning signal is wasted, it's hard to see the expected benefit of adding parallel environments.
Success Guided Sampling focuses on the frontier. Octi Zhang and six co-authors propose SGS, a simple adaptive sampler that concentrates training around the frontier of the policy's capabilities.
Over one million parallel environments. Experiments ran up to 2^20 parallel environments. At that scale, SGS lets RL solve multi-terrain quadruped locomotion and contact-rich assembly that prior methods fail to solve.
Zero-shot transfer to real hardware. The team distilled the manipulation policies into RGB-based policies, which transferred zero-shot to several challenging assembly tasks on real hardware.
Cover figure: Zhang, Octi et al., arXiv:2610.12465, CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/), https://arxiv.org/abs/2610.12465
