DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation

Imitation learning from human hand motion data presents a promising avenuefor imbuing robots with human-like dexterity in real-world manipulation tasks.Despite this potential, substantial challenges persist, particularly with theportability of existing hand motion capture (mocap) systems and the complexityof translating mocap data into effective robotic policies. To tackle theseissues, we introduce DexCap, a portable hand motion capture system, alongsideDexIL, a novel imitation algorithm for training dexterous robot skills directlyfrom human hand mocap data. DexCap offers precise, occlusion-resistant trackingof wrist and finger motions based on SLAM and electromagnetic field togetherwith 3D observations of the environment. Utilizing this rich dataset, DexILemploys inverse kinematics and point cloud-based imitation learning toseamlessly replicate human actions with robot hands. Beyond direct learningfrom human motion, DexCap also offers an optional human-in-the-loop correctionmechanism during policy rollouts to refine and further improve taskperformance. Through extensive evaluation across six challenging dexterousmanipulation tasks, our approach not only demonstrates superior performance butalso showcases the system’s capability to effectively learn from in-the-wildmocap data, paving the way for future data collection methods in the pursuit ofhuman-level robot dexterity. More details can be found athttps://dex-cap.github.io

Further reading#

• Access Paper in arXiv.org