Teach a Robot to Move Before Teaching It to See and Listen

Today's robot brains learn three hard things at once: how to see, how to understand commands, and how to move. The seeing and understanding come pre-loaded from existing AI models. But the moving part starts blank, forced to figure out smooth physical motion at the same moment it learns to connect what it sees to what its hands should do. That's a lot to juggle, and it shows.

This paper splits the job in two. First, the robot's motion part trains alone, watching nothing but raw movement trajectories — sequences of joint positions over time — with no images and no language attached. It just learns what natural motion looks like across different robot bodies: how arms tend to swing, how grips tend to close. Think of it as practicing handwriting before you're asked to write an essay; the muscle memory comes first.

Then comes the second stage. That pre-trained motion knowledge gets carried into the full system, so when the robot finally learns to link vision and commands to action, it isn't starting from zero on the physics. A bonus: the same trained component can squeeze a robot's entire history of past states and moves into one small summary, giving it memory cheaply.

Across 13 different tasks on real and simulated robots, this learned-motion-first approach trained faster and succeeded more often — especially on real-world jobs with little data. Feed it more motion data up front, and downstream performance keeps climbing. The paper doesn't claim it solves every manipulation problem, and the gains were measured on these specific tasks.

Why you should care: Real robot data is scarce and expensive to collect. Pre-loading raw motion lets a robot learn useful skills from fewer hands-on demonstrations — the bottleneck holding household and warehouse robots back.