Teaching robot hands to copy any cooking video on YouTube

Robots with fingers are clumsy because nobody has enough training data. You can't easily wire up a five-fingered robot hand to learn from. But there's a massive free library sitting right there: ordinary videos of human hands doing things — chopping, pouring, picking things up — shot on a single regular camera.

The problem is that those videos are flat. A normal video records color (that's what "RGB" means — the red, green, blue pixels), but no depth. So the computer can't tell exactly where the hand is in 3D space, how the fingers wrap around an object, or how the object moves. And even if you figure all that out, a human hand and a robot hand aren't shaped the same — so a robot can't just mimic the joint angles directly.

DO AS I DO tackles both gaps. First it reconstructs what the hand and object are actually doing in 3D from the flat video — whether the camera is on the person's head looking down at their own hands, or filming them from across the room. Then it "retargets": it translates that human motion into a sequence of moves a differently-shaped robot hand can actually execute in the real world.

The result is real, usable robot training data extracted from random clips. On benchmark videos with known correct answers, the method reconstructs hands and objects more accurately than previous approaches.

Whether robots trained on this data reliably succeed at full tasks isn't shown here — they prove the data is good, not yet the robot.

Why you should care: Every cooking or repair video ever filmed becomes potential robot lessons, instead of needing someone to manually puppet a robot hand for each new skill.