Turning a single shaky video into a full 3D moving object

Point your phone at a dog shaking off water and you capture only one angle, one frame at a time. The far side of the dog, the parts hidden behind its own body — your camera never sees them. So how do you rebuild the whole dog, in 3D, moving through every moment of the clip? That's the problem Lift4D tackles.

Older methods split into two camps, both flawed. One trains an AI to guess the full moving 3D shape directly — but there's barely any training data showing objects moving in 3D, so it guesses badly. The other builds a 3D shape from the first frame, then bends it to match the rest of the video — but once it starts bending, it's flying blind, with only the flat video to correct it.

Lift4D borrows from both. First it runs a 3D-from-one-image model on every frame, but links the frames so the shape doesn't jitter between them. That gives a clean starting model. Then it "sculpts" that model to match the video: where the camera can see the surface, it copies the real details faithfully; where the object is hidden, it fills in the gaps using an AI trained to imagine plausible unseen sides.

The paper shows results look clearly better, but offers no hard numbers — "clearly improves" is the authors' own phrasing, not a measured score.

Why you should care: The hidden, occluded parts of a moving object are exactly where every prior method failed. Lift4D's trick of imagining those unseen sides is what finally makes everyday phone clips of bending, twisting things rebuildable in 3D.