Teaching robots by thumbs-up: a smarter way to pick what to try next

How do you train a robot when you can't write down exactly what "good" looks like? One trick: show it two attempts and say which you prefer. The robot slowly builds a guess about what you want from thousands of these thumbs-up, thumbs-down comparisons. The problem is speed. Most methods wait around for comparisons to trickle in, so early learning is painfully slow.

UBP2 fixes this by making the robot pick its next move deliberately instead of stumbling around. It keeps three separate mental models: one guessing what you reward, one guessing how the world responds to its actions, and one guessing how valuable a situation is. For each model it runs a small committee of versions — when the committee disagrees, that's a sign the robot doesn't really know yet. That disagreement is its measure of "how unsure am I."

Before acting, the robot scores possible future paths on one combined number: how good it expects the outcome to be, plus how much it would learn by going there. So it actively steers toward places where it's both promising and uncertain — chasing reward and chasing information in one shot, with no bolted-on "explore randomly" rule.

The authors prove the mistakes pile up slowly over time, and on the Meta-World robot-task benchmark UBP2 learned from far fewer comparisons than rival methods. They don't show it beyond simulated tasks, though.

Why you should care: Every preference comparison costs a human's time. UBP2 wrings more learning out of each one, making thumbs-up training of machines actually practical.