AIs that play card games fail because they forget what they saw, not because they choose badly

Play a game of Memory: flip two cards, see a fox and a moon, flip them back. The whole game depends on remembering where the fox was. Now ask an AI to do the same — and watch it lose, not because it picks the wrong card, but because it simply forgot what it saw three turns ago.

That's the gap RNG-Bench is built to expose. Most tests for these vision-and-language AIs let the model see everything it needs on the current screen. Real tasks don't work that way: you have to act on things no longer in front of you. RNG-Bench forces that with two games. In Matching Pairs, card faces flash briefly and must be recalled later. In 3D Maze, the AI gets only a first-person view at each step and must stitch those glimpses into a mental map.

The clever part is the Memory Gap metric. It separates two reasons an AI fails: forgetting an earlier observation, versus remembering fine but choosing a dumb move. The verdict: most errors are forgetting. The AI's judgment is okay; its memory leaks.

The hardest versions feed a model 350 images and 128,000 tokens of context per game, and today's best models are nowhere near solving them. Fine-tuning a 9-billion-parameter model on examples of good play helped, and the gains carried over to other benchmarks without hurting general skills — though that's one model, not proof the fix scales.

Why you should care: Any AI you trust to handle a multi-step task — booking, navigating, tracking a conversation — has to remember what scrolled off the screen. This benchmark shows frontier models still leak that memory, and pins the blame precisely.