The best AI image models on the standard test rank worst at making pictures from text

Here's an uncomfortable fact for AI image research: the methods that win on the field's favorite test tend to lose at the job people actually want.

Nearly every team building diffusion transformers — the AI systems that generate images by starting with noise and gradually cleaning it up — tests on the same task. It's called ImageNet class-conditional generation: you give the model a label like "golden retriever" and it produces a matching picture. Researchers tweak their methods, watch a quality score (FID, lower is better) improve, and call it progress.

But the thing most people want is text-to-image: type any sentence, get a picture. Teams skip testing this because they assume it's too expensive to train and measure.

The authors built NanoGen, a single toolkit that trains and scores models on both tasks — and switching from one to the other takes just 12 lines of settings, at similar cost. Then they trained 21 models and ranked them on each task. The rankings didn't just fail to match; they pointed opposite ways. The correlation ran from -0.377 to -0.580 (a perfect match would be +1, no relation 0, a perfect inversion -1). So a method that climbs the ImageNet leaderboard often goes nowhere — or backward — on text-to-image.

The study doesn't explain why the tasks reward different things; it shows that they do. Their fix, DiffusionBench, simply reports both scores together.

Why you should care: Years of "improvements" measured only on ImageNet may not transfer to the text-prompt image tools you actually use — this work shows that scoreboard can quietly mislead.