K2

K² · Artificial intelligence

One AI brain doing two jobs at once? Split it, and it learns better.

The authors formulate the state-prediction separation hypothesis, which proposes that disentangling a Transformer's next-token prediction role from its state-storage role yields better language modeling performance.

GM
NG

Giovanni Monea, Nathan Godey, Kianté Brantley et al.

4 authors · cs.CL, cs.AI, cs.LG

arXiv preprintArtificial intelligenceJul 2026 · ~65s read

The 30-second scan

A language model like ChatGPT is quietly doing two jobs with the same set of calculations.

  1. They designed a Transformer variant that uses two separate computation streams to separate the prediction function from the state-storage function.
  2. They conducted pretraining experiments across various scales to test the hypothesis.
  3. Their experiments show that state-prediction separation consistently offers better data and compute efficiencies.
Outperforms standard Transformers by 2 to 3 percentage points on average on downstream tasksTwo computation streams used to separate the prediction and state-storage functions
Read the originalPDF