K² · Inteligencia artificial
Un ChatGPT que hace dos tareas a la vez rinde peor que uno que las separa
The authors formulate the state-prediction separation hypothesis, which proposes that disentangling a Transformer's next-token prediction role from its state-storage role yields better language modeling performance.
GM
NG
Giovanni Monea, Nathan Godey, Kianté Brantley et al.
4 autores · cs.CL, cs.AI, cs.LG
El vistazo de 30 segundos
Un modelo de lenguaje como el que hay detrás de ChatGPT hace dos cosas al mismo tiempo con la misma maquinaria: adivina la siguiente palabra y, a la vez, guarda apuntes internos que le harán falta para adivinar las palabras que vienen después.
- They designed a Transformer variant that uses two separate computation streams to separate the prediction function from the state-storage function.
- They conducted pretraining experiments across various scales to test the hypothesis.
- Their experiments show that state-prediction separation consistently offers better data and compute efficiencies.
Outperforms standard Transformers by 2 to 3 percentage points on average on downstream tasksTwo computation streams used to separate the prediction and state-storage functions