The thing that's interesting about transformers specifically is that you have this operation called attention, which requires you to look at all of the previous tokens every single time you generate a new token. So if you have a very long context, meaning a very long conversation or document, you have to do an enormous amount of memory reads every single time you want to generate even one new word. And that's actually a really hard problem.
The thing that's special about transformers is the attention mechanism, and the attention mechanism has this property that every single token has to interact with every other token. The amount of computation required scales with the square of the sequence length. So as you try to process longer and longer contexts, it gets quadratically more expensive.
1w ago
Underscored — save the words that stop you in your tracks.
