The thing that's interesting about transformers specifically is that you have this operation called attention, which requires you to look at all of the previous tokens every single time you generate a new token. So if you have a very long context, meaning a very long conversation or document, you have to do an enormous amount of memory reads every single time you want to generate even one new word. And that's actually a really hard problem.
1w ago
Underscored — save the words that stop you in your tracks.
