The thing that's special about transformers is the attention mechanism, and the attention mechanism has this property that every single token has to interact with every other token. The amount of computation required scales with the square of the sequence length. So as you try to process longer and longer contexts, it gets quadratically more expensive.
1w ago
Underscored — save the words that stop you in your tracks.
