Transformer attention has a working memory problem
Researchers found that transformer attention mechanisms lack the executive control functions that let human brains manage working memory. The models can retrieve information, but they cannot suppress irrelevant context.
Read the note 1 min read