attention-mechanisms.

Transformer attention has a working memory problem

Researchers found that transformer attention mechanisms lack the executive control functions that let human brains manage working memory. The models can retrieve information, but they cannot suppress irrelevant context.