Training transformers in seconds, not hours
Someone built a 22KiB transformer that trains in 13 seconds. The trick is not the model size, it is how they squeezed PyTorch out of the picture.
Read the note 1 min read
From the notebook
transformers.
5 notes on this topic.
Transformers without loops: the Inception architecture drops self-attention
A new transformer variant ditches the attention loop entirely. Faster inference, cleaner scaling, same performance.
Read the note 1 min read
Etched built an ASIC that only runs transformers
A startup ditched everything GPUs do well and made a chip that runs one architecture 20 times faster.
Read the note 1 min read
Transformer attention has a working memory problem
Researchers found that transformer attention mechanisms lack the executive control functions that let human brains manage working memory. The models can retrieve information, but they cannot suppress irrelevant context.
Read the note 1 min read
Someone built a transformer in Rust using category theory
A programmer rewrote PyTorch's transformer architecture using Rust and abstract algebra. The result is dense but fast.
Read the note 1 min read