Training transformers in seconds, not hours
Someone built a 22KiB transformer that trains in 13 seconds. The trick is not the model size, it is how they squeezed PyTorch out of the picture.
Read the note 1 min read
From the notebook
cuda.
2 notes on this topic.
