Someone built a transformer in Rust using category theory
A programmer rewrote PyTorch's transformer architecture using Rust and abstract algebra. The result is dense but fast.
Someone just built a machine learning framework in Rust by treating neural network operations as categorical functors. It is the kind of project that makes you feel simultaneously impressed and confused. The author reimplemented a transformer model, the architecture behind GPT, using category theory primitives. Instead of tensors and backprop as first-class concepts, everything is a morphism between objects in a category. Forward passes compose functions. Gradients are dual maps. The entire computation graph is just arrows in a diagram. This is not how most ML engineers think. PyTorch and JAX abstract away the maths so you can stack layers without a PhD. This framework does the opposite. You write code that looks like a textbook proof, and the compiler figures out how to run it on a GPU. The performance numbers are decent. Training a small transformer on synthetic data hits comparable speeds to PyTorch once the JIT compiler warms up. Memory usage is lower because the type system can prove which tensors are dead and free them immediately. Rust’s ownership model maps cleanly onto automatic differentiation, so there is no reference counting overhead. The tradeoff is legibility. If you hand this code to a data scientist who learned ML from Kaggle tutorials, they will stare at it like hieroglyphics. But if you are building low-level inference engines or custom autodiff systems, the categorical abstraction might actually clarify what is happening. Functors compose. Side effects are explicit. The type checker catches gradient dimension mismatches at compile time. I would not train a production model with this. But I would read the source code to understand how transformers actually work, because stripping away PyTorch’s magic forces you to see the structure underneath.