AWS Trainium gets an open-source LLM runtime
Someone built TrnRT, an inference engine that makes AWS's AI chip accessible without vendor lock-in. First real alternative to NVIDIA for model serving.
From the notebook
8 notes on this topic.
Someone built TrnRT, an inference engine that makes AWS's AI chip accessible without vendor lock-in. First real alternative to NVIDIA for model serving.
The GLM-5.3 model on Cloudflare Workers AI was leaking KV cache state across user requests. Your chat might have seen fragments of someone else's conversation.
Someone built a browser tool that throttles ChatGPT to show you what inference feels like on different hardware configurations.
MLX now runs multi-node inference across Mac Studios. Apple proved you can scale transformer serving without spinning up cloud instances.
A startup ditched everything GPUs do well and made a chip that runs one architecture 20 times faster.
Three years after first commit, the project that made local LLMs possible ships a stable release.
Iroh built a system that splits LLM inference across volunteer nodes. The networking stack handles dropouts mid-inference. Wild.
An enthusiast loaded a 1T-parameter model into 768GB of Intel Optane DIMMs and got 4 tokens per second on a single GPU. Slow, but it worked.