Back to the notes

AWS Trainium gets an open-source LLM runtime

Someone built TrnRT, an inference engine that makes AWS's AI chip accessible without vendor lock-in. First real alternative to NVIDIA for model serving.

Automatic fan controller for server racks (38933511880)
Dilshan Jayakody from Maharagama, Sri Lanka / Wikimedia Commons. Resized and converted to WebP. CC BY-SA 2.0

AWS Trainium chips have been out for two years but practically no one runs production LLMs on them. The tooling was AWS-only, the documentation was thin, and most teams stuck with NVIDIA because PyTorch just works. TrnRT might change that. It is an open-source inference runtime built specifically for Trainium. The interesting bit is not the chip support, it is that the project aims to match the Triton Inference Server API. If you have ever deployed a model with Triton, you already know the config format. That means you can test locally on NVIDIA, deploy to Trainium in production, and the same YAML files work. The README shows FP8 quantisation support and tensor parallelism across multiple Trainium cores. Those are table stakes for serving a 70B model at reasonable latency. What matters more is the vLLM-style continuous batching, which lets you pack more requests into the same hardware. AWS charges by the hour for Trainium instances. Better batching means you serve more tokens per dollar. The timing is deliberate. NVIDIA H100s are still hard to get and expensive when you do. Trainium is available on-demand in most AWS regions. If the performance numbers are real, this could be the first time a non-NVIDIA chip makes economic sense for LLM inference at scale. I am curious whether the quantisation actually preserves quality at FP8. Most papers show minimal degradation but production always finds the edge cases. The repo does not list benchmark numbers yet. That will be the test.


Source: Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium

Back to all notes

Behind the notes

Vikrant
Sharma.

Artificial Intelligence Engineer intern at Voxon Photonics in Adelaide. Studying a Master of Information and Communications Technology at UniSC, with a focus on data, machine learning and security.

Meet the person behind the work