aws A field note by Vikrant Sharma
AWS Trainium gets an open-source LLM runtime
Someone built TrnRT, an inference engine that makes AWS's AI chip accessible without vendor lock-in. First real alternative to NVIDIA for model serving.
AWS Trainium chips have been out for two years but practically no one runs production LLMs on them. The tooling was AWS-only, the documentation was thin, and most teams stuck with NVIDIA because PyTorch just works. TrnRT might change that. It is an open-source inference runtime built specifically for Trainium. The interesting bit is not the chip support, it is that the project aims to match the Triton Inference Server API. If you have ever deployed a model with Triton, you already know the config format. That means you can test locally on NVIDIA, deploy to Trainium in production, and the same YAML files work. The README shows FP8 quantisation support and tensor parallelism across multiple Trainium cores. Those are table stakes for serving a 70B model at reasonable latency. What matters more is the vLLM-style continuous batching, which lets you pack more requests into the same hardware. AWS charges by the hour for Trainium instances. Better batching means you serve more tokens per dollar. The timing is deliberate. NVIDIA H100s are still hard to get and expensive when you do. Trainium is available on-demand in most AWS regions. If the performance numbers are real, this could be the first time a non-NVIDIA chip makes economic sense for LLM inference at scale. I am curious whether the quantisation actually preserves quality at FP8. Most papers show minimal degradation but production always finds the edge cases. The repo does not list benchmark numbers yet. That will be the test.
Source: Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium