llm A field note by Vikrant Sharma
Simulating local LLM speed before you buy the GPU
Someone built a browser tool that throttles ChatGPT to show you what inference feels like on different hardware configurations.
Someone made llmspeed.dev, a site that simulates how fast a local LLM would run on hardware you do not own yet. You pick a model, pick your target hardware, and it throttles the output tokens to match real-world inference speeds. The clever bit is it uses public ChatGPT but artificially slows the response to match the tokens per second you would get on an M1 MacBook Air, a 4090, or whatever else you are considering buying. No downloads, no local install. This matters because LLM inference speed is the one metric no one publishes honestly. Every benchmark shows batched throughput or maximum tokens per second under ideal conditions. Nobody shows you the interactive latency when you are typing a prompt at midnight and waiting for the model to finish a sentence. That gap between 15 tokens per second and 80 tokens per second is the difference between usable and unbearable. I have been looking at used 3090s for local inference and every forum thread devolves into people arguing about VRAM versus memory bandwidth. This tool would have saved me three hours of reading Reddit speculation. You just try it. If 22 tokens per second on Llama 3.1 70B feels too slow for your workflow, you know a 3090 is not enough. The site is basic but the idea is good. It exposes the one number that actually predicts whether you will use local models or give up and pay OpenAI. Feels faster than it sounds is not a selling point when you are spending two thousand dollars on a GPU.
Source: Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware
