LLM coding assistants have a variance problem

Dan Luu ran the same coding prompt through Claude twenty times and got wildly different results. Not small differences. One run would produce working code. The next would hallucinate APIs that do not exist. The variance between runs of the same model was larger than the benchmark difference between Claude 3.5 Sonnet and Claude 3 Opus. Which means comparing models on a leaderboard is less useful than we think. You are not choosing between a 72 percent model and a 68 percent model. You are choosing between two dice that both land anywhere from 40 to 90. This matches what I see when I use Copilot for Python data pipelines. Some days it generates clean Pandas transforms. Other days it suggests a column name I never defined. Same task, same context window, different outcome. Luu’s fix is to run the same prompt multiple times and pick the best result. Sounds obvious, but most people do not do it. We treat the LLM like a deterministic function. It is not. It is a probabilistic system with a temperature knob that never quite turns all the way to zero. The practical takeaway: if the output looks wrong, regenerate it. Do not assume the model is bad at the task. It might just be having a low-roll day. And if you are benchmarking models, test variance before you test accuracy. A model that averages 70 with a standard deviation of 5 is more useful than one that averages 75 with a standard deviation of 20.

---

Source: Agentic test processes, LLM benchmarks, and other notes on agentic coding fr