#benchmarks

LLM coding assistants have a variance problem

Dan Luu measured the same coding task twenty times with the same prompt. The variance in output quality was higher than the difference between model versions.

Read post ↗