The gap between what LLMs can do and what people think they can do

Andrej Karpathy tweeted today that the gap in shared understanding of what large language models can actually do is widening. Not shrinking. Widening. This rings true. I see two groups pulling in opposite directions. One camp builds tooling around GPT-4 and thinks text generation solves everything. The other camp reads a hallucinated citation and decides the entire technology is theater. Both are wrong, but they are wrong in ways that make collaboration harder. The perception gap matters more in ML engineering roles than in pure software roles. If you are building a vector search pipeline or a RAG system, you need to know what the model will reliably do versus what it will do sometimes versus what it will never do. Misestimating capability in any direction breaks the system design. I have seen projects scope too narrow because someone thought the model could not handle multi-hop reasoning. I have also seen projects scope too wide because someone thought the model would learn domain logic from three examples. The Hacker News thread on this has 410 comments. Half of them are people describing totally different mental models of the same technology. One person says LLMs are stochastic parrots. Another says they are reasoning engines. A third says they are lossy compression. All three models predict different things about what you can build. This is not an abstract philosophy problem. If your mental model says the LLM cannot generalise, you will not try certain architectures. If your mental model says it always generalises, you will not add the guardrails that prevent garbage outputs in production. I do not have a fix for this. The gap exists because LLMs are capable enough to surprise people in both directions. The solution is probably just to run more experiments and log more failures. Vibes-based reasoning about capability does not survive contact with a test set.

---

Source: The gap in shared understanding of LLM capability is widening