The router everyone built then threw away

Manifest deprecated their LLM router after building it, shipping it, and watching customers not use it. The pattern is familiar: route queries to cheap models for simple tasks, expensive models for hard ones. Save money, keep quality. The router worked. It cut costs by 40 percent in their tests. Customers still turned it off. The problem was not accuracy. The problem was that routing added a hop. Every request hit the router first, then the model. That extra network call added 200 to 400 milliseconds of latency. For a chatbot, that delay is the difference between snappy and sluggish. Customers picked speed over savings. They would rather pay for GPT-4 on every query than wait an extra third of a second for the router to decide which model to call. Manifest tried optimising the router. They cached decisions, pre-warmed connections, moved the logic closer to the models. Latency dropped but never disappeared. The fundamental issue was architectural: one more thing in the path. The post does not say routers are bad. It says routers are a trade-off, and for their customers, the trade was wrong. If you are processing batch jobs overnight, 300 milliseconds per call is invisible. If you are running a customer-facing chat interface, it is the thing users complain about. I have seen this pattern before in data pipelines. You add a smart layer to save compute, then discover the smart layer costs more in coordination overhead than it saves in efficiency. The trick is knowing which bottleneck matters more. Manifest now lets customers call models directly. No routing logic. No latency tax. They still offer cost optimisation, but it happens at the model selection level, not the request level. Pick the right model for your use case, then call it fast. The lesson is not that routing is dead. The lesson is that latency beats cost optimisation when humans are waiting.

---

Source: Everyone is building LLM routers, we deprecated ours