llm A field note by Vikrant Sharma
Why I started logging every LLM call through a local proxy
Most teams have no idea how many tokens they burn or which prompts actually work. Running your own proxy fixes that.
Someone on Hacker News asked a question I have been asking myself for months: if you do not run your own LLM proxy, how do you know what is going on? The answer is you do not. You send requests to OpenAI or Anthropic, get responses back, maybe log the final output. But the token counts, the retry logic, the rate limit errors, the actual cost per call? All invisible. I hit this two weeks ago while prototyping a RAG pipeline for threat intelligence extraction. I knew the embeddings API was being called, but I had no idea how often or what the batching looked like. The bill showed 2.3 million tokens. I could not explain where half of them came from. A proxy sits between your code and the LLM provider. Every request flows through it. You log the prompt, the response, the token count, the latency, the model version. You can rate limit by user, cache repeated queries, or block calls that exceed a token budget. You own the observability. The article argues for LiteLLM Proxy or something similar. I am using a Flask wrapper I wrote in an afternoon. It writes every call to a PostgreSQL table with the full prompt text and metadata. Not elegant, but it works. I can query it to see which few-shot examples are burning tokens without improving accuracy. The real benefit is not the logging. It is the moment you realise your application is retrying the same failed call six times because the error handling is broken, and each retry costs you thirty cents. You cannot see that without a proxy. If you are calling LLMs in production and you are not proxying the requests, you are flying blind. The first time you get a bill you cannot explain, you will understand why.
Source: If you don’t run your own LLM proxy, how do you know whats going on?
