Back to the notes

You can reverse-engineer the prompt from the output

A new paper shows you can reconstruct what someone asked an LLM by analysing what it generated. That is a privacy problem.

01equipment.jpg
Alice Wondrak-Biel / Wikimedia Commons. Resized and converted to WebP. Public domain

A paper from arxiv demonstrates you can work backwards from an LLM’s output to recover the original prompt. The technique uses previous-token prediction, the same mechanism that generates text, but runs it in reverse. The implications are obvious. If you are sending sensitive queries to a hosted LLM and someone intercepts the response, they can deduce what you asked. Medical queries, legal drafts, internal company questions. All potentially recoverable from the sanitised output. The method works because language models are probability machines. Each token depends on what came before. If you know the model weights and the generated sequence, you can infer the likely preceding tokens with reasonable accuracy. The researchers frame this as a security concern, not a theoretical curiosity. This matters for anyone building LLM-powered tools. You cannot assume the prompt is private just because you only expose the completion. If someone has access to the model and the output, they have a path back to your input. That includes API providers, cloud platforms, anyone with logging access. The obvious mitigation is end-to-end encryption, but that breaks most hosted inference setups. Local models solve it at the cost of capability. Differential privacy might help but adds noise you probably cannot afford in production. I am curious how this scales. The paper does not say whether prompt reconstruction works on multi-turn conversations or if context length breaks the attack. If it does work at scale, every chatbot with logging becomes a prompt harvesting tool.


Source: Previous-Token Prediction Based LLM Inversion for Prompt Reconstruction

Back to all notes

Behind the notes

Vikrant
Sharma.

Artificial Intelligence Engineer intern at Voxon Photonics in Adelaide. Studying a Master of Information and Communications Technology at UniSC, with a focus on data, machine learning and security.

Meet the person behind the work