Data2prompt stuffs entire projects into LLM context windows

Someone built data2prompt, a CLI that takes your messy data science project and compresses it into a single text blob optimised for LLM context windows. It walks your directories, respects .gitignore rules, and spits out a formatted prompt with your code, notebooks, and documentation all inline. The timing matters. Claude 3.5 Sonnet ships with a 200,000 token context window. GPT-4 Turbo sits at 128,000. We finally have the headroom to throw an entire codebase at a model and ask it to debug, refactor, or explain without playing the game of “which three files do I paste first”. What makes this different from a recursive cat is the filtering. It skips binary files, handles Jupyter notebooks as rendered markdown, and lets you exclude patterns beyond .gitignore. You are not sending your venv folder or your 8GB parquet file into the void. The output is markdown-wrapped code blocks with file paths as headers, which most LLMs parse cleanly. The use case I see is pair programming with a model that has short-term memory but no filesystem access. Instead of pasting functions one at a time and losing context, you load the whole project once. The model can reference utils.py when it is debugging main.py without you explaining the import chain. One catch: 200K tokens is not infinite. A medium-sized repo with dependencies or large CSVs will still blow the limit. You still need to curate what goes in. But for the 80 percent case where your project is fifteen Python files and two notebooks, this tool turns “explain this codebase” from a multi-hour Slack thread into a single prompt. I would use this for code reviews on personal projects where I do not have CI set up yet. Load the diff, ask the model what breaks, see if it catches the off-by-one I missed. Faster than setting up a full IDE plugin.

---

Source: CLI tool that packages data science projects for LLM context windows