Back to the notes

Monospace fonts for LLM tokens, not characters

Someone built a tool that generates fonts where every tokenizer chunk takes the same visual width. Suddenly prompt length is readable.

An open laptop on a wooden desk beside a notebook and small plant
Kari Shea / Unsplash Unsplash License

I found a tool that generates fonts where every LLM token is the same width, not every character. This is the kind of sideways thinking that makes you stop and recalibrate. Most tokenizers chunk text into units that do not align with what you see. The word “token” might be one token. The word “tokenization” might be three. When you are debugging a prompt that hits a context limit, you count characters and guess. This tool renders the text so every token takes equal space on screen. Suddenly you can see where the bloat is. The author built this for GPT-4 and Claude tokenizers. You paste text, pick a model, download a custom font file. Open your prompt in that font and the visual width maps to token count. A 50-token prompt looks shorter than a 200-token prompt, proportionally. This matters when you are optimising prompts. I have written system messages that felt short but burned 400 tokens because of how the tokenizer split technical terms. Counting by hand is tedious. Guessing is wrong. A font that shows token density in real time is the kind of tooling that should have existed years ago. The implementation is straightforward. The tool queries the tokenizer API, measures token boundaries, then generates a font where each token glyph has uniform width. It is not pretty. It is not meant to be. It is a debug view. I would use this for system prompts and few-shot examples. Those are the parts of a prompt where token efficiency matters and you do not notice the waste until you hit a limit. A visual diff between character count and token count would catch padding early. One limitation: the font only works for the specific tokenizer you built it against. GPT-4 and Claude tokenize differently. You need separate fonts. Still, for anyone tuning prompts at scale, this is a faster feedback loop than pasting into a token counter every edit.


Source: Generate fonts where every LLM token is the same width

Back to all notes

Behind the notes

Vikrant
Sharma.

Artificial Intelligence Engineer intern at Voxon Photonics in Adelaide. Studying a Master of Information and Communications Technology at UniSC, with a focus on data, machine learning and security.

Meet the person behind the work