Generate fonts where every LLM token is the same width
A project aims to generate fonts where every LLM token has the same width, transforming a font and tokenizer into a token-space font. The ctok tool, an unofficial approximation of Claude's tokenizer, has limitations including a 256-step search limit and issues with invisible characters affecting token boundaries. Audit results show ctok versions v3, v4.7, and v4.8 have various count/width and semantic failures, indicating it does not reliably agree with Claude's actual tokenization. These issues remain unfixed, suggesting a need for more expert implementation.
This report details specific failures in ctok versions v3, v4.7, and v4.8, unlike other accounts that might only mention its unofficial status.
Time & source
Times shown in UTC
Display time zone: UTC
Local time zone unavailable; showing UTC.
IngestedOffset at this time: UTC+0Sep 27, 2026, 00:00 UTC
- Ingested
- Sep 27, 2026, 00:00
- Source type
- Unclassified
Full text isn't available here.
Read at source →