Tokenization: A Survey for Modern NLP [R]

A new comprehensive survey on tokenization in modern NLP has been released, compiled by 32 tokenizer researchers over approximately eight months. This survey covers various aspects including algorithms, evaluations, multilinguality, encodings, and theory. It also explores potential replacements for tokenizers, such as latent or visual tokenization, and discusses related topics like constrained generation, token healing, and tokenizer security concerns. The full survey is available at alphaxiv.org.

This survey is the first to cover every aspect of tokenization, including algorithms, evaluations, multilinguality, and potential replacements, unlike previous, less comprehensive studies.