Continuous Diffusion Language Models (CDLM's)

Continuous diffusion models for language, after a period of dormancy, are experiencing a resurgence. While discrete diffusion methods previously dominated, recent developments suggest a shift. This comeback is marked by several papers published in late 2022, including Diffusion-LM, DiffuSeq 16, SSD-LM 17, Difformer 18, SeqDiffuSeq 19, GENIE 20, LD4LG 21, self-conditioned embedding diffusion (SED) 22, and continuous diffusion for categorical data (CDCD) 23. The approach involves lifting the corruption process from discrete input space into a continuous embedding space.