Applying multirate DSP principles to LLMs: A hierarchical "Semantic Vocoder" architecture [P]

A discussion on applying multirate DSP principles to LLMs highlights issues with a hierarchical "Semantic Vocoder" architecture. The adapter transmits the semantic signal too efficiently, causing the base GPT to become over-reliant on the 384D semantic vector as a hash-key, rather than learning robust local grammar. This leads to an artificially high Top-1 accuracy of ~85% even with 15% Semantic Dropout, potentially causing exposure bias and repetitive loops during greedy decoding. The author seeks solutions for stabilizing similar hierarchical text models or applying more aggressive continuous noise injection.

This discussion uniquely details how over-efficient semantic signal transmission in a hierarchical "Semantic Vocoder" architecture leads to GPT over-reliance on 384D semantic vectors, unlike typical LLM discussions.