Are there any interesting architectural innovations that we seem to be on the verge of for LLM models or AI models that might be a big deal? (Excluding maybe N-gram, since everyone is already well aware of that one)

A discussion seeks insights into architectural innovations for LLM or AI models, excluding widely known concepts like N-gram, quantization improvements, MTP, D-flash, and D-spark. The goal is to identify and explain potentially significant advancements that are not yet common knowledge within the developer community, focusing on their interesting aspects and potential impact.