Transformers Explained Visually

A Transformer is a neural network architecture that utilizes a self-attention mechanism to integrate information across tokens. Unlike self-attention, the Multi-Layer Perceptron (MLP) within a Transformer processes tokens independently, mapping each token representation from one space to another. This process, represented by the formula QKV_{ij} = (\sum_{d=1}^{768} \text{Embedding}_{i,d} \cdot \text{Weights}_{d,j}) + \text{Bias}_j, enriches the overall model capacity.

This explanation uniquely details the Multi-Layer Perceptron's role within the Transformer, unlike many others that focus solely on the self-attention mechanism.