OpenArch – PyTorch implementations of modern LLM architectures

OpenArch provides PyTorch implementations of modern open-source LLM architectures, written from scratch as a learning resource. It includes models like GPT-2 XL, Llama 2, Llama 3, OLMo 2, DeepSeek R1, Gemma 3, Mistral 3, Llama 4 Maverick, Qwen 3, Kimi K2, GLM 4.5, GPT-OSS, Grok-2.5, PaliGemma, and Qwen3, with sizes ranging from 1.5B to 1T. These implementations detail normalization, positional encoding, and attention mechanisms, and are based on publicly available papers and technical reports.