HyperSAE: Decoupled Poincaré Geometry for Sparse Autoencoders -- 9.8% MSE reduction, 0.2% dead latents on Gemma-2-2B [P]

I built HyperSAE, a PyTorch library that applies Poincaré hyperbolic geometry to Sparse Autoencoders for mechanistic interpretability.

GitHub: https://github.com/vishal-dehurdle/hypersae Paper: https://vishalvermalabs.com/papers/empirical-validation-hypersae-poincare-geometry/ Install: pip install hypersae

Motivation

Standard SAEs embed dictionary atoms in Euclidean space, where volume grows as O(r d.) The concepts LLMs learn form branching hierarchies that expand as O(b r.) At 16K+ dictionary sizes, this mismatch causes feature collisions at the boundary, dead latents, and reconstruction degradation.

Architecture

HyperSAE uses a decoupled dual-speed design:

- Forward pass remains entirely Euclidean. Zero inference overhead. Causal steering stays a single vector addition.

- During training, dictionary weights are projected into the Poincaré ball. An entailment cone loss organizes parent concepts near the origin and child concepts near the boundary, where hyperbolic volume expands exponentially.

Results (Gemma-2-2B Layer 13, 20M tokens FineWeb-Edu, NVIDIA L4)

Metric FlatSAE HyperSAE Delta Reconstruction MSE 4.5724 4.1232 -9.8% CE Loss Recovery 75.5% 78.9% +3.4pp Dead Latents 3.8% 0.2% -3.6pp MMLU-Pro Accuracy 16.11% 16.26% +0.15pp GPQA Diamond 100% 100% -- The library includes co-activation queue tracking, TriPartite loss (reconstruction + L1 sparsity + entailment), and a single-class trainer interface.

Feedback on the geometric formulation or benchmark methodology is welcome.