Kog is going deeper to squeeze more inference out of GPUs

French startup Kog is aiming to significantly accelerate AI inference on conventional GPUs, despite the market's positive reception to specialized chips like Cerebras. Kog claims its technology can achieve "30x faster LLM inference." A demo showcased an impressive 3,000 per-request tokens per second (TPS) using the open-sourced Laneformer 2B, a purpose-built small model with approximately 2 billion parameters. This demonstrates Kog's approach to extracting more performance from existing GPU hardware.