Show HN: TurboGPT: train 22KiB transformer in 13s
TurboGPT is a tiny byte-level GPT training system implemented in CUDA C++ and released under the MIT license. It can train a 22KiB transformer in 13 seconds. Users can build it on Linux/NixOS using nix-build or on Windows with Visual Studio 2022 and CUDA 13.4 using .\build.ps1. Training runs store checkpoints and generate TensorBoard-compatible logs, with a reported result of 2.5295 BPB after 1.5G training tokens on hn1g.
Time & source
Times shown in UTC
Display time zone: UTC
Local time zone unavailable; showing UTC.
PublishedOffset at this time: UTC+0Sep 29, 2026, 19:20 UTC
IngestedOffset at this time: UTC+0Sep 29, 2026, 22:00 UTC
- Published
- Sep 29, 2026, 19:20
- Ingested
- Sep 29, 2026, 22:00
- Source type
- Dev community
- Tier
- Community
- Source status
- Healthy
Tier is a per-source editorial setting, not a per-item score.
Tiny byte-level GPT training in CUDA C++. MIT.
Build
Linux/NixOS:
nix-build -o build/nix-result
Windows, Visual Studio 2022 C++ tools, and CUDA 13.4:
.\build.ps1 - CudaArch 86
CudaArch is the GPU compute capability from NVIDIA's CUDA GPU list .
Run
.\build\ turbogpt.exe -- data hn1g.txt -- log - to runs / ctx4
The run stores its checkpoint at runs/ctx4/ctx4.pt, containing model, optimizer, scheduler, and trainer state. Use --load CHECKPOINT.pt to resume it.
runs/ctx4/report.json is derived from the log directory. Logs are TensorBoard-compatible: one report per batch, capped at 8Mi reports, and flushed with periodic or final checkpoints.
Result
- hn1g after 1.5G training tokens: 2.5295 BPB.
Tests
python tests\verify.py