Update #2: Post training yandex/AliceAI-80B-A3B [instruct!] from scratch

An update on the yandex/AliceAI-80B-A3B instruct fine-tune progress details training a base model using a synthetic distilled dataset. The developer is excited to release the instruct version of this model, which was trained on 3, 32GB V100s. They have already open-sourced the distillation engine (SFTMill) and are willing to share custom kernels and other patches. The developer is also seeking remote or NYC-based dev or ML engineer roles.

This update provides a rare look into the training process of an 80B-parameter model on V100 GPUs, unlike most reports focusing on A100s or H100s.