Who've told you that distributed training is impossible? Democratizing AI: The Psyche Network Architecture

AI 摘要

分布式训练并非不可能，且已有多项模型通过此方法完成训练。这种训练方式的关键在于参与网络的 GPU 数量，它能够有效避免因数据传输瓶颈导致的训练效率低下问题。

It seems that not only it is totally possible without incurring in unfeasible excessively narrow train data transfer bottlenecks but that several models have already been trained using this method. It mostly depends on how many GPUs join such kind of network.