Dust: Pretraining Transformers Without Backpropagation

Q Labs Research introduces "Dust," a zeroth-order optimization algorithm designed to replace backpropagation in pretraining transformers. Unlike traditional evolution strategies (ES) such as EGGROLL (Sarkar et al., 2025) that perturb weights and scale with population size, Dust perturbs activations. This method creates a "virtual population" by independently perturbing activations at every token, allowing a single forward pass to evaluate thousands of members per sequence, thus overcoming the cost and scaling limitations associated with materializing and evaluating individual members in weight-perturbing ES methods.