We benchmarked 24 LLMs against human writers on 475 creative writing prompts

A new benchmark has been released, comparing 24 large language models (LLMs) against human writers on 475 creative writing prompts. This benchmark uses a custom reward model trained on human preferences to predict what a large group of readers would prefer. The results indicate that the strongest frontier LLMs already outperform talented amateur writers, though professional writers still maintain a significant lead. The full benchmark and model outputs are available for review.