A dataset with 52 Text to image model evaluation [P]
Heat trend
The percentage is based on available heat signal, not comment count or independent people.
I created a simple text to image benchmark.
I curated 192 prompts that are difficult for T2I models in various ways: text rendering, spatial reasoning, human realism, negations, etc...
I then asked a VLM to judge every output against a pre-specified binary question with the ground truth baked in.
I'm publishing all the results including the images. (Most public T2I leaderboards don't publish the actual images and that's a shame IMO)
There is currently 52 model tested! more than 9k images have been generated and analysed!
Full methodology: https://imagebench.ai/methodology-v1
Hugging face dataset: https://huggingface.co/datasets/dh7/imagebench (it contains the prompts to reproduce the results AND the results)
Github: https://github.com/dh7/image-bench-ai
Gallery to inspect the results: https://imagebench.ai/gallery
Leaderboard: https://imagebench.ai/imagebench-v1
Limitations: it's text to image only, and VLM are not perfect as a judge.
Let me know what could be useful from there!