Most open-source AI detectors can't hold a 0.5% false-positive rate [P]
Heat trend
Collecting trend data
The percentage is based on available heat signal, not comment count or independent people.
Hugging Face model activity is surfacing — worth tracking for capability changes, ecosystem impact, and availability.
A study evaluating open-source AI detectors found that most cannot maintain a 0.5% false-positive rate. The detectors were tested using a consistent protocol, with results showing varying performance across models like "tropa-mini" (0.968 ROC-AUC) and "desklib/ai-text-detector-v1.01" (0.875 ROC-AUC). One of the six tested detectors was developed by the study's authors and released with open weights under Apache-2.0, with all methodology and datasets available on Hugging Face.
We needed to know where the open-source AI-detection field actually stands, so we ran every notable open detector through the same protocol.
Setup:
- Public data only: Jabarian & Imas 2025 (NBER), Liang 2023 TOEFL essays, a 1,060-text frontier set (GPT-5.x, Claude Opus 5, Gemini 3.x), 5,000 pre-LLM (2018) FineWeb pages as human pool
- Every model gets its threshold set on the same 6,930 human docs to a matched 0.5% FPR
- Then measure recall per group: raw AI, humanizer-paraphrased AI, frontier models
What surprised us:
- 4 of 6 models effectively can't reach 0.5% FPR. MAGE scores >0.9999 on 26% of ordinary human web text. The old OpenAI RoBERTa detector lands at AUC 0.31 - worse than coin flip on modern generators.
- Humanizer-paraphrased text is where everything collapses: best model catches 42%, second best 4%.
- All models flag non-native essays at a higher rate than they flag native essays. This is a fundamental flaw in the entire class of models, not just one.
model ROC-AUC raw AI humanized AI frontier models tropa-mini 0.968 93.2 % 41.6 % 33.6 % desklib/ai-text-detector-v1.01 0.875 83.9 % 4.0 % 1.8 % SuperAnnotate/ai-detector 0.824 0.5 % 1.4 % 0.6 % Hello-SimpleAI/chatgpt-detector-roberta 0.571 0.8 % 0.4 % 0.2 % yaful/MAGE 0.507 —* —* —* roberta-large-openai-detector 0.313 0.0 % 0.1 % 0.0 % More information and data on HF
\ recall at matched 0.5% FPR; * MAGE can't reach 0.5% FPR at any threshold (flags 26% of human web text with score >0.9999)**
Disclosure: one of the six is ours - we run a hosted detector and released this one as open weights (Apache-2.0). All datasets and the methodology are in the model card, so you can rerun everything: huggingface.co/wasitaigeneratedcom/ai-text-detector-small