How many repeated LLM queries are enough? Testing a pilot-based reliability protocol [R]
热度趋势
趋势数据积累中
百分比基于当前可用热度信号，而非评论数或独立用户人数。
Rankfor.AI 的创始人兼一篇新预印本的作者，测试了一项基于试点方案的重复 LLM 查询可靠性协议，该协议专门用于审计品牌推荐。这项研究使用了三个独立收集的语料库，涵盖了政治倾向问卷和基准稳定性。在 39 个预测单元中，有 37 个达到了预设的复制标准，另有两个为部分匹配，这表明了该协议的强大可靠性。外部验证材料已在 GitHub 上公开。
I’m the author of a new preprint on repeated-query auditing of LLM brand recommendations, and the founder of Rankfor.AI.
The practical question: how many times should we repeat a prompt before comparing results?
The paper applies generalizability theory: estimate variance components from a pilot, then calculate the repeat count needed for a chosen reliability target.
Tested the reliability predictions on three independently collected corpora covering political-orientation questionnaires and benchmark stability. Across 39 prediction cells, 37 met the prespecified replication criterion and two were partial matches.
The fixed iteration thresholds did not transfer. Other preregistered tests, including parts of the drift diagnostics, also failed. Those results are reported in the paper.
An important limitation i see is that these external corpora do not contain brand recommendations. They test the statistical machinery outside our original application which is independent replication on repeated brand-recommendation data remains outstanding.
I’d particularly welcome criticism of the pilot-based variance estimates and the reliability validation design. Does anyone know an independently collected brand-recommendation dataset with repeated identical prompts?
Preprint: https://arxiv.org/abs/2609.04047
External validation materials: https://github.com/Rankfor/rankfor-open/tree/main/research/dice-roll-method/external-validation