How are you guys actually benchmarking specific prompts? (Local vs. API, Cost vs. Quality)
一个开发者社区正在讨论如何对新的语言模型进行特定提示的基准测试。用户指出，通用基准测试对于他们的具体用例来说基本无用。他们希望测试精确的提示，以评估新的API是否物有所值，或者较小的本地模型是否足够好且更经济。该用户渴望了解其他人的工作流程和推荐工具。
- 发布
- 2026年9月7日 12:51
- 来源类型
- 开发者社区
- 档位
- 社区
- 信源状态
- 正常
时间以 UTC 显示
更多信息
With new models dropping every week, general benchmarks are basically useless for my specific use cases. I want to test my exact prompts to see if a new API is actually worth the cost, or if a smaller local model is good enough to run on the cheap.
Right now, I’m just eyeballing outputs and it’s driving me crazy.
How do you guys actually handle comparing models on a single prompt or a small test set?
Scoring: How do you define a "good" response when the output is subjective?
The Judge: If you use an LLM to grade the outputs, how do you stop it from just voting for its own writing style?
The Tools: What's the easiest way to fire one prompt at multiple models (both cloud APIs and local models) and compare them side-by-side?
Would love to hear your workflows or any tools you recommend!