Best models for generating red-team attacks? Also looking for public datasets [R]
这条记录涉及编程工具或代码能力更新，适合开发者评估工作流变化和可复用价值。
一位用户正在开发一个评估大型语言模型（LLM）应用和AI代理安全性的框架，并寻求关于生成红队攻击的最佳模型和公共数据集的建议。他想知道哪些闭源和开源模型能生成高质量、真实且具有挑战性的攻击，包括毒性、提示注入、SQL注入、越狱等多种攻击类型。此外，他还希望找到一个预定义的高质量攻击公共数据集，用于基准测试或验证AI代理的安全性，以避免从头开始生成所有攻击。…
Hi everyone, I'm currently working on a framework to evaluate the security of LLM applications and AI agents, and I've been stuck on one part for a while.
Most red-teaming frameworks rely on an LLM to generate adversarial prompts. My question is more about which model to use.
- Which closed-source models would you recommend for generating high-quality attacks?
- Which open-source models have worked well for you?
- Have you noticed any models that consistently generate more realistic or challenging attacks than others?
I'm looking for models that can generate attacks such as Toxicity, prompt injection, SQL injection, jailbreaks, indirect prompt injection, prompt leakage, tool misuse, multi-turn attacks, and other agent-specific attacks ect...
I also have another question.
Is there a good public dataset that people use to benchmark or validate the security of AI agents? I'd prefer a "golden" dataset with predefined, high-quality attacks rather than generating everything from scratch.
I'm curious about what people actually use in practice if you've worked on LLM security or red teaming, I'd really appreciate any recommendations, whether it's models, datasets, papers, or GitHub repositories.