Benchmark your custom Pi tools
A new tool called RoastMyHarness has been introduced for benchmarking custom Pi setups. Developed by a data analytics professional, this work-in-progress tool helps users test their custom Pi tools to determine what is and isn't working effectively. It aims to assist Pi users in fine-tuning their tools, especially given the challenge of surpassing base Pi in task quality and token efficiency. The tool has primarily been tested on Linux.
Time & source
Times shown in UTC
Display time zone: UTC
Local time zone unavailable; showing UTC.
PublishedOffset at this time: UTC+0Sep 13, 2026, 01:40 UTC
IngestedOffset at this time: UTC+0Sep 13, 2026, 15:01 UTC
- Published
- Sep 13, 2026, 01:40
- Ingested
- Sep 13, 2026, 15:01
- Source type
- Dev community
- Tier
- Community
- Source status
- Healthy
Tier is a per-source editorial setting, not a per-item score.
A few people here mentioned interest in a way to test their custom Pi setups, so I figured I’d drop this here: RoastMyHarness
The basic idea is a small engine that sets up an environment to run DeepSWE benchmark tasks using bare Pi as a control and a variant of your choice, your Pi harness, an extension, a skill, AGENTS.md file, etc. I used a Pi extension to have a wizard set it up for you so its easy. run the same coding tasks against bare Pi and your modified harness, then look at what actually got better, what broke, and what it cost.
Since I like to play around with custom tools, I use this to get direction as to what is and isn't working. I know a lot here are making cool tools so I figured some might be interested in using it to help fine tune theirs. Any Pi users might be interested in figuring out if their tools are working like they think they should. You'd be surprised how hard it is to beat base Pi when it comes to task quality / token efficiency.
Its a WIP. I do data analytics by trade but otherwise a vibe coder and I only really tested it on Linux.