Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

Research · September 3, 2026 · The Armature team

Disclaimer: Armature sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked.

As agents take over more and more parts of the coding journey, there is one specific part everyone outsources to their agent, from vibe coders with no software background to senior engineers: selecting which service to implement for a specific need in an existing codebase.

Let’s take the example of selecting a database:

- A vibe coder builds a personal travel app and realizes the app resets at each connection. They ask Claude Code:

I need you to store what I input in the app somewhere so that next time it’s still there when I reopen the app

Claude Code analyzes the codebase and answers 5 minutes later:

You need a database and Neon fits well because it has a free tier, is simple to install and won’t pause your app like Supabase does if you don’t use it too often.

The user accepts and the agents installs it. Done.

- A senior engineer working on a production app asks Cursor:

What is the best database solution for this app, it should have predictable costs and be fully managed.

5 minutes later, the conclusion is the same, Neon is the recommendation with clear reasons why competitors don’t fit the need. The engineer approves and the agent implements.

This is actually an experiment we ran. Two sandboxes, different agents, different codebases, different personas and prompts, same conclusion. So we wondered: if we generalize the test to other tool categories, making context / codebases / personas vary even more, will the result change?

It is an important question for developers trying to know if they can trust their agent’s judgement on what truly fits their needs. But it matters even more for vendors whose survival will soon depend on getting picked by coding agents (last April, Vercel shared that “ over 30% of deployments were initiated by coding agents, up 1000% from six months ago” ).

That’s why we decided to run the largest experiment ever to understand how coding agents think about tools, how they discover and pick them and which one ends up winning in each category. We watched almost 17k sessions across different types of personas (e.g., vibe-coders, junior engineers in startups, senior developers at enterprises) with 1,163 prompt variations, 75 repositories and 3 coding agents (Claude Code, Codex, Cursor) actually implementing the solutions instead of just recommending one.

Today we are sharing everything: aggregated results and leaderboards per category, but also every observation and even the entire traces with the user prompts, thinking traces and actual code diffs applied by the agent.

You can start exploring the results right here, or continue reading the article below

How did we run all these experiments concretely?

Our panel of repositories

We started by running an analysis over thousands of public GitHub repositories from which we extracted statistics about programming languages & frameworks, third-party services, deployment platform, team sizes, and codebase age. Since Tech startups are more likely to have open-source repositories than large enterprises, and stacks are likely very different we then unbiased our statistics based on publicly available data and reached our ideal panel distribution.

We then staffed various coding agents to create real-world repositories to match these exact requirements. Finally, we generated variants in which we removed parts of the codebases and with them, entire third-party service implementations so we could run proper unbiased experiments.

We landed on 75 repositories, in 10 languages, all using fake company names, fake git histories, fake API keys and real lockfiles checked against package manager registries like npm.

Real-world tasks

Each experiment is a real task to be performed inside a repository, asked by one of the following 4 profiles: