Benchmarking AI Coding Agents: A Deep Dive into Tool Selection Across 17,000 Experimental Runs

Key Takeaways

• Rigorous Methodology: The study utilized a panel of 75 repositories across 10 languages, built from an analysis of thousands of GitHub repositories to reflect real-world tech stacks.
• Persona-Based Testing: Four distinct developer profiles (Vibe-coder, Junior, Senior, and Enterprise) were used to evaluate how agents respond to different levels of technical specificity and constraint.
• Unbiased Data Distribution: Researchers adjusted statistics from public repositories to ensure the testing environment reflected a mix of both startups and large enterprises.
• Extensive Variation: The experiment included 1,163 prompt variations, with 20-25% of cases specifically testing the impact of cost and usage volume constraints on tool selection.
• Controlled Execution: All experiments were conducted in dedicated ephemeral sandboxes to ensure clean, reproducible results for every run.

In-Depth Analysis

Constructing a Realistic Testing Environment

The foundation of this study lies in its data-driven approach to repository selection. Rather than using arbitrary code samples, the researchers performed a deep analysis of thousands of public GitHub repositories. This analysis extracted critical statistics regarding programming languages, frameworks, third-party service integrations, deployment platforms, team sizes, and the age of the codebase.

Recognizing that open-source data is often skewed toward tech startups, the team "unbiased" their statistics using publicly available data to create a distribution that accurately represents both agile startups and large-scale enterprises. The result was a curated panel of 75 repositories spanning 10 languages. To maintain the integrity of the experiment while ensuring realism, these repositories featured fake company names, fake git histories, and fake API keys, but utilized real lockfiles checked against official package manager registries like npm. This allowed the coding agents to interact with authentic dependency structures without compromising security or privacy.

The Four Persona Framework

A critical component of the research was the implementation of four distinct user profiles, each representing a different level of engineering expertise and organizational context. This allowed the researchers to measure how the "vague" or "precise" nature of a prompt influences the AI's tool choice:

1. The Vibe-coder: This profile describes symptoms and ideal states without using technical category names, testing the agent's ability to infer needs from high-level descriptions.
2. The Junior Engineer: This persona mentions the desired state and the specific category of tool needed, providing a baseline for standard technical requests.
3. The Senior Engineer: Characterized by precision, this profile details specific requirements and explicitly lists things to avoid, testing the agent's adherence to professional standards.
4. The Enterprise Engineer: This profile introduces complex constraints, including compliance, procurement, and specific organizational limitations, reflecting the high-stakes environment of large-scale corporate development.

By running 1,163 variations of tasks—such as implementing an email notification system for generated invoices—the study captured a wide spectrum of AI behavior. Notably, 20-25% of these prompts were modified to include mentions of cost or usage volume, specifically to observe if agents prioritize budget-friendly or scalable solutions when prompted.

Experimental Execution and the Runner

To ensure the validity of the 17,000 runs, the researchers developed a dedicated "Runner" system. Each experiment was executed in a dedicated ephemeral sandbox. This approach prevents cross-contamination between runs and ensures that the AI agent starts with a clean slate for every task. By using real lockfiles and package manager registries, the environment simulated a genuine development workflow, allowing the agents to "install" and configure tools as they would in a production setting. This level of environmental control is essential for determining whether an agent's tool choice is a result of its internal logic or a byproduct of the existing environment.

Industry Impact

This research marks a significant step forward in the evaluation of AI coding assistants. As tools like Claude, Codex, and Cursor become integrated into professional workflows, understanding their inherent biases in tool selection is paramount. The study's focus on "unbiasing" repository data highlights a growing need for benchmarks that reflect the diversity of the software industry, rather than just the most visible open-source projects.

Furthermore, the persona-based analysis provides valuable insights for prompt engineering. By demonstrating how different levels of detail and constraint (such as cost and compliance) affect the final output, the study helps developers and enterprises understand how to better communicate with AI agents to achieve desired architectural outcomes. The use of ephemeral sandboxes for large-scale benchmarking also sets a high standard for future research into the reliability and autonomy of AI-driven software engineering.

Frequently Asked Questions

Question: How did the researchers ensure the repositories were realistic?

They analyzed thousands of GitHub repos to extract stats on languages, frameworks, and team sizes. They then adjusted these stats to balance the representation of startups versus large enterprises. While names and histories were faked for the experiment, the repositories used real lockfiles verified against registries like npm.

Question: What was the purpose of using different personas like the "Vibe-coder"?

The personas were designed to test how sensitive AI agents are to the way a task is described. By varying the level of detail—from vague symptoms (Vibe-coder) to strict compliance and procurement constraints (Enterprise Engineer)—the researchers could measure how effectively agents select tools under different professional pressures.

Question: Were the AI agents tested on their ability to handle budget constraints?

Yes. In approximately 20-25% of the 1,163 prompt variations, specific mentions of costs or usage volume were added to the tasks. This was done to determine if the agents would change their tool recommendations based on the financial or scale-related requirements provided in the prompt.