AI Drawing Arena: Evaluating GPT-5.6, Claude Fable 5, Gemini 3.6, and Grok 4.5 on Artistic Tool Use

Key Takeaways

• Autonomous Tool Integration: Models were given direct control over a digital canvas using a suite of tools including color selection, tip width, pressure sensitivity, smudging, and erasing.
• Frontier vs. Open-Weight Gap: The experiment highlighted a significant disparity in capability; while frontier models could attempt the tasks, open-weight models were largely unusable, often returning blank canvases.
• Performance Variations: Grok 4.5 struggled with the basic drawing tasks, whereas Claude Fable 5 demonstrated high capability but at a significantly higher cost and slower execution speed.
• Beyond Traditional Benchmarks: The 'Drawing Arena' serves as a visual indicator of model reasoning and iterative improvement (via the view_canvas function) rather than a standard numerical benchmark.

In-Depth Analysis

The Methodology of the Drawing Arena

The Drawing Arena represents a shift from static evaluations to dynamic, tool-based environments. In this setup, models are not simply generating an image via a single prompt; they are acting as autonomous agents within a workspace. Each model is provided with a blank white canvas and a specific set of colored-pencil tools. The process requires the model to manage multiple variables: setting the color, determining the tip width, and applying specific pressure.

Crucially, the workflow involves an iterative feedback loop. Models lay down batches of strokes, use smudging tools for blending, and employ erasers for corrections. The inclusion of a view_canvas function is a critical component of this analysis, as it allows the model to observe its own progress and make self-directed decisions on what needs to be fixed or refined. This simulates a human-like creative process, moving from initial sketches to finished reproductions of complex targets like the Mona Lisa and Van Gogh's Starry Night.

Performance Disparities and the Frontier Gap

The results of the 28 drawings conducted across four vision models—GPT-5.6 Sol, Claude Fable 5, Grok 4.5, and Gemini 3.6 Flash—reveal a clear hierarchy in the current AI landscape. The researchers noted that this task effectively "cuts through" the phenomenon of "benchmaxxing," where models are optimized specifically to score well on standard tests but fail at open-ended, fuzzy tasks.

Grok 4.5 was described as "rough" even at these basic drawing requirements. More tellingly, the open-weight models tested were unable to participate effectively, with several failing to produce any marks on the canvas. This suggests that the reasoning required to coordinate tool use with visual feedback remains a primary differentiator for frontier-class models. The report mentions that the upcoming Kimi K3 model will be evaluated once it is fully open-sourced to see if it can bridge this gap.

Cost and Execution Efficiency

A significant portion of the analysis focuses on the practicalities of running long-duration autonomous tasks. The experiment tracked not just the quality of the output, but the time and financial investment required for each model to complete a drawing. Claude Fable 5 emerged as a point of interest in this regard; while capable, it consistently took longer than its peers and incurred much higher costs. This data point is vital for developers and enterprises looking to deploy autonomous agents, as it highlights that frontier capability often comes with a trade-off in operational efficiency and budget.

Industry Impact

The Drawing Arena experiment underscores a growing trend in the AI industry: the move toward evaluating models as agents rather than just text or image generators. By forcing models to use tools and iterate based on visual feedback, the test exposes the "real frontier-vs-open gap" that numerical benchmarks might obscure.

For the industry, this highlights the importance of "frontier capability" in handling loose, open-ended tasks that require a high degree of coordination. It also serves as a cautionary tale regarding the costs of long-running tasks. As models become more integrated into creative and technical workflows, the ability to balance artistic output with computational cost and speed will become a key competitive advantage. The failure of current open-weight models in this arena suggests that while they may replace frontier models for simple execution work, they are not yet ready for complex, iterative autonomous tasks.

Frequently Asked Questions

Question: Why use a drawing task instead of standard AI benchmarks?

Standard benchmarks can often be "maxed out" by models optimized for specific test parameters. Drawing is a loose, open-ended task that provides a more interesting visual indicator of a model's true reasoning and tool-use capabilities, making it easier to see the difference between frontier models and the rest.

Question: How did the models interact with the canvas?

Models were given a set of tools to control color, tip width, and pressure. They could lay down strokes, smudge them to blend colors, and erase mistakes. Most importantly, they used a view_canvas function to see their work and decide on subsequent actions, allowing for iterative improvement.

Question: Which models performed the best and worst in this test?

While the full objective scores were not detailed for every model, the report noted that Grok 4.5 was "rough" at the task, and open-weight models were largely unusable. Claude Fable 5 was capable but noted for being significantly more expensive and slower than the other frontier models like GPT-5.6 Sol and Gemini 3.6 Flash.