By admin 
Both sets of results, while seemingly contradictory, are in fact real and defensible, stemming from a critical difference in experimental parameters: token and time budgets. These figures, often relegated to footnotes or hidden within technical specifications, are proving to be the linchpin in understanding the performance discrepancies. Alibaba’s own documentation reveals that its coding benchmarks were allocated a generous five-hour timeout and up to twelve hours per run on PaperBench. In contrast, the independent VulcanBench harness imposed a much tighter constraint, allowing between 45 and 60 minutes of wall-clock time for each test. This substantial disparity in time budgets, with Alibaba’s side enjoying a five to sixteen times larger allocation, directly explains the vast chasm observed in the reported performance metrics.
This widening gap underscores a pressing need for a paradigm shift in how we evaluate and select AI models, particularly for agentic computer use. Two fundamental adjustments are necessary. Firstly, the most relevant metric for assessing AI model performance in real-world applications is the cost per successful task. This metric accounts for the total expenditure, encompassing all attempts – successful or otherwise – divided by the number of tasks that definitively meet predefined acceptance criteria. It moves beyond simple per-token pricing or raw performance scores to reflect actual economic efficiency. Secondly, time and token budgets must be elevated from implicit details to explicit components of acceptance criteria. They should no longer be hidden variables that can disproportionately influence outcomes, but rather clearly defined parameters that users must specify and models must adhere to.
Price Per Token Has Stopped Predicting the Bill
In the initial flurry of analysis surrounding Qwen 3.8-Max’s release, the prevailing comparison was based on pricing, as it was the only readily available data point. This initial assessment revealed Qwen 3.8-Max to be a considerably expensive model. For instance, DeepSeek-V4-Flash-0731, which entered public API beta on July 31st, lists its pricing at a competitive 14 cents per million input tokens and 28 cents per million output tokens. In stark contrast, Qwen 3.8-Max is priced at $2 per million input tokens and $6 per million output tokens. Kimi K3, another competitor, sits at $3 per million input and $15 per million output.
However, these headline pricing figures are becoming increasingly unreliable indicators of the actual cost of operation, especially for reasoning-intensive models like Qwen. The underlying reason is specific to the internal workings of these advanced AI systems: achieving a coherent and accurate result often consumes a significant portion of the model’s "thinking" tokens. When a model dedicates the majority of its token allowance to internal reasoning processes, it can hit its token cap before it even begins to formulate the final answer. This results in an empty or incomplete output, functionally indistinguishable from a complete failure, yet incurred at the full cost of a complete model run.
Artificial Analysis has provided some of the clearest published measurements illustrating how this phenomenon impacts real-world agent spend. Their Intelligence Index, when run on DeepSeek-V4-Flash at maximum effort, required an astonishing 210 million output tokens, far exceeding the class median of 100 million. Despite this high token usage, the absolute cost remained relatively low due to the model’s exceptionally cheap token rates. However, this verbosity comes at a cost beyond just monetary expenditure; it also consumes valuable time. Depending on the specific use case, this time overhead can be a decisive factor in a project’s success or failure.
What is critically needed is a metric that encapsulates the entire investment, including all failed attempts and reasoning cycles, against the number of tasks that were successfully completed within the user-defined time and token budgets. This is precisely what the cost-per-success metric aims to illuminate, providing a far more realistic view of an AI model’s economic viability and practical performance.
Your Failure Rate Is Partly a Configuration Setting
The distinction between a model producing an incorrect answer and a model failing due to budget exhaustion is significant, as each scenario necessitates different troubleshooting approaches. Unfortunately, almost no AI model benchmark harnesses currently differentiate between these two types of failures, and consequently, very few leaderboards report this crucial split. This oversight was personally encountered during the development of an independent agent benchmark, where the harness simply logged a generic "failure" without any indication of its root cause. It became clear that this distinction needed to be explicitly added to gain meaningful insights. Upon implementing this differentiation, it became apparent that budget exhaustion, rather than outright incorrect reasoning, was the dominant factor leading to task incompletion.
A recent study, "Long-Horizon-Terminal-Bench," published in July, subjected seventeen frontier AI models to forty-six tasks using a shared harness, with each model receiving a single 90-minute attempt. The findings were revealing: timeouts accounted for a staggering 79% of unresolved runs. In contrast, agents that stopped execution independently represented 19% of failures, and harness errors constituted a mere 3%. The authors of the study are careful to qualify these results, noting that the timed-out runs were not close to completion, with mean rewards ranging between 0.10 and 0.35. Therefore, it is not definitively established that additional time would have guaranteed success. Nevertheless, the overarching lesson is clear: benchmarks inherently measure time efficiency, whether this aspect is prominently advertised or not.
Perhaps the most illustrative example of this underlying mechanism comes from VulcanBench, the same open-source harness that provided the contentious Qwen benchmark results. In a report dated July 26th, Claude Opus 5’s lowest-effort setting proved to be its most effective, successfully completing 20 out of 23 tasks. In comparison, its high-effort setting managed only 18 tasks. While the increased reasoning in the high-effort setting did yield fewer incorrect answers (one incorrect answer compared to three in the low-effort setting), it significantly increased the likelihood of clocking out. A timeout, in the context of these benchmarks, scores zero. Two of Claude Opus 5’s three regressions in the high-effort setting were specifically due to cutoffs on tasks that the lower-effort setting had successfully resolved. When granted unlimited time on both settings, the high-effort configuration only matched its cheaper counterpart, but at an exorbitant 3.1 times the cost.
This dynamic has direct implications for anyone designing routing ladders, a common architectural pattern in AI agent development. The standard approach involves escalating to a more powerful and presumably more capable model when an initial, cheaper attempt fails. This is predicated on the assumption that the next rung in the ladder is inherently better and simply incurs a higher cost. However, for a significant proportion of model and task combinations, this assumption proves to be fundamentally flawed. Instead of achieving better results, users end up paying the premium price for the more advanced model only to encounter the same or even worse outcomes, such as timeouts or hitting token caps, effectively escalating into a failure at a higher expense.
Who Is Already Measuring This
The growing recognition of the limitations of traditional benchmarking metrics is evidenced by the independent emergence of the cost-per-successful-task methodology across several research groups in recent months. This convergence of independent efforts strongly suggests that it is rapidly becoming the industry standard for evaluating AI model performance.
VulcanBench has consistently reported dollars per solved task as a prominent headline column in its benchmarks since its earliest publications, demonstrating a long-standing commitment to this practical metric. Similarly, Long-Horizon-Terminal-Bench provides per-task cost alongside accuracy figures. Its most insightful comparison highlights GPT-5.4, which incurs a cost of approximately $26 per task but achieves a significantly lower pass rate than Grok 4.5, which costs around $11 per task. TestEvo-Bench takes a different approach by running agents under a strict cost cap. This methodology reveals a stark drop in Claude Code’s test-generation score from 71% to 44% when subjected to a tighter budget constraint.
The adoption of the "cost per successful task" paradigm is not limited to academic research; leading vendors are also actively embracing this approach to better align their offerings with customer needs. HubSpot, for example, transitioned its Breeze Customer Agent in April to a pricing model of 50 cents per resolved conversation, a notable reduction from its previous $1 per handled conversation rate. Zendesk also employs a billing model based on automated resolutions, ensuring that customers only pay for tangible outcomes. Fin, a conversational AI platform, charges 99 cents per outcome, with billing triggered exclusively upon end-to-end resolution. This widespread adoption across various sectors signifies a clear industry consensus on the value and necessity of measuring AI performance based on successful task completion and economic efficiency, rather than solely on raw computational power or token-based pricing.
