Microsoft Finds Sonnet 5 Can Raise AI Agent Costs

Microsoft's developer evaluation found that a cheaper AI model rate card can still produce higher agent bills when token consumption rises during enterprise coding tasks.

Its test compared Claude Sonnet 4.6 and Claude Sonnet 5 inside GitHub Copilot Chat in Visual Studio Code on Windows, using Microsoft AI model upgrade cost findings across architecture and SharePoint Framework work.

The result was not a simple upgrade story.

Microsoft found that Sonnet 5 had lower listed token prices, but the model sometimes used far more tokens to complete the same engineering prompts.

The evaluation gives CIOs and developer-tool teams a practical warning: model pricing alone does not show what an AI coding agent will cost in production.

Microsoft Tested 150 Agent Tasks Across 15 Scenarios

Microsoft said the assessment tested 150 agent tasks across 15 technical scenarios.

The company said engineers ran five executions per model per scenario, with each run checked against a Select gate and quality dimensions scored by an LLM judge calibrated for consistency.

The work covered two workload groups.

Microsoft tested Azure architecture design tasks grounded in Microsoft Learn documentation and SharePoint Framework upgrade tasks, including build-system migration from gulp to Heft and a legacy ESLint migration to flat configuration.

Microsoft said it calculated costs from actual per-turn token data using GitHub Copilot rates.

The calculation means AI agent bills are created by token consumption during execution, not by the headline price per million tokens alone.

Sonnet 5 Had Lower Rates But Higher Token Variance

Microsoft said Sonnet 5 had a 33 percent price reduction across token categories in the rate-card comparison.

Microsoft said input tokens cost $2 per million for Sonnet 5, compared with $3 for Sonnet 4.6.

It listed cached input at $0.20 rather than $0.30, and output tokens at $10 per million rather than $15; those vendor-provided prices were not independent performance benchmarks.

Those discounts did not remove usage risk.

In 12 architecture scenarios, Microsoft said Sonnet 5 used 12 times more tokens at the median than Sonnet 4.6.

Microsoft said one architecture run consumed 47 times the typical baseline token volume.

Microsoft reported that Sonnet 5 averaged $0.47 per run for the architecture tasks, compared with $0.54 for Sonnet 4.6, because the price discount was enough to offset the extra tokens in that context.

Quality did not move in the same direction.

Microsoft reported that both models completed the architecture tasks at a 75 percent Select-gate success rate.

Microsoft reported that Sonnet 4.6 scored 90 percent on the Idiomatic quality dimension across the nine scenarios where both models produced usable output, while Sonnet 5 scored 78 percent.

SharePoint Upgrades Made The Cost Gap Larger

The SharePoint Framework upgrade tests showed a different trade-off.

Microsoft said Sonnet 5 passed the Select gate in 100 percent of the SharePoint runs, compared with 60 percent for Sonnet 4.6.

In an upgrade from SPFx 1.21.1 to 1.22.0, Microsoft said Sonnet 4.6 failed all five attempts because it overrode the instruction and adopted version 1.22.1 from its Microsoft Learn grounding context.

That instruction-following improvement carried a higher execution cost.

Microsoft reported that token consumption reached a factor of 10 across the 15 code-upgrade runs per model.

Microsoft reported that Sonnet 5 cost $2.01 per run in those upgrade tasks, or 3.7 times the $0.55 median cost of Sonnet 4.6.

The evaluation also recorded an extreme outlier.

Microsoft said one Sonnet 5 run consumed 69 million tokens while using extensive web fetching to find undocumented migration steps.

The same run met 21 out of 30 strict evaluation criteria, but Microsoft said four out of five runs in each scenario did not reach that depth.

Microsoft's test also left a hard limitation for both models.

Configuration correctness remained at zero percent across all SharePoint Framework scenarios, and neither model completed the structural toolchain migrations involving gulp, Heft or ESLint.

Enterprises Still Lack Predictable AI Agent Cost Proof

The findings point to a procurement problem for companies adopting AI coding agents.

A model can be cheaper per token, stronger at following instructions and still less predictable in real operational cost.

Microsoft framed the issue through actual agent runs rather than static benchmark scores, which makes the evaluation useful for teams comparing model upgrades inside developer tools.

The disclosed test did not include customer deployment data, audited production bills, remediation guidance for token spikes, or a guarantee that the same results will hold outside the tested GitHub Copilot, Visual Studio Code and Windows setup.