AI IN QA WORKS. WHO PAYS THE TOKEN BILL?

QA LinkedIn has been entertaining lately. One person lets AI open a browser through Playwright MCP. Another says it explores flows, generates tests, repairs locators and reads logs by itself. Some demos finish in minutes. Some call it a “super-powered manual tester”. Others can already see a future where testers design strategy and AI quietly does everything else.

Nice. I am not being sarcastic here. Letting an agent inspect the state of a page and decide what to do next genuinely opens more interesting possibilities than copying the DOM into ChatGPT and praying.

But one part is usually missing by the end of the post:

How many tokens did it use?

No number of agent loops. No input, output or reasoning token count. No cache-hit rate. No mention of how many attempts were needed to produce the pretty video. No review time, accepted tests, or next month's bill.

We watch a car cross the street in three minutes and applaud its speed. Fuel price, payload, and whether a tow truck was needed at the destination can wait for the sequel.

One prompt is not one billable event

Suppose I ask an agent to test checkout with Playwright MCP. From outside, the prompt is one line:

Test the checkout flow, find edge cases and write Playwright tests.

Inside, the run may look like this:

1. Read requirements, source code and existing tests.
2. Open the page and receive an accessibility snapshot or DOM state.
3. Choose the next action and call a tool.
4. Receive another snapshot, console log or network result.
5. Notice a wrong turn, go back and try another locator.
6. Generate tests, run them, read the failure, fix them and run again.

Ten tool steps are not ten free clicks. At every round the model needs enough context to know where it is. Conversation history, tool definitions, source, page snapshots, logs and reasoning may all enter the next call again.

A 20-token user prompt can therefore produce hundreds of thousands or millions of tokens across one task. My instruction being short does not make the conversation behind it short. Ordering one bowl of pho does not mean the kitchen's only job is lifting the bowl onto a tray.

Reasoning tokens do not grow on trees either. When a provider bills reasoning as output, a long internal “thought” still costs money even if the interface does not show all of it. Add web search, computer use or separately priced tools and the bill gains lines beyond text tokens.

To be fair, Playwright and MCP are not inherently expensive. A focused workflow with bounded context, sensible stopping and effective caching can be cheap. A workflow that resends the repository, the whole accessibility tree and 40 old messages after every click will find another price point. The expensive part is model × context × loops × retries × implementation, not the letters MCP.

One study of agentic coding found that agent tasks consumed dramatically more tokens than ordinary code chat; even repeated runs of the same task varied by up to 30 times. More tokens did not guarantee better accuracy.¹

Another study found price-order reversals in 21.8% of model pairs tested: a model with a cheaper listed price cost more on the actual task because it generated more thinking tokens. Some pairs differed by as much as 28 times.²

Comparing $1/M input with $2/M input and declaring the first model twice as cheap is therefore premature. The price of a kilo of rice does not tell you the cost of dinner when one cook makes two bowls and another throws away eight attempts before serving one.

Price snapshot: 16 August 2026

The table uses API prices per million tokens in USD. It is a snapshot from the article's publication date, not a stone tablet. Providers change models and prices fast enough for a blog post left alone for a few months to become archaeology.

GPT prices come from official OpenAI documentation. GPT-5.3-Codex is a coding model for agentic work; GPT-5.6 Sol is included as the current frontier reference. If one GPT-5.6 Sol request exceeds 272K input tokens, that entire request is billed at twice the input price and 1.5 times the output price. The examples below accumulate 1M input across a loop while assuming each individual request stays below that threshold. Throw more than 272K into one request and the calculation changes.³

Claude Sonnet 5 currently costs $2 input and $10 output. Anthropic initially planned to raise it to $3/$15 in September, but amended the announcement on 10 August and kept $2/$10 as the long-term price. Sonnet 5 also uses a new tokenizer, so the same material may become roughly 1.0–1.35 times as many tokens as before depending on the data. The unit price can stay still while the bill works out how to move.⁴

Kimi K2.7 Code costs $0.19 for cache-hit input, $0.95 for cache misses and $4 for output. Much cheaper than the premium group, but it always runs in thinking mode. We are back at the same problem: cheap output is not enough; we need to know how much it emits and whether the task passes.⁵

DeepSeek changed prices on the day this article was written.

How much did DeepSeek raise its price?

The new prices took effect at 16

V4 Flash also raised cache-hit input from $0.0028 to $0.007 off-peak or $0.014 peak: 2.5 and 5 times respectively.⁶

That reveals another operational risk. Yesterday your spreadsheet showed lovely ROI because caching was almost free. Tomorrow the provider changes a price; the workflow has not changed one line of code, but its economics are now a different story. Model pricing belongs in monitored configuration, not copied into a proposal and stamped permanent.

Let us price one agent run

We need something to multiply, so here is a reference run:

• The complete loop consumes 1M input tokens.
• Output and reasoning together consume 100K tokens.
• The 1M input is accumulated over multiple requests, not one enormous prompt.
• Cache, tool fees, tax and retries outside the run are excluded.

This is an illustration, not an “industry QA average”. A real workflow needs numbers from usage logs. If there are no usage logs, that is precisely the problem this article is about.

Three usage levels:

• Pilot: 1 QA × 2 runs/day × 22 days = 44 runs/month.
• Medium team: 10 QA × 5 runs/day × 22 days = 1,100 runs/month.
• Large scale: 50 QA × 10 runs/day × 22 days = 11,000 runs/month.

Baseline results:

Every model looks cheap in the pilot. A hundred-odd dollars; the company can pay that. This is where demos win easily. Give the same workflow to ten daily users and GPT-5.3-Codex becomes $3,465 per month; GPT-5.6 Sol becomes $8,800. At 50 people, the numbers begin to resemble several salaries.

What about the $20, $100 and $200 plans?

This cannot be ignored when people use Codex or Claude Code through personal accounts. API tokens and subscriptions are two different mechanisms.

At publication, Codex offered Plus at $20/month, Pro 5x at $100 and Pro 20x at $200. Claude had Pro at $20, Max 5x at $100 and Max 20x at $200. Here 5x and 20x describe relative usage compared with the $20 plan, not a number of tokens deposited into a wallet.⁷

OpenAI publishes Codex limits as approximate local-message ranges in a five-hour window:

That tenfold range from 10 to 100 is not a typo. OpenAI says usage depends on the model, task size and complexity, local versus cloud execution, context, reasoning, tools, retrieval and caching. Local messages also share the five-hour window with cloud chat, and additional weekly limits may apply.⁸

Anthropic does not provide a fixed message range in its plan table. Claude Pro is the baseline; Max 5x provides five times the capacity per session and Max 20x provides 20 times. Sessions reset after five hours, weekly limits also apply, and Claude, Claude Code and IDE usage all come from the same bucket.⁹

How many tokens does a plan provide? The correct answer is there is no fixed token entitlement promised by the provider. A message that edits one small function and a message that asks an agent to read a repository, open a browser and retry do not consume the same quota. Dividing a subscription price by the API rate and announcing that a $20 plan “contains 6.7 million tokens” mixes two different products.

We can calculate a comparison point, but it must be called API-equivalent, not a subscription token entitlement. Using the 1M input + 100K output reference run:

This table answers only: “If I spent that amount on the API with this assumed token mix, how many runs would it buy?” It does not say Plus or Max gives exactly those tokens. A subscription may be more valuable than API usage if the work fits the quota, or less valuable if a session or weekly limit arrives just when the work matters.

By advertised capacity, the $100 plan is fairly linear: five times the $20 price for five times the usage. It mainly buys headroom. The $200 plan promises 20 times the usage for ten times the price, so its nominal capacity per dollar is better—but only when you use enough and do not hit some other limit first.

Subscriptions also make marginal cost look like zero. If all 44 pilot runs really fit within quota, the $20 plan is $0.45/run, the $100 plan is $2.27/run, and the $200 plan is $4.55/run. The denominator still needs to be accepted work:

Subscription cost per accepted task = monthly fee / tasks accepted before reaching a limit

If 44 runs do not fit, the choices are to wait for a reset, reduce the workload or pay more. Both Codex and Claude allow extra credits after included usage is exhausted; the excess returns to usage-based pricing. For $20 or $100 to be a true spending ceiling, do not buy extra credits, enable auto-reload, or accidentally fall back to an external API key.¹⁰

There is also scope. A personal plan is excellent for one QA experimenting, working interactively and setting a monthly cap. It does not become a production budget for shared CI by magic. OpenAI recommends API keys for automation in shared environments such as CI; Anthropic explicitly says Claude Pro does not include the Claude API. Ten QAs buying ten $20 plans means $200/month in subscription fees, but capacity is split across accounts and accepted results, review time and operations still need measuring.

How much can caching save?

Keep 1M input + 100K output and assume cache hits are billed at the table rate:

Caching is worth doing, but an 80% hit rate does not appear because someone ticked a box. Change a prefix, tool definition, context order or session branch and a cache miss can follow. Some providers also charge separately for cache writes. Put 80% in a proposal only when logs prove it, not because faith feels cheaper.

For 250K input + 25K output, multiply every number above by 0.25. For 2M + 200K, multiply by two. The relationship is linear here because output remains 10% of input and no individual request hits long-context pricing. Real agents are less obedient: difficult tasks can reason longer, retry and pull more context.

Tokens are still not ROI

Suppose a manual task takes 60 minutes. With AI, the tester spends 20 minutes reviewing. Seventy per cent of outputs are usable; after reviewing the remaining 30%, the tester still has to do the full 60 minutes by hand.

Expected human time with AI:

20 minutes review + 30% × 60 minutes rework = 38 minutes

Expected saving:

60 - 38 = 22 minutes/run

At a loaded QA cost of $15/hour, one run saves:

22 / 60 × $15 = $5.50

For the medium team running 1,100 times per month, expected labour benefit is $6,050. We are not finished. Someone still builds the workflow, monitors it, adjusts prompts, updates tools, handles CI failures and pays for supporting infrastructure.

Assume:

• Initial setup: 80 engineering hours × $30, amortised over six months = $400/month.
• Maintenance: 16 hours/month × $30 = $480/month.
• Logging, CI and supporting infrastructure: $200/month.
• Fixed cost: $1,080/month.

Net gain = labour benefit - API - fixed cost

The cheap models win beautifully. Hold the champagne.

I just forced every model to have the same 70% acceptance rate, the same 20-minute review and the same token count. There is no evidence guaranteeing that. Kimi or DeepSeek may be cheaper but need more reruns. GPT-5.6 Sol may cost more per run and pass more difficult tasks. Claude Sonnet 5 uses a different tokenizer, so the same document can produce a different token count. Change acceptance or review time and the entire ROI table moves.

The useful metric is not price per million tokens but:

Cost per accepted task = total cost of all runs / tasks reviewed and accepted

A $0.66/M model that needs five attempts for one usable test may lose to a $2/M model that succeeds once. An expensive model that saves a tester 40 minutes may beat a cheap one whose output needs 35 minutes of repairs. Token price is an input. An accepted result is closer to what the company is buying.

At pilot scale, the $1,080 fixed cost already exceeds the labour benefit of 44 × $5.50 = $242, before choosing a model. The pilot may still be worth running to learn and measure, but calling it positive ROI is a little magical. A pilot buys evidence. It is not evidence that production will make money.

Self-hosting: stop paying for tokens and start paying for GPUs

At this point someone will say the model is open weight, so self-hosting solves everything.

Half right.

Kimi K2.7 Code has one trillion total parameters and 32 billion active per token. DeepSeek V4 Pro has 1.6 trillion total and 49 billion active. MoE means each token does not traverse every parameter, and quantisation reduces memory, but the weights still need to live somewhere. “32B active” does not mean loading exactly 32B onto one card and waiting for the other 968B to grow by themselves.¹¹

An illustrative eight-H100 configuration at a public rental price of roughly $2.49 per GPU-hour costs:

8 × $2.49 × 730 hours = $14,541.60/month

That is GPU cost for running 24/7. No storage, network, spare capacity, monitoring, operations engineer or KV-cache capacity yet. It is a lower bound for seeing the scale of the bill, not a claim that eight H100s can run every model at your required long context and concurrency.

Compared with Kimi API at $1.35 per reference run, the GPU bill alone equals more than 10,700 runs per month. If the team uses 1,100, hiring a cluster so its GPUs can meditate most of the day is an inventive approach to saving money.

Research on self-hosting cost points to utilisation as the variable calculators often forget. On the same H100, low load can raise effective cost per million output tokens by 2.5–24 times; near-idle operation pays an even larger penalty.¹²

What about a smaller model on a workstation?

That is more practical. Qwen3-Coder-30B-A3B-Instruct has 30.5B total parameters, 3.3B active, and quantised versions better suited to workstations. It may handle narrow jobs such as generating skeletons, converting formats, classifying logs or suggesting locators. Just do not place a local 30B model next to a frontier model and assume equal quality because both return code with colourful syntax highlighting.¹³

Suppose a workstation costs $6,000:

• Depreciation over 36 months: about $167/month.
• Power and cooling: $80/month.
• Operations at 8 hours/month × $30: $240.
• Illustrative total: about $487/month.

Sounds cheap. If the workload is regular and the local model keeps a good acceptance rate, it is worth testing. At 44 runs per month, though, those costs alone exceed $11/run. Kimi API in the example is $1.35. The machine can do other work, so it is not a perfect comparison, but it is enough to kill “we already bought the GPU, therefore inference costs zero”. A GPU charges up front, eats electricity later and occasionally demands human care.

Self-hosting also brings data control, customisation and some protection from provider price changes. In return the team owns patching, capacity planning, uptime, security and model upgrades. It wins when the task is narrow enough, the model good enough, demand steady enough and the infrastructure well used. Self-hosting is not the cheap default. It is another operating model.

When 500 people send requests at 9 a.m.

There is a more irritating variable than average utilisation: demand does not arrive evenly.

An organisation may have developers, testers and BAs using the model, but most work the same office hours. Twenty-two days × eight hours is only 176 working hours in a 730-hour month. If the eight-H100 cluster stays on 24/7 for this workload, even being fully busy throughout office hours means time utilisation of only about 24.1%.

176 / 730 = 24.1%

The $14,541.60 GPU bill becomes $82.62 per occupied hour—4.15 times the cluster's advertised $19.92 hourly price. We can turn it off at night, of course, but the problem does not vanish. The system still needs peak capacity, while starting instances, loading weights, warming caches and scaling replicas becomes work for an operator.

Suppose 200 people start agent runs after the daily meeting. Each run generates 5,000 output/reasoning tokens across its loops. The minimum time to clear the burst can be estimated as:

Burst clear time = runs × generated tokens/run / aggregate output tokens/second

This is sensitivity analysis, not an eight-H100 benchmark. It excludes long-context prefill, tools, retries, network and time to first token. Agent loops also depend on one another: the model returns an action, then the browser runs, then browser state returns before the model thinks again. Batching increases throughput but does not remove the critical path of each run.

As arrival rate approaches service capacity, queue time grows rapidly. If requests arrive faster than they can be served throughout the peak, the queue only lengthens. A self-host benchmark therefore needs at least p50/p95 time to first token, time per output token, end-to-end latency, queue time, waiting requests, and input/output throughput at peak concurrency. Inference servers such as vLLM expose this family of production and capacity metrics.¹⁴

Latency has a price. Suppose 500 people lose an extra 15 minutes per day at a loaded cost of $20/hour, but only 30% of the wait truly blocks them because they can switch tasks for the rest:

500 × 15/60 hours × $20 × 30% × 22 days = $16,500/month

That is only the assumed productivity loss, not the full wage cost of all 15 minutes. It already exceeds the 24/7 eight-H100 rental example. Total economic cost is no longer $14,541.60 for GPUs; it has crossed $31,000 before operations, review and rework.

Free models have the same problem. OpenCode Zen currently lists several free models for a limited time, such as DeepSeek V4 Flash Free, but its documentation promises no throughput or token rate for the free route.¹⁵ If an internal benchmark finds 10 tokens/second for the free route and 50 for the paid route, a 5,000-token run takes about 8.3 minutes instead of 1.7. The 6.7-minute difference, at $20/hour with 30% blocked time, costs about $0.67/run. At 1,100 runs per month, “free” has created roughly $733 of illustrative waiting cost.

Can slow and free still be worthwhile? Yes, if the job runs in the background, nobody waits, it finishes before the deadline and quality is sufficient. No, if it sits inside interactive debugging, exploratory testing, agent-assisted test writing or blocking CI; each slow loop then follows the previous slow loop. Compare this instead:

Cost/accepted task = infrastructure + tokens + operations + waiting + rework

To decide between another GPU, a paid API and a free route, load-test the real peak-hour request distribution and measure p95 latency on real tasks. Average tokens per month tells you how large the bill may become. It does not tell you how many people are watching a blinking cursor at 9 a.m.

Before showing the workflow, show me the denominator

A demo completing in minutes proves the workflow can run. It does not prove the workflow should run 11,000 times every month.

If a team claims AI improves QA productivity, it should at least know:

• Average tokens and cost per run.
• Cost per accepted task.
• Acceptance rate and retry count.
• Review, repair and rework time.
• Actual cache-hit rate.
• Peak concurrency, p95 queue time, time to first token and end-to-end latency.
• Setup, maintenance, logging and infrastructure cost.
• Quality: which risks were tested, which bugs were found, and how many false positives appeared.

The last part matters because a cheap test checking the wrong thing is merely rubbish manufactured at an optimised unit cost. In the previous article, LLMs Don't Read Requirements Like Testers Do, I argued that output shaped like a test case may still have the wrong oracle. This article adds another problem: even when the output is correct, ask what it cost to produce.

I still want to use AI in QA. It genuinely helps extract documents, create drafts, explore flows, read logs and build automation faster. But once it becomes a production workflow, the demo screen has to make room for usage logs and a spreadsheet.

A working feature is good. But if every run leaves one QA cleaning the output and Finance cleaning the invoice, the ROI is not good. And ROI not good is not good.

References

• Playwright MCP for manual testing — LinkedIn
• AI-driven browser automation with Playwright MCP — LinkedIn
• Claims about Playwright MCP and maintenance — LinkedIn
• Playwright MCP, CLI and token efficiency — LinkedIn
• Playwright MCP exploring a site and writing tests — LinkedIn
• Claims about self-healing and test generation — LinkedIn
• GPT-5.3-Codex pricing — OpenAI Docs
• GPT-5.6 Sol pricing — OpenAI Docs
• Claude Sonnet 5 — Anthropic
• Claude Opus 5 — Anthropic
• Prompt caching — Claude Platform
• Kimi K2.7 Code
• DeepSeek V4 pricing
• User discussion of the DeepSeek API price increase — Reddit
• Codex plans, usage limits and credits — OpenAI Docs
• Choose a Claude plan — Claude Help Center
• Use Claude Code with Pro or Max — Claude Help Center
• Manage usage credits for paid Claude plans — Claude Help Center
• How Do AI Agents Spend Your Money?
• The Price Reversal Phenomenon
• Kimi K2.7 Code model card
• DeepSeek V4 model card
• Qwen3-Coder-30B-A3B-Instruct model card
• Lambda Labs disclosure with reference H100 rental price
• Beyond Per-Token Pricing: A Concurrency-Aware Methodology
• vLLM production metrics
• OpenCode Zen models and pricing