Measuring AI Workloads

Chapter 7 connected an AI service’s bills to its access, throughput and budget constraints. A gateway or platform can count the calls that pass through it, but it cannot say which task a call served or whether anyone accepted the result. As I said in the Introduction, we did not build a discipline for assessing AI workloads and their return before generative AI arrived. This chapter sets out what that discipline has to measure, following the work behind the bill: which calls a task required, what each call consumed, and whether the result was accepted. It is the Introduction’s question about consumption applied to tokens: not only what they cost, but what behavior creates the consumption and what the economics reveal about the system underneath. Without that connection, a lower token bill can mean either a useful improvement or work that no longer gets done.

There is an old reason to follow the behavior rather than the price. In 1865 the economist William Stanley Jevons, writing about Britain’s coal supply, observed that improvements in the steam engine had increased the coal the country burned, because each one made steam power worth using for more work. “It is wholly a confusion of ideas to suppose that the economical use of fuel is equivalent to a diminished consumption,” he wrote. “The very contrary is the truth.”¹ Tokens are repeating the lesson. Epoch AI estimates that the cost of achieving a given level of AI performance has fallen about 47 percent a quarter since 2023, or 13 times a year.² Meanwhile Google’s monthly token volume went from 9.7 trillion in May 2024 to more than 3.2 quadrillion in May 2026,³ and researchers at MIT estimate that the price of running frontier models is rising 3 to 18 times a year, because the models are bigger and do more reasoning.⁴ A cheaper token invites more tokens, and more ambitious ones. A falling unit price says nothing about whether the total is buying useful work, which is why this chapter counts the work.

How to read this chapter

The useful measurement connects call-level consumption to accepted tasks. The proposed weighted index adds a view of the models being used, with substantial limits:

1. What the workload actually consumes — tokens per step, by model, with retries and rejected work included.
2. Describing the model mix — the weighted index, its arithmetic and the test it has to pass.
3. Choosing the benchmarks — comparable evidence and test conditions.
4. Who else is building this — related research and what it does not establish.
5. What the weighted index cannot tell you — three limits.
6. Workloads that mix models — planner, worker and coordination costs.
7. The purpose question: what no consumption measure can say — the audit questions.
8. Right-sizing: the cheaper ways to do the work — simpler and deterministic alternatives.
9. What the weighted index is for — the record a routing experiment needs.

The decision still turns on accepted work at an affordable cost; the index is useful only if it helps explain that result.

What the workload actually consumes

The foundation is unglamorous: you cannot weight what you have not counted. The counting artifact is a workload model — the same discipline Chapter 5 applied to web traffic, transposed from requests to tokens.

So the measurement begins as a table. For each kind of call in the workload, four facts: which model serves it, how often it happens, how many input tokens it typically consumes, and how many output tokens it typically produces. Where do the facts come from? Not from the vendor’s example payloads and not from guesses — from the organization’s own metering, at the current mix of work. Before launch there is no metering yet, so estimates from a pilot or test traffic fill the table until metered values replace them.

Here is a constructed example, separate from the book’s running portfolio, for 100,000 attempted tasks per month. A quarter of the tasks need the check step, so it runs 25,000 times. All rates are hypothetical, there is no caching, and output includes billed reasoning tokens.

Each monthly figure is attempted tasks × calls per task × tokens per call; check input, for example, is 100,000 × 0.25 × 5,000 = 125 million tokens. Total consumption is 625 million input plus 95 million output, or 720 million tokens. Suppose the route step’s model charges $0.10 per million input tokens and $0.50 per million output tokens, and the model behind the draft and check steps charges $2 and $10. In millions of tokens, the model bill is (100 × $0.10 + 10 × $0.50) + (400 × $2 + 80 × $10) + (125 × $2 + 5 × $10) = $1,915. That excludes subscriptions, tools, hosting and human correction. If 90,000 attempts are accepted, the model-only cost per accepted task is $1,915 / 90,000, or about 2.13 cents, against 1.92 cents per attempt: the 10,000 rejected attempts stay in the bill but leave the denominator. The same counts give 720 million / 90,000 = 8,000 tokens per accepted task, or 125 accepted tasks per million tokens in Chapter 7’s terms.

Open full-size diagram ↗

Keep distributions beside the averages. Mean token counts help estimate totals, but long inputs, retries and difficult tasks determine the expensive tail, and they tend to arrive together. Test a peak or a routing change with those relationships intact. Keep the model version and configuration too: a different model can consume different numbers of tokens on unchanged tasks.

Summing the calls gives consumption by model and billing category. That is a useful input to a cost forecast, provided cached input, uncached input and output are priced separately. It does not yet tell us how many useful tasks the service completed.

Connect the service metrics to the task

For a Foundry deployment, Azure Monitor exposes InputTokens, OutputTokens and ModelRequests, with dimensions including deployment, model and version. Request counts also distinguish status codes. Those observations let the team separate a change in traffic from a deployment change or a rise in failed calls. Microsoft Cost Management supplies the corresponding consumption charges, which Microsoft says can take about five hours to appear.⁵ Align the time windows and the billing treatment before reconciling the two; a provisioned deployment’s bill cannot be reconstructed by multiplying its tokens by a Standard deployment’s rate.

The application still has to connect those calls to the task. To the platform, a draft the application rejected and then retried is two successful requests. I would log a task identifier, step, deployment, attempt number and acceptance result with every call, under an appropriate policy for sensitive data. If a shared deployment’s token total rises, that log should tell us whether the assistant served more customers, read longer histories or repeated failed work. The platform metrics and the application log answer different parts of the question; together they make the call-path table explainable.

Describing the model mix

For this illustration, $T_k$ counts input plus output across all calls in the period, including retries, with billed reasoning included in output. In a real record, check which usage fields are totals and which are subsets: cached input and reasoning must each be counted once. Keep their subtotals for pricing. Different models can tokenize the same material differently, so equal token counts do not establish equal amounts of text or work. The input–output choice matters too: input was 625 of the 720 million tokens in the call-path example, so this convention mostly tracks how much context each model reads. An output-only index is also defensible and gives a different answer; pick one and keep it.

Dividing $B$ by total tokens, $T = \sum_k T_k$, gives benchmark intensity, the token-weighted average of the selected benchmark scores:

$$\varphi = \frac{B}{T}.$$

In the two-model example, $B$ is 1.3 million and $T$ is 2 million, so $\varphi = 0.65$. I would treat it as a model-mix indicator. A high value does not prove overbuying, and a falling value does not prove improved efficiency. Neither this index nor raw tokens defines a budget’s capacity for useful work; that requires measured accepted tasks and their cost.

Open full-size diagram ↗

I would compute the index one step at a time, among models that could genuinely do that step, with a benchmark matched to it; a single index across route, draft and check would grade the router on a drafting test. Even then it is an unvalidated proposal that has to earn its place. I would keep it in the monthly review only while it explains something the table of tokens and cost by model does not, and drop it after a few reviews in which it changes no question and no decision.

Choosing the benchmarks

Choose evidence that resembles the step under review. A coding evaluation and a tool-use evaluation test different behavior; a human preference ranking answers a different question again. Artificial Analysis and Epoch AI publish aggregate measures and methodology, which can help locate candidate models.⁶⁷ Keep them out of the multiplication. Elo-style ratings and Epoch’s index have arbitrary zero points — Epoch sets Claude 3.5 Sonnet at 130 and GPT-5 at 150 — so tokens multiplied by them, or ratios between them, mean nothing. A fraction of test cases passed has a true zero. Shortlist with the composites; put a pass rate from a matched test into the formula.

Three checks help.

Match the benchmark to the skill the step needs. Evaluate coding steps with coding evidence and tool-selection steps with tool-use evidence. A broad ranking can conceal a model’s strength on a narrow task; Epoch itself notes that highly specialized models may receive low scores on its index despite being very capable within their domain.⁸ If using the weighted index, calculate separate indices for each benchmark and version; do not sum coding and tool-use scores into a common capability total. Then test the candidate models on the actual step.

Check how the result was produced. Scores in a model provider’s own launch material are vendor claims: the seller chose the tests, the settings and what to publish. Read an independent evaluator’s methods, funding disclosures and uncertainty estimates with the same care. Look for replication where the decision is consequential, and preserve the model, effort setting, harness and test version. Artificial Analysis released four versions of its Intelligence Index in September 2026 alone, adding, removing and replacing evaluations, refreshing its judge panels and re-anchoring a rating scale.⁹ A small score difference may be noise, and disagreement between tests may reflect different skills or different methods.

Keep different skills visible. A profile of coding, retrieval and tool-use results is often more informative than a single average. If a composite is useful, document its normalization and weights; do not average incompatible scales. Changing the score definition requires rebuilding historical comparisons.

Every benchmark is a proxy for what your steps actually need, and the weighting is only as honest as the proxy’s fit. Use the best fit you can find and verify it against your own acceptance results.

Open full-size diagram ↗

Who else is building this

The closest related work comes from economics, from benchmark aggregation and from routing research.

Louis Yiven Zhu of the University of Oxford built quality-adjusted price indexes for AI inference by joining about 21,000 posted prices for 3,208 models to their benchmark results. Per token, quality-adjusted prices fell steeply. Per completed benchmark task, over the seven-month window the paper’s pre-stated rule selects, the buyer’s price stopped falling, because reasoning models raised token consumption faster than token prices fell. Zhu treats only the gap between the two rates as established; over thirteen months, the per-task series fell faster than the per-token one.¹⁰ The price of a token and the cost of a task can tell different stories, and the story can depend on the window. His validity audit holds a second warning: dropping benchmarks flagged for possible contamination barely changed the models’ capability estimates, which correlated at 0.998 with the originals, yet moved the index by about half a log point a year. A stable leaderboard does not guarantee a stable number to plan against. None of this validates the proposal here; Zhu estimates a latent capability scale to study market prices, while this chapter multiplies one organization’s token counts by a matched test score.

Epoch AI’s Capabilities Index stitches more than fifty benchmarks onto one scale and publishes its methodology, confidence intervals and data.¹¹ It covers only models with at least four benchmark results, and values can shift as new results are fitted. Use it to shortlist candidates for evaluation, not to convert a token budget into promised work.

FrugalGPT sends a query to cheaper models first and escalates when a learned scorer judges the answer unreliable; on its datasets it matched the best single model’s accuracy at up to 98 percent lower cost.¹² RouteLLM trains a router on human preference data to send each query to either a strong or a weak model, and reported cutting cost by more than half on public benchmarks without losing quality.¹³ Its authors note that cascades can add latency by querying several models, while a router sends each request to one. Both show up in the call-path table: a cascade’s escalations as a fractional row like the check step, a router as a step of its own. Neither paper supplies our acceptance rate; the enterprise version still needs its own tasks, routing policy and cost boundary.

What the weighted index cannot tell you

Three limits stay beside any result the index produces.

The benchmark is a prior, not a verdict. The score describes what a model did on someone else’s tasks under someone else’s conditions. Your acceptance criteria belong to your business — a model can score brilliantly on a reasoning index and still draft refund language your legal team rejects. The published indexes say this themselves;¹⁴ the weighting inherits the limitation rather than repealing it.

Acceptance forecasts need workload evidence. A claim that a route will preserve 95 percent of accepted work at 60 percent of the cost must come from representative evaluation, not from the weighted-token formula. If production acceptance falls below the forecast, investigate task mix, routing, tools, prompts and model behavior. Adjusting the weights until the index looks right would conceal the failure rather than explain it.

Routing has a bill of its own. Switching models can prevent reuse of a cached prefix. Include cache writes, reads, misses, router calls and escalation in the comparison. A saving calculated at one cache-hit rate does not survive automatically at another. Chapter 9 follows the other infrastructure costs around those calls.

Workloads that mix models

A mixed-model workflow might use a capable planner to divide the task, cheaper workers to execute bounded steps, and a verifier to check the result. The assignment is a hypothesis about which skills each step needs. Workers are not interchangeable merely because the plan is clear, and verification can itself require a capable model. Test the whole workflow against an all-in-one baseline, including failure and rework.

The call-path table already gives each role its own model and consumption. Report the planner’s share of tokens and cost as a role-based measure; do not define it as the share spent on the top model, since that model may also perform worker calls. Reducing planner spending might be sensible or might increase downstream rework. The share alone predicts neither acceptance risk nor savings.

Coordination adds calls: the planner writes instructions, workers consume them, and somebody reads and reconciles the results. In a hypothetical case, a $4 single-model task does not become cheaper when a $2 worker saving is accompanied by $3 of extra planning and checking: $4 − $2 + $3 = $5. Put every coordination call in the table and measure the resulting task, rather than assuming the worker discount survives the full run.

Chapter 9 takes up the topologies themselves — orchestrator-worker, fan-out and subagent isolation — and the machinery that runs them.

The purpose question: what no consumption measure can say

Databases are the closest analogy this book has for what consumption measures cannot see; the database world has lived with this gap for decades. A well-instrumented database deployment measures query efficiency — execution plans, buffer hit ratios, index usage, slow-query logs — and usage: connections, storage growth, table sizes. It almost never measures whether the data has a valid purpose. You can run a relational database against a workload that wants a document store, or keep four systems that each hold a drifting copy of the same customer table, and every efficiency metric on every one of them can look fine. Those discoveries come from audits and migrations, because the telemetry was designed to answer questions about performance.

A low cost per classified item can still hide a design that would work better with ordinary code or a smaller model. Neither token counts nor the weighted index can settle that choice. We have to test the alternative.

A February 2026 preprint compared fine-tuned BERT-family encoders with GPT-4o and Claude Sonnet 4.5, prompted zero- and few-shot, on four classification benchmarks: IMDB, SST-2, AG News and DBPedia. The encoders’ quality was competitive and often better, at one to two orders of magnitude lower estimated cost and with lower latency.¹⁵ A capacity planner should read the method before the headline. The encoder costs assume a serving instance kept busy, priced at January 2026 rates, and exclude training. At low or bursty volume, a self-hosted encoder either pays for idle capacity or accepts cold-start latency by scaling to zero. The study supports testing a smaller alternative. It does not establish the result for every classification workload, or make the training data, tuning and hosting effort free.

The capacity team makes the waste visible — per-task-class consumption, duplicate shapes by footprint, capability bought where a cheaper class would serve — and does not adjudicate purpose alone. Whether a workload has a valid purpose is the work’s owner’s question, informed by architecture judgment the capacity team should convene, not supply. The failure mode to avoid is the capacity team becoming a new bureaucracy that audits everything and understands nothing; the design that works surfaces the questions with numbers attached and hands them to the people who can answer them.

The boundary applies to every measure in this book: a measure cannot tell you whether the work it measured had the right architecture, the right approach, or the right purpose. Efficiency is fitness within an assumed design. The design itself is judged by people, with evidence the measures provide, in conversations the review structures, under accountability the organization assigns.

Right-sizing: the cheaper ways to do the work

The audit should test alternatives to the large generative models of Chapter 7 wherever they fit the task.

Open full-size diagram ↗

BERT-family encoders read tokenized text and produce representations that can feed a classifier or another task-specific output. They are language models too; the distinction here is between an encoder and a large generative model.¹⁶ A self-hosted classifier brings training, deployment and maintenance obligations, and a serving footprint that someone has to size. Include all of them in the comparison with API inference.

An embedding model maps text to numerical vectors for uses such as similarity search. It can support retrieval without generating a rewritten query, although the two operations are not interchangeable. The comparison needs the retrieval results, index cost and maintenance as well as the embedding charge.¹⁷ A simple rule-based router is another candidate where observable properties of the task reliably determine the route; keep an escalation path for cases the rules do not cover.

Distillation trains a smaller model using outputs from a more capable one.¹⁸ It can make a repeated task cheaper, but transfers neither every skill nor every safeguard automatically. Evaluate the smaller model and count the training, data and operating costs. Revisit the comparison as products and workload requirements change, without assuming each new small model will lower the total bill.

Some AI work is poorly described by text-token billing at all. Meta’s V-JEPA 2 learns representations from video, and its action-conditioned version planned robot-arm movements in labs whose robots had contributed no training data.¹⁹ The model still turns video into patch tokens and consumes compute; text input and output tokens are simply the wrong accounting unit for the task. A robotics workload needs its own compute, memory, latency and outcome measures.

What the weighted index is for

For the next routing change, I would want a record that says: these task classes, this call path, this observed cost, these acceptance results, and this proposed experiment, with the result that would make us keep the route or reverse it. The benchmark profile can explain why we selected the candidate. The weighted index can describe how our model mix changed. The decision to keep the route comes from what happened to accepted work, latency and cost when we tried it.

---

Notes and sources

1. W. Stanley Jevons, The Coal Question: An Enquiry Concerning the Progress of the Nation, and the Probable Exhaustion of Our Coal-Mines (London: Macmillan, 1865), 103; 2nd ed. (1866), 123, chap. 7, “Of the Economy of Fuel,” Library of Economics and Liberty, checked October 4, 2026.↩︎

2. Luke Emberson and David Roodman, “The Plunging Price of Thought”, Epoch AI, September 22, 2026, checked October 4, 2026. An earlier Epoch analysis found the rate varies widely with the level of performance, from 9 to 900 times a year: Ben Cottier, Ben Snodin, David Owen, and Tom Adamczewski, “LLM Inference Prices Have Fallen Rapidly but Unequally across Tasks”, Epoch AI, March 12, 2025, checked October 4, 2026.↩︎

3. Sundar Pichai, “Google I/O 2026: Sundar Pichai’s Opening Keynote”, The Keyword, Google, May 19, 2026, checked October 4, 2026: “Two years ago, we were processing 9.7 trillion tokens a month across our surfaces — a huge number. Last year at I/O, that grew to roughly 480 trillion tokens. Fast forward to today, that number jumped 7x to over 3.2 quadrillion per month.”↩︎

4. Hans Gundlach, Jayson Lynch, Matthias Mertens, and Neil Thompson, “The Price of Progress: Price Performance and the Future of AI”, arXiv:2511.23455, November 28, 2025, revised March 23, 2026, checked October 4, 2026. The same paper finds the price of reaching a fixed benchmark score falling about 5 to 10 times a year.↩︎

5. Microsoft, “Monitor model deployments in Microsoft Foundry Models,” https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/monitor-models, checked October 4, 2026. Vendor documentation of metric names, dimensions, diagnostic settings and Cost Management integration, including the note that cost data can take about five hours to appear. Available measures depend on model and deployment type; service metrics do not establish business acceptance.↩︎

6. Artificial Analysis, “Intelligence Benchmarking,” https://artificialanalysis.ai/methodology/intelligence-benchmarking, checked October 4, 2026. Independent evaluator’s methodology for Intelligence Index v4.3.2: ten evaluations in four weighted categories, language-model graders for some tests, and a stated 95 percent confidence interval below ±1 percent. The page says the index “may not apply directly to every use case.” Versions 4.2 through 4.3.2 are all dated September 2026; they added, removed and replaced evaluations, refreshed pairwise judge panels and re-anchored the GDPval-AA rating. Keep the version with any score used in a comparison.↩︎

7. Epoch AI, “Epoch Capabilities Index,” https://epoch.ai/eci, checked October 4, 2026. Methodology, confidence intervals and CC BY data for a composite of more than fifty benchmarks. The scale is arbitrary, set so that Claude 3.5 Sonnet scores 130 and GPT-5 scores 150; models need at least four benchmark results, values shift as results are added, and highly specialized models may score low despite strong domain capability.↩︎

8. Epoch AI, “Epoch Capabilities Index,” https://epoch.ai/eci, checked October 4, 2026. Methodology, confidence intervals and CC BY data for a composite of more than fifty benchmarks. The scale is arbitrary, set so that Claude 3.5 Sonnet scores 130 and GPT-5 scores 150; models need at least four benchmark results, values shift as results are added, and highly specialized models may score low despite strong domain capability.↩︎

9. Artificial Analysis, “Intelligence Benchmarking,” https://artificialanalysis.ai/methodology/intelligence-benchmarking, checked October 4, 2026. Independent evaluator’s methodology for Intelligence Index v4.3.2: ten evaluations in four weighted categories, language-model graders for some tests, and a stated 95 percent confidence interval below ±1 percent. The page says the index “may not apply directly to every use case.” Versions 4.2 through 4.3.2 are all dated September 2026; they added, removed and replaced evaluations, refreshed pairwise judge panels and re-anchored the GDPval-AA rating. Keep the version with any score used in a comparison.↩︎

10. Louis Yiven Zhu, “The Price of Intelligence: A Quality-Adjusted Price Index for AI Services,” August 30, 2026, https://arxiv.org/abs/2608.29843. Working paper: 21,024 posted-price observations across 3,208 models and 86 providers, February 2024 to August 2026, joined to 4,605 benchmark scores through a latent capability index. Per-task prices use token consumption measured once per model on a fixed benchmark suite. On the seven-month window the per-token index fell about 0.51 log points a year while the per-task point estimate rose 0.47; the 0.98 gap has an interval of +0.02 to +1.95, and the per-task sign itself is not established. The contamination exclusions come from the author’s own audit and measure dependence on those benchmarks, not proven contamination. Related measurement research, not validation of this chapter’s proposal.↩︎

11. Epoch AI, “Epoch Capabilities Index,” https://epoch.ai/eci, checked October 4, 2026. Methodology, confidence intervals and CC BY data for a composite of more than fifty benchmarks. The scale is arbitrary, set so that Claude 3.5 Sonnet scores 130 and GPT-5 scores 150; models need at least four benchmark results, values shift as results are added, and highly specialized models may score low despite strong domain capability.↩︎

12. Lingjiao Chen, Matei Zaharia and James Zou, “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance,” Transactions on Machine Learning Research, 2024, https://arxiv.org/abs/2305.05176. LLM cascade with a learned answer scorer; the up-to-98-percent saving is measured against GPT-4 on the paper’s datasets and API prices at the time, not on another workload.↩︎

13. Isaac Ong, Amjad Almahairi and colleagues, “RouteLLM: Learning to Route LLMs with Preference Data,” ICLR 2025, https://arxiv.org/abs/2406.18665. Routers trained mainly on Chatbot Arena preference data, evaluated chiefly between GPT-4 and Mixtral-8x7B on MT Bench, MMLU and GSM8K. The paper contrasts single-model routing with cascades that query several models. Reported savings do not establish another workload’s acceptance rate.↩︎

14. Artificial Analysis, “Intelligence Benchmarking,” https://artificialanalysis.ai/methodology/intelligence-benchmarking, checked October 4, 2026. Independent evaluator’s methodology for Intelligence Index v4.3.2: ten evaluations in four weighted categories, language-model graders for some tests, and a stated 95 percent confidence interval below ±1 percent. The page says the index “may not apply directly to every use case.” Versions 4.2 through 4.3.2 are all dated September 2026; they added, removed and replaced evaluations, refreshed pairwise judge panels and re-anchored the GDPval-AA rating. Keep the version with any score used in a comparison.↩︎

15. Alberto Andres Valdes Gonzalez, “Cost-Aware Model Selection for Text Classification: Multi-Objective Trade-offs Between Fine-Tuned Encoders and LLM Prompting in Production,” February 2026, https://arxiv.org/abs/2602.06370. Preprint comparing BERT, RoBERTa and DistilBERT with GPT-4o and Claude Sonnet 4.5 on IMDB, SST-2, AG News and DBPedia. API prices are a January 22, 2026 snapshot. Encoder costs are estimated from median latency on Google Cloud Run under sustained utilization, and training cost is excluded, so idle capacity and cold starts are not represented.↩︎

16. Jacob Devlin and colleagues, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” 2018, https://arxiv.org/abs/1810.04805. Introduces a Transformer encoder pre-trained as a masked language model and then fine-tuned for tasks such as classification.↩︎

17. OpenAI, “Vector embeddings,” https://developers.openai.com/api/docs/guides/embeddings, checked September 29, 2026. Vendor documentation: embeddings represent text as numerical vectors for similarity search and related uses; it does not establish retrieval quality on another corpus.↩︎

18. OpenAI, “Model Distillation in the API,” October 1, 2024, https://openai.com/index/api-model-distillation/. Vendor product announcement describing the use of a larger model’s outputs to fine-tune a smaller model for specific tasks; a particular deployment still requires evaluation.↩︎

19. Mido Assran and colleagues, “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” June 2025, https://arxiv.org/abs/2506.09985. Video representation learning and V-JEPA 2-AC robotic planning; does not establish a token-free computation model.↩︎