Quick summary · Chapter 8
Measuring AI Workloads
First editionThe essential ideas, decisions, and diagrams from this chapter. Follow the section links for the examples, calculations, and fuller argument.
First-edition visual companion · Selected from the current chapter · 8x10-inch PDF
The main idea
Ten million tokens sent through a highly rated model do not establish a known amount or quality of completed work. A smaller model may be sufficient for the task; a larger one may succeed where it cannot. Only the application’s acceptance results can settle the comparison.
This chapter also proposes benchmark-weighted consumption: an unvalidated, descriptive index that joins token counts to model benchmark scores. It can show a change in model mix that a raw token total hides. It cannot turn tokens into a measured quantity of intelligence or substitute for counting accepted work.
What the workload actually consumes
For an AI workload, the unit of consumption is not the user request. It is the model call. One task — a support question answered, a report drafted — typically triggers several calls: a cheap model to interpret the question, a capable model to produce the answer, another model to check the result, occasionally a retry when something failed. Each call consumes input tokens and output tokens on a particular model.
Describing the model mix
Start with a single benchmark version run under comparable conditions. For illustration, suppose two models score 0.85 and 0.45, where the score is a fraction of test cases passed. These are hypothetical results on one test, not universal capability ratings.
For each model, multiply its token count by that score, then add the products. If model consumes tokens and scores , the index is:
One million tokens at 0.85 contribute 850,000 benchmark-weighted tokens; one million at 0.45 contribute 450,000. This is an accounting convention proposed here, not an established measure of effective capability purchased.
An increase from ten million to twelve million weighted tokens is (12 − 10) / 10 = 20 percent growth in the index, not evidence of 20 percent more useful work. Repeating a prompt unnecessarily would raise the index too. A change in tokenization, benchmark version or task mix can also break the comparison. Publish those conditions alongside it.
Suppose routing moves work to a model with a lower benchmark score. The index falls for the same token count, making that model-mix change visible. It does not establish lost capability on this workload: the cheaper model may pass every task that matters here. Compare acceptance rates, correction effort and total cost before calling the routing change a saving.
Choosing the benchmarks
The purpose question: what no consumption measure can say
So the measurement design needs a second mechanism: an audit question set, asked per task class, by people who can judge the answers:
- Does this step need a model at all? Formatting, validation and deterministic transformations may be better served by ordinary code, whose compute and maintenance still cost money.
- Does it need this class of model? Compare a smaller model on representative examples, including difficult cases and changes in the label set.
- Does it duplicate another workload? Two teams may have good reasons for separate systems, but the reasons should survive being asked.
- Is the architecture appropriate? Repeated retrieval, excessive context and unnecessary agent loops are testable design choices.
