Knowing Your Workloads

That means asking what the system does, what stops it from doing more, and who can tell us what the business will need from it next.

How to read this chapter

This chapter is about characterizing work before sizing supply: what the machines are being asked to do, what drives that work, and what makes that description fit to use in a decision.

1. Begin with the work, not the machine — what a request contains, who sends it, and how that is changing.
2. Use simple arithmetic where it is useful — the utilization law, and how far a changed mix moves the answer.
3. Look at utilization, waiting, and errors — the USE method, locks and imposed limits, and cache sensitivity.
4. Test the arrivals you expect to receive — open and closed load, coordinated omission, and Little’s law.
5. When scale changes the system — the Universal Scalability Law and the limits of a fitted curve.
6. The apparently idle resource — deciding before the evidence is complete.
7. A record someone can challenge — an evidence record a specialist can dispute line by line.

The lesson to carry: a measurement earns its meaning from the work it describes and the conditions of the test, not from the machine it was taken on.

Begin with the work, not the machine

Our constructed portfolio starts from 250 completed requests per second per worker, with p95 latency no greater than 200 milliseconds and errors no more than 0.1 percent of offered requests: a synthetic fixture, assumed to come from representative tests under a particular request mix.

Consider everything hidden inside “a particular request mix.” Fetching a cached description, updating an account, searching years of history, and initiating work in half a dozen other services can all appear in a report as requests, so a stable request count can conceal a considerable change in the work. Before asking how many requests we expect next month, I want to know whether they will resemble the ones we have measured.

Brendan Gregg summarizes workload characterization as four questions: who is causing the load, why it is being called, what it consists of, and how it is changing over time.¹ For capacity planning, the first and last are business questions as much as technical ones. Telemetry knows who sent last month’s requests; Product may know that next month’s will come from different customers.

Suppose the staged rollout reaches customers with much larger histories than those in the pilot. Product’s estimate of 2,000 additional requests per second and Engineering’s successful test may both be accurate, and we still need a test containing the new customers’ operations and data. A safety factor on the old result would produce another number without teaching us anything about the change.

The why matters too. Two deployments of the same messaging software can carry similar ordinary traffic while one sends routine notifications and the other a holiday campaign whose messages are worthless a day late. I would rather ask the business owner which message becomes useless if it arrives tomorrow than spend another week measuring an interval in which nobody needed that answer.

Use simple arithmetic where it is useful

Some questions get a useful first answer from very little machinery. Take a hypothetical, deliberately simple workload: stateless, CPU-bound tasks on interchangeable cores, with a tested demand of 0.02 CPU-seconds per task, measured as CPU-seconds consumed divided by tasks completed over the same interval. The utilization law of operational analysis says that utilization, counted here in busy cores, equals throughput multiplied by that demand, $U = XD$.² At 1,000 tasks per second, average CPU demand is:

$$1{,}000\ \text{tasks/second}\times0.02\ \text{CPU-seconds/task} =20\ \text{cores}.$$

At a planning utilization of 60 percent, the first estimate is 20 / 0.60 = 33.33, rounded up to 34 cores. Under textbook queueing assumptions, one server that is 60 percent busy makes the average task wait 0.6 / (1 − 0.6) = 1.5 service times; our 34 cores, sharing one queue and about as busy, leave fewer than one task in 300 waiting at all.³ A percentage that is cautious for a pool already means real waiting for a single lock, disk, or partition consumer.

The same arithmetic shows how far the web service’s request mix can move before Chapter 4’s plan fails. With the borrowed batch workers, the service keeps 13,500 requests per second after losing a failure group, against a staged estimate of 12,000; since 13,500 / 12,000 = 1.125, the margin disappears if the average request needs 12.5 percent more of whatever limits a worker. Suppose history searches are 5 percent of the tested mix at 22 milliseconds of CPU each, against 2 for everything else: an average of 0.95 × 2 + 0.05 × 22 = 3.0 milliseconds. If the first stage’s customers push searches to 7 percent, the average becomes 0.93 × 2 + 0.07 × 22 = 3.4. A CPU-limited worker would then qualify at about 250 × 3.0 / 3.4 ≈ 221 requests per second, probably less once longer searches stretch the latency tail, and 54 of them carry about 11,900.

Arithmetic is most valuable when its answer is lopsided, far above the proposed allocation or comfortably inside it. When a plausible change is bigger than the margin, as a two-point shift in searches is here, its job is to name the test to run before the launch. My objection comes when a preliminary estimate survives into a presentation with all the confidence of a result and none of the conditions that made it provisional: that the work parallelizes, that a shared database keeps up, that a queue spreads the work evenly.

Look at utilization, waiting, and errors

The method extends to software resources and imposed limits: a lock’s utilization is the time it was held, its saturation the threads queued for it. In a cloud account, the limit that saturates is often an allowance, such as a container’s CPU limit or a disk’s provisioned IOPS, which is one way a machine can look idle while its workload is throttled.⁵

For a proposed change, follow the customer’s work alongside the resource measurements. If completed work stays flat while the queue grows, where is that work waiting? If latency rises while CPU remains low, what consumed the elapsed time? An error rate can even improve because more work is turned away before it reaches the boundary we measure. Those questions need the application engineer and the database or platform specialist in the same conversation.

PostgreSQL exposes this kind of saturation directly. Each session in pg_stat_activity reports the event it is waiting for, if any: I/O, a heavyweight or lightweight lock, a buffer pin, or the client, among others.⁶ Sampled every second or so through a busy period, this query shows where active sessions spend their time:

SELECT wait_event_type, wait_event, count(*) AS sessions
FROM pg_stat_activity
WHERE backend_type = 'client backend'
  AND state = 'active' AND pid <> pg_backend_pid()
GROUP BY wait_event_type, wait_event
ORDER BY sessions DESC;

A null wait type usually means the session is on a CPU or queued in the operating system for one. If most sessions wait on Lock, another core will not shorten the wait. We need the transaction holding the lock, which Chapter 12 follows into schema design and application behavior, a long way from the smaller machine the meeting set out to buy.

I use data-bound workload for work whose limiting behavior depends materially on data access, movement, layout, state, or coordination. Disk-bound would be too narrow, as a small cache example shows. At 10,000 logical page requests per second and a 99 percent hit rate, assume each miss requires one physical read and hits require none. Physical read demand is:

$$10{,}000\times(1-0.99)=100\ \text{reads/second}.$$

Suppose reducing memory moves the hit rate to 95 percent. The new demand is 10,000 × (1 − 0.95) = 500 reads per second: five times as much physical I/O from a four-percentage-point change. If our storage path can sustain 300 reads per second at the required quality, an apparently modest adjustment has crossed the service’s limit.

These are constructed numbers; a real investigation has to establish the cache behavior and the storage limit, and PostgreSQL adds a trap. Its blks_hit counter includes only hits in its own buffer cache, and its I/O statistics cannot tell a kernel page-cache hit from a read that reached storage. The documentation suggests starting shared_buffers at a quarter of a dedicated server’s memory and doubts that more than 40 percent helps, because PostgreSQL also relies on the operating system’s cache.⁷ Shrink the machine, and some of the cache we remove may be one the database’s hit ratio never counted.

Test the arrivals you expect to receive

Whether waiting is a flaw depends on the work. In a closed model, a fixed population of clients each waits for a response, and perhaps a pause, before sending again; in an open model, work arrives on its own schedule. Schroeder, Wierman, and Harchol-Balter found a vast difference in behavior between the two.⁹ An employee working through a form is roughly closed; devices reporting on a timer and customers arriving from a campaign do not slow down because the service has. Either way, offered, started, completed, rejected, timed-out, and skipped work need to stay distinguishable, or the workload can appear to improve just as the experiment becomes less demanding.

Azure Load Testing runs JMeter and Locust scripts at scale, and both tools are closed by default. Microsoft’s sizing guidance gives the rate a test can generate as virtual users divided by latency: 2,000 users produce about 100,000 requests per second at 20 milliseconds, and only 10,000 at 200.¹⁰ That is Little’s law: the average number of requests in a system equals throughput multiplied by the average time each spends there, here latency plus any scripted think time.¹¹ It is also a quick check on any test report, whose concurrency, throughput, and latency have to reconcile. For a launch question stated as an arrival rate, I would use an arrival-rate design where the tool has one and add a fail criterion on requests_per_sec beside those on p95(response_time_ms) and percentage(error), so that a run that fell short of the intended rate fails instead of passing quietly.

For our web service, the report needs the latency distribution, the rate actually offered, and the successful completions together, with the full histograms kept. HdrHistogram records distributions that can be added together; percentiles cannot, and two runs’ p95s do not average to the p95 of both.¹² Once all that survives is a p95 on a slide, we can no longer combine results or investigate an unexpected interval.

When scale changes the system

The portfolio’s capacity arithmetic multiplies workers by 250, which Chapter 2 justified by treating every configuration as having passed the same test at its full size. Neil Gunther’s Universal Scalability Law describes what that leaves out. It gives the relative capacity of $N$ workers, their throughput divided by that of one, as

$$C(N)=\frac{N}{1+\alpha(N-1)+\beta N(N-1)},$$

where $\alpha$ measures contention, waiting for something shared, and $\beta$ measures coherency, the cost of keeping copies of shared state consistent.¹³ With both at zero, the law is our multiplication. Small coefficients are not small in effect: a contention coefficient of 0.01 gives 54 workers the capacity of 54 / 1.53 ≈ 35, and no number of workers can reach $1/\alpha = 100$ times the throughput of one. Any coherency cost produces a peak at $N=\sqrt{(1-\alpha)/\beta}$, beyond which more workers deliver less; with $\alpha = 0.02$ and $\beta = 0.0002$, it comes at 70 workers.

Gunther is careful about what the model can do. He asks for at least half a dozen measured load points, warns that the law “is not a crystal ball” and cannot foretell broken measurements or intrinsic pathologies, and calls its application to production data with mixed workloads a current area of research. Those cautions are where my concern begins. Coefficients fitted to one schema, data distribution, and request mix describe that workload. Why should they hold after the cache is exhausted, a partition becomes unusually busy, a rebuild competes with customer traffic, or the rollout shifts the mix toward searches that work the shared database harder? Holding back measurements to test a model’s predictions is useful; so is writing down which changes would make us stop using it.

For the web service, 13,500 is therefore a statement about the tested range and the tested mix, and the batch machines still have to become operationally equivalent within the two-day preparation window. I would accept the multiplication once tests at the new mix show per-worker throughput holding across the configurations we intend to use, with the shared database measured at the rate we intend to promise. Until then we have a promising option to qualify, and a tipping point to look for: the relationship that stops holding and the mechanism that changed it.

The apparently idle resource

Now we reach the situation that makes this work uncomfortable: we cannot complete the investigation before the decision is due, and somebody has already counted the apparently idle resource toward a savings target. The Cassandra cluster in the Introduction, at 1 to 2 percent CPU, is the expensive version of that mistake.

Consider a constructed data service with an infrequent, urgent rebuild. The operating team knows it can require a large allocation, but a recent data change has left them without a representative measurement. Ordinary utilization is low, the next consequential event is near, and although the allocation could eventually be restored after removal, nobody has established that it could be restored in time.

Two questions decide how much evidence the decision needs. Could we undo it faster than we would need to? If quota, supply, preparation, and qualification can restore the allocation within the warning we would get before the next rebuild, releasing it is an experiment we can watch and reverse; if not, it is one-way and should wait for evidence about the event it exposes. And could the missing evidence plausibly change the answer, as it could for the web service’s margin? Here neither answer favors releasing the allocation now.

I would preserve it for a bounded period and fund the test, with the rebuild owner explaining the obligation we are protecting. The decision needs to carry the cost of holding the resource, what remains unknown, and the evidence or date at which we will reconsider it. Useful work may be able to run there meanwhile, under an agreement to return the capacity when the rebuild needs it.

What I would resist is letting the exception become permanent because everybody agrees the technology is difficult. Somebody has to sponsor the investigation and own the retained capacity, or the issue needs escalation rather than another quiet extension.

Nor can we escape the judgment by inventing a probability. Suppose an optimization saves $2,000 a month, or $24,000 a year, and a relevant failure would cost $400,000. In a simple model with at most one such loss per year, an added annual probability of 0.06 would give 0.06 × $400,000 = $24,000 in expected annual loss. Nobody can tell us whether six percent is right or whether $400,000 captures the consequence, but the break-even turns an unanswerable question into one the rebuild owner can answer: would removing the allocation add more than about a one-in-seventeen chance per year of that failure?

The person responsible for the business may reject that exposure even when expected cost appears favorable; FinOps usage-optimization guidance likewise asks for value-versus-risk thresholds for each workload.¹⁴ The proposal should make that exposure specific enough that the person deciding knows what they are agreeing to carry.

A record someone can challenge

Another month of ordinary CPU averages would not change a line of that record. Having established what we need to know, we can choose the tools that will help us find it. That is a much better order than acquiring a dashboard and then trying to make the business fit the measurements it happens to display.

Notes and sources

1. Brendan Gregg, Performance Analysis Methodology, https://www.brendangregg.com/methodology.html, accessed October 4, 2026. Gregg notes that the general process predates his 2013 write-up, pointing to Raj Jain, The Art of Computer Systems Performance Analysis (1991). Attaching business-change intelligence is this book’s emphasis.↩︎

2. Peter J. Denning and Jeffrey P. Buzen, “The Operational Analysis of Queueing Network Models,” ACM Computing Surveys 10, no. 3 (1978): 225–261, https://doi.org/10.1145/356733.356735; Edward D. Lazowska et al., Quantitative System Performance (1984), chapter 3, https://homes.cs.washington.edu/~lazowska/qsp/, accessed October 3, 2026. The law holds for measured averages without distributional assumptions; it says nothing about latency.↩︎

3. Mor Harchol-Balter, Performance Modeling and Design of Computer Systems (Cambridge University Press, 2013), chapters 13–14, https://www.cs.cmu.edu/~harchol/PerformanceModeling/book.html, accessed October 3, 2026. M/M/1 and M/M/k results; the 34-core figure, about 0.29 percent, was computed for this chapter with the Erlang C formula. Real arrivals are often burstier.↩︎

4. Brendan Gregg, The USE Method, https://www.brendangregg.com/usemethod.html, accessed October 4, 2026. Gregg also notes a capacity-based definition of utilization; the method sets no universal utilization target.↩︎

5. Gregg, The USE Method, https://www.brendangregg.com/usemethod.html, accessed October 4, 2026, sections on software resources (including locks) and cloud computing (hypervisor and container limits). The provider examples are the author’s.↩︎

6. PostgreSQL 18 documentation, “The Cumulative Statistics System,” https://www.postgresql.org/docs/18/monitoring-stats.html, accessed October 4, 2026. The query ran on a disposable PostgreSQL 14 instance, using columns documented for version 18; it has not been run on PostgreSQL 18 or a production database.↩︎

7. PostgreSQL 18 documentation, “The Cumulative Statistics System” (blks_hit, kernel page cache) and “Resource Consumption,” https://www.postgresql.org/docs/18/runtime-config-resource.html (shared_buffers), accessed October 4, 2026. The cache arithmetic is a constructed model, not benchmark data.↩︎

8. Gil Tene, “How Not to Measure Latency,” QCon San Francisco 2012, https://qconsf.com/sf2012/dl/qcon-sanfran-2012/slides/GilTene_HowNotToMeasureLatency.pdf, accessed October 3, 2026; wrk2 README, https://github.com/giltene/wrk2, accessed October 4, 2026. The percentiles and the 39,500 requests recorded after a 10-second calibration period are from the README’s second example; the other request counts are this chapter’s arithmetic. wrk2 is an HTTP tool.↩︎

9. Bianca Schroeder, Adam Wierman, and Mor Harchol-Balter, “Open Versus Closed: A Cautionary Tale,” NSDI 2006, https://www.usenix.org/conference/nsdi-06/open-versus-closed-cautionary-tale, accessed October 4, 2026. The workload examples are the author’s.↩︎

10. Microsoft Learn, “What is Azure Load Testing?”, “Configure Azure Load Testing for high-scale load”, and “Load test configuration YAML”, checked October 4, 2026: supported tools, the RPS formula and 2,000-user example, and fail criteria. JMeter’s Open Model Thread Group is marked experimental; Locust’s wait time “can only constrain the throughput.” The 200-millisecond case is this chapter’s arithmetic. No Azure load test was run for this book.↩︎

11. John D. C. Little, “A Proof for the Queuing Formula: $L = \lambda W$,” Operations Research 9, no. 3 (1961): 383–387, https://doi.org/10.1287/opre.9.3.383. Quantitative System Performance, chapter 3, adds think time and recommends the law for checking the consistency of measurements. It relates averages, not percentiles.↩︎

12. HdrHistogram README, https://github.com/HdrHistogram/HdrHistogram, accessed October 4, 2026. Histograms are additive and can correct for coordinated omission when the intended interval is known; neither shows the workload was representative.↩︎

13. Neil J. Gunther, “Universal Scalability Law,” https://www.perfdynamics.com/Manifesto/USLscalability.html, accessed October 4, 2026. Formula, ceiling, peak, and the cautions quoted; first published in 1993 as the “super serial” model. The numerical examples and limits on extrapolation are the author’s.↩︎

14. FinOps Foundation, “Usage Optimization,” https://www.finops.org/framework/capabilities/usage-optimization/, accessed October 4, 2026. The guidance also warns about longer-cycle peaks such as month-end; it does not set thresholds for any particular workload.↩︎