Capacity is not a CPU count

Yesterday’s post took the long view — the biggest changes to cloud capacity in the last decade. This one is about the shortest conversation I have on a capacity team. Somebody reports that a system is “maxed out,” or asks for a bigger machine, and I have to explain that computers do more than just have CPUs.

The resource everybody sees

The CPU is the resource everyone sees. It is on the first page of every cloud console, every dashboard shows it, most autoscalers watch it, and it is usually the first limit a small system runs into. It is also one of the lowest levels of infrastructure you can think about. A machine has memory, storage, network, and limits on all of them, and a workload can run out of any one of them before it runs out of processor.

Different workloads are gated by different things

This is the sentence I keep having to deliver: some workloads are gated by memory, some are gated by disk, and other workloads are definitely gated by CPU. Gated is the right word for it. When a workload is gated by a resource, it stops getting faster when that resource runs out, and adding any other resource does nothing. A batch job doing heavy computation is gated by CPU. A cache that keeps touching more data than fits in memory is gated by memory. A system ingesting a firehose or copying large files is gated by disk. Same machine, three different answers, and only one of them shows up on the CPU graph.

Databases measure the combination

Databases sit one level up from this, and there the workload is usually a combination of those resources. One query costs a little of everything: it reads pages from storage, hopes most of them are already in memory, spends CPU on the rows it finds, and sometimes waits on a lock instead of any hardware at all. The database world has been measuring that split for so long that most of its headline metrics barely mention the CPU.

PostgreSQL ships a cumulative statistics system — pg_stat_database, pg_stat_io, and friends — that counts block reads, buffer hits, and writes per database. MySQL operators watch the InnoDB buffer pool hit rate: how often the engine finds a page in memory instead of reading it from disk. Amazon RDS for PostgreSQL publishes wait events, and AWS’s own documentation describes IO:DataFileRead as a connection waiting “to read a required page from storage because the page isn’t available in shared memory,” adding that if the event “occurs frequently, your shared buffer pool might be too small to accommodate your workload.”¹ That is a memory-and-storage diagnosis, printed in the vendor’s docs, for a service people routinely resize by CPU.

Microsoft went furthest. Azure SQL Database sells capacity in DTUs, and Microsoft defines a DTU as “a blended measure of CPU, memory, reads, and writes.” The DTU benchmark documentation even warns that “an IO-intensive application could hit IO thresholds sooner, or a CPU-intensive application could hit CPU limits sooner.”² Read that carefully. The vendor selling database capacity by the blended unit is telling you, in the same document, that different workloads hit different limits first. The industry had to invent a number that mixes the resources, because no single one of them tracks the workload.

Where the CPU-only view breaks

The narrow view usually comes from a specific place: an organization that takes the infrastructure-as-a-service approach to capacity. When the cloud sells you the pieces separately — instances, disks, IOPS allowances — each piece has its own limit, and the CPU dashboard tells you nothing about the disk’s. AWS’s RDS documentation walks operators through the result: one recommended action for those IO waits is to scale up the instance, because what you really buy is memory for the buffer pool, not just more cores. I call this pattern the apparently idle resource in chapter 5: a disk’s provisioned IOPS is one way a machine can look idle while its workload is throttled.

The CPU-only view survives early on for a boring reason. Small systems fit their working set in memory and run on disks that are oversized for them, so the processor really is the first thing that runs out, and nothing on the dashboards contradicts it. Then the data grows past the cache, the working set stops fitting, the IOPS allowance starts throttling, and the binding constraint moves among CPU, memory, and storage as the workload mix changes. A team that only watches CPU meets each new gate during an incident instead of in the plan.

So if you don’t have a more sophisticated view of capacity, you really need one before you start operating at scale. Track the database’s own measures — cache hit ratios, wait events, reads and writes against what you provisioned — next to the CPU graph. The capacity plan has to hold every resource a workload can be gated by, not just the one on the front page of the console.

Sources