A SKU Is Not a Machine

Yesterday I put spot instances on my list. Today the list gains another entry: the same SKU is not always the same machine.

When you buy a SKU from a cloud provider, you are buying a labeled box. Eight vCPUs, thirty-two GB of memory, a price, a name. That label is the whole contract. The provider keeps the right to change what is underneath it. Google says it outright in the Compute Engine documentation: some machine types can run on more than one CPU platform¹. The E2 series, the one most people reach for first, can run on Intel Broadwell or AMD EPYC Milan, and there is no way to pin it. Google offers a minimum-CPU-platform setting on some machine types. E2 is not one of them.

Azure is the same story. The Dv4 family runs on Emerald Rapids, Sapphire Rapids, Ice Lake, or Cascade Lake — four generations of Xeon behind one SKU name, and you cannot pick which one you get².

At a few machines, none of this matters. The label holds. Eight vCPUs is eight vCPUs. The lesson comes when you dedicate thousands of them, because that is when you push the fleet to the limits of what it can do, and there you start to see performance differences between machines wearing the same label. I remember one case where the difference came down to the way memory was configured in one subtype of the SKU. The machines were built as the same thing, but one part of the fleet was on a newer generation of chips than the other. It was the same SKU and it was not the same machine.

Jeffrey Dean and Luiz André Barroso published The Tail at Scale in 2013. Their argument, in their own words: “Temporary high latency episodes which are unimportant in moderate size systems may come to dominate overall service performance at large scale.”³ In a fleet of a few dozen machines, one slow node is a curiosity. In a fleet of ten thousand, one slow node is the machine the whole distributed system waits for.

That is what made the memory behavior matter. There was a caching layer behaving differently across nodes. There were slight latency differences on paths that critical distributed systems shared. Slight differences like that sit at the heart of how distributed systems fail: one node a little slower than the rest, every request that touches it slower by the same margin, the whole service holding the pace of its slowest participant.

And here is the frustrating part. You cannot get this insight when you buy the SKU. The provider’s documentation, at best, names the possible platforms. The differences themselves — the memory configuration, the cache behavior, the latency profile — only show up when you run the thing at a scale where they start to matter. You learn the hardware roster of your SKU by operating it, not by reading about it.

So what do you do with this, in the general case?

First, expect to learn the SKU’s hardware roster by performance, and expect to learn it at the worst possible time. The differences are invisible at low utilization and become the thing your on-call rotation learns about under load.

Second, where the provider gives you a control, use it from day one. Google’s minimum-CPU-platform setting exists on some machine types precisely because fleet operators eventually need to lock a fleet onto one chip generation. If your workload is sensitive to the hardware underneath, pinning the platform is cheaper than discovering the difference in production.

Third, when you find a performance difference, treat the fix as a capacity decision, not a code hunt. If part of the fleet is on the older chip generation, the fix is to move that part onto the newer one, or pin the whole fleet onto a platform, or re-slice the fleet deliberately — not to keep debugging the application as if the fleet were homogeneous when it is not.

This is also why the book runs an experiment on shared hardware before committing to a fleet (chapter 10). The billing unit is not the machine. You pay for a SKU and you operate machines, and the provider can change the machines underneath the SKU at any time. Measure what you are actually operating before ten thousand of it are yours.

Sources