Who Gets a Choice at Scale

Yesterday I wrote about the budget nobody announces (The Second Budget). Today’s question comes from the same size problem, from the other side: should people have a choice?

It is worth slowing down on, because the question hides a series of smaller ones.

Start with what “operating capacity at scale” actually means. For one thing it means the company is very large. You have so many customers that it is difficult to even remember who’s who. At a company that size, you end up operating a system that works much like a cloud provider. You have customers of your own. They want compute, storage, databases, networking. And that puts in front of you the question every provider has to answer: what do you expose to your internal customers?

One provider is already a lot

The first instinct is to pass everything through. Your internal customers can see the same product sheet you see, drawn from whichever providers you run. But providers do not match. Different cloud providers have different classes of compute, different SKUs for their services, different databases, different ways of doing networking. If you expose all of it, you are exposing a heterogeneous collection of products to tens of thousands of people.

A lot of people will hear me say that and laugh. Multi-cloud is more expensive, they will tell you, because you have to build the support to move work between providers. That is not untrue. It is also the case that you cannot really span providers with everything, and databases are always the gravity well that forces some selection. If your data lives in one provider, you want your applications as close to that data as possible, to keep latency down.

Here is the part people skip: some of the span is buildable. If you treat a cloud provider as infrastructure rather than as a product catalog, you can run one database across both. Apache Cassandra, for example, defines a replication strategy that “requires a specified replication factor for each datacenter in the cluster,” and a datacenter in that system is a logical grouping, not a building.¹ Nodes in two providers’ regions can sit in one cluster and replicate to each other. It is not ideal, it costs more money than the built-in products, and most people should not do it. But if you are big enough, and your platform engineering is good enough, you can run it and expose your own SKU to your customers: one simple one, instead of the whole product sheet from every provider.

The full list, or three sizes

Suppose you keep one provider. The list is still enormous. Amazon’s own EC2 documentation currently tables 1,187 instance types across 181 families, from memory-optimized monsters to two-vCPU burstables, and the count moves every few weeks.² Nobody’s internal customers need all of that.

So the next question is whether you expose the full list, or whether you run a system of size variants instead: a small system, a medium system, a large system, with exceptions made for the top 20 or 30 largest systems. Most companies that have thought about this land on the variants.

That level of support buys something specific. It lets you optimize within the classes. You can tell people we are moving you to a new SKU, and it is a change you execute, not a phone campaign. Because calling thousands of people to tell them their machine changed does not work. Unmanaged choice is also where fragmentation comes from, the thousands of slightly different configurations that make every migration harder than the last one (chapter 10).

The point

Most people do not do any of this. They choose one provider, or they give teams console access and let them pick from the full list. Those are both answers, and the trade-offs are real either way. The point of this post is not to hand you the answer. It is to say that decisions like this one, decisions about what your internal customers get to choose, are decisions you only meet at scale. Scale Without Surprise works through a number of them, trade-offs included, and I would encourage you to read it.

Sources