The Long Tail and the Spike

How to read this chapter

This chapter is about the projects you do not examine individually, and the shared drivers and peak contributions that make them matter.

1. The projects you know and the remainder you estimate — the twenty you meet and the thousand under “Other”.
2. Measure concentration before naming its distribution — power-law claims, bursts, and the coincident peak.
3. A small change can spend the remaining headroom — sensitivity to a shared change.
4. Errors do not necessarily cancel — bias, covariance, and measuring dependence in your own forecasts.
5. The population is allowed to change — arrivals and retirements.
6. A small bill can belong to a large obligation — ten thousand $100 bills and who answers for them.
7. Give the remainder an owner — cohorts built on shared drivers, and percentiles that do not add.
8. Put an investigation threshold next to the headroom — triggers a platform team can test before a rollout.

The lesson to carry: the remainder is understood at the level of its shared drivers, and it needs an owner and an investigation threshold just like any named project.

The projects you know and the remainder you estimate

For illustration, suppose twenty large projects consume 800 units of a shared resource at the pool’s busiest interval, and another thousand consume 200 units between them in that interval, leaving the large projects responsible for 80 percent of demand. We’ll model the twenty individually and the thousand as an explicit remainder, then ask what a modest change in that remainder does to the plan.

All the units refer to one resource pool at one time resolution and service target. The largest consumers of dollars, CPU, storage, or token rate may be different customers, so I’d repeat the analysis for each consequential constraint rather than add them into one capacity total.

Measure concentration before naming its distribution

To describe the concentration, we rank project demand from largest to smallest. If project $i$ consumes $d_i$ units and total demand is $D$, the share used by the largest $k$ projects is:

$$S_k = \frac{\sum_{i=1}^{k} d_i}{D}.$$

Plotted against $k$, $S_k$ traces the cumulative share curve. Here, $S_{20}=800/1{,}000=0.80$: twenty projects, about 2 percent of the 1,020, consume 80 percent of demand, far more concentration than the 80/20 rule anyone might be tempted to quote. Read for each constraint, the curve tells us where attention will pay this quarter. It doesn’t tell us the distribution has a shape we can extrapolate.

By long tail, I mean the many smaller consumers in this ranked portfolio. That’s different from the statistical upper tail of rare, large observations over time, which is where a distribution’s shape does reach an operating decision. If a member’s burst sizes followed a power law with an exponent between 2 and 3, the range Clauset and colleagues describe as typical, they would have no finite variance, and the expected largest burst would grow as a power of the record’s length: at an exponent of 2.5, doubling the record raises it by about 60 percent ($2^{1/1.5}\approx1.59$).² A margin set from last quarter’s largest burst would keep being overtaken until a quota or the pool itself cut the bursts off. A customer can be small in the ranking and still produce bursts like that.

Across a thousand members, individually rare bursts are routine and already in the pool’s measured peak; the danger is bursts that arrive together. That’s why the 800 and 200 above are contributions at the pool’s busiest interval, not each project’s own maximum. If project A peaks at 30 units on Monday and project B at 25 on Saturday, each running at 10 on the other’s peak day and less the rest of the week, a pool serving only the two needs 40 units; adding their individual maxima gives 55. Chapter 14 made the forecasting version of this point. For ranking, I’d use each member’s average demand across the pool’s busiest intervals, its contribution to the coincident peak, which promotes a small project that bursts when the pool is busy and demotes a large one that peaks when it’s quiet. Summing the maxima is a legitimate stress scenario only when someone can explain why those peaks would coincide.

Spend can reorder the ranking again: a favorable contract might make a workload inexpensive while it occupies scarce resources in the only location where another customer can operate. I’d want both views, because the financial opportunity and the supply problem may sit with different customers.

A small change can spend the remaining headroom

Let $q$ be the tail’s share of baseline demand and $g$ its fractional growth. With unchanged large-project demand, total demand changes by the fraction $q \times g$: a tail that is 20 percent of demand and grows 10 percent adds 0.20 × 0.10 = 0.02, or 2 percent. If free capacity is $h$ times baseline demand, the tail alone exhausts it when $q \times g$ reaches $h$, so the growth that breaks the plan is $h/q$, here 0.03 / 0.20 = 15 percent, or 30 units. This is sensitivity arithmetic, not a probability estimate.

Now let the head move the other way. If an efficiency project trims the 800-unit head by 2 percent and saved units are interchangeable within this pool, it releases 16 units, which the tail’s 20-unit increase more than absorbs; headroom falls from 30 to 26. If the pool is held under a commitment, the released units don’t shrink the invoice either. Their value is the purchase they let us avoid.

Presenting only the net increase of four would erase the improvement from the report and leave Finance wondering why we want more supply right after an efficiency gain. I’d show the sixteen released units and the twenty newly consumed units together, with the changes that produced them.

Errors do not necessarily cancel

Many small forecasts offer a tempting reassurance: some will be high and some low, so perhaps the errors will cancel. Before relying on that, I’d separate random error from persistent bias. If every small project is underestimated by $b$ units, total underestimation is $N \times b$, where $N$ is the number of small projects, and adding projects makes that problem larger.

Random errors can also move together. Variance measures how widely errors spread around their mean; covariance describes whether two errors tend to depart from their means together. For errors $e_i$ with finite variances, the assumption a heavy-tailed burst would break, the variance of their sum is:

$$\operatorname{Var}\left(\sum_{i=1}^{N}e_i\right) =\sum_{i=1}^{N}\operatorname{Var}(e_i) +2\sum_{i<j}\operatorname{Cov}(e_i,e_j).$$

The first sum measures each project’s error separately; the second accounts for errors moving together, and omitting it assumes those movements contribute nothing to the total. Suppose every project’s error has the same standard deviation $\sigma$ (the square root of variance, back in our original units) and every pair has correlation $\rho$. Aggregate standard deviation is then:

$$\sigma_{\mathrm{total}}=\sigma\sqrt{N[1+(N-1)\rho]}.$$

For $N=1{,}000$ and $\sigma=0.05$, uncorrelated errors give $0.05\sqrt{1{,}000}\approx1.58$ units. With $\rho=0.05$, the result becomes $0.05\sqrt{1{,}000\times50.95}\approx11.29$ units. Errors moving in lockstep would simply add to $N\sigma=50$. Since $1+(N-1)\rho$ is never smaller than $N\rho$, no number of members can push the aggregate below $\sqrt{\rho}\approx22$ percent of that sum, and here covariance supplies 98 percent of the aggregate variance. These are still standard deviations under the stated assumptions; the calculation hasn’t established a normal distribution, forecast coverage, or a high-confidence capacity limit.

Set the bias beside the noise. The spread of independent errors grows with $\sqrt{N}$; a shared bias grows with $N$. The 0.02 configuration change is less than half of one project’s error standard deviation, invisible project by project, yet across the thousand it is 20 units, more than twelve times the 1.58 units of independent noise.

I wouldn’t apply a convenient single correlation to a real portfolio without evidence, and the formula shows how to get some. Using past forecast errors at the horizon the decision depends on, compare the variance of a cohort’s total error with the sum of its members’ error variances; the excess is the covariance term, and in this illustration the total’s variance would be about 51 times that sum. Each member’s covariance with the total error is its share of the aggregate variance, and the shares add up to the whole, so summing them by cohort ranks cohorts by the uncertainty they contribute, which need not follow their size. Both checks need only each member’s error history and the total’s, not the half-million pairwise covariances among a thousand members. Where even that history is missing, an explicit stress scenario may be more useful than a precise-looking margin built on estimates we cannot defend.

The question I’d bring to the platform owners is what could make many consumers depart from their forecasts in the same direction at the same time. A base image that adds a sidecar, a logging default that keeps data longer, or an AI gateway that moves every application to a model that writes longer answers can do it without any owner choosing to consume more; so can a common campaign, or a restriction in one location that redirects work into another. Time spent only with the largest consumers won’t lead us to the person who knows the shared change is coming.

The population is allowed to change

We can also underestimate the remainder while getting every existing member’s forecast right. In this separate sensitivity calculation, the thousand small projects continue to use exactly 0.2 units each, or 200 in total, but the platform adds fifty similar projects. Those new members contribute:

$$50\times0.2=10\ \text{units}.$$

That uses one-third of the original 30 units of headroom without any individual forecast being wrong. If twenty existing projects also retire, they release $20\times0.2=4$ units to this interchangeable pool, a net increase of six. I’d verify the releases first: an announced retirement may leave an old system running through a migration, or preserving data after its users have moved.

I’d want to see continuing members, arrivals, and departures separately. A stable average among continuing members can coexist with substantial population growth, and a stable total can conceal new demand temporarily offset by retirements.

On a platform built to make application creation easier, the adoption plan belongs in this analysis: how quickly many more people could deploy small applications, and what the common defaults will consume. I’d begin with the platform owner, who can influence those defaults and the shared reserve, rather than sending every new creator a cost report and hoping the aggregate takes care of itself.

A small bill can belong to a large obligation

In a separate dollar illustration, ten thousand projects spend $100 each per day: $1 million a day, $30 million over a 30-day period, and about $365 million a year at that rate. That’s a major financial responsibility made up of customers whose individual bills may not justify a separate optimization program.

If those customers have little control over the underlying platform, making each explain its bill creates work without giving them a way to change the result, what Volume I, Chapter 3 calls visibility without agency. A central team can maintain the attribution, investigate exceptional behavior, and work on common defaults while the users concentrate on what the platform enables them to do. I’d consider that a deliberate service to the customers, rather than a failure to make everyone sufficiently interested in cost.

Some members will still warrant direct attention, such as a low-volume service supporting a critical operation or a new application that could grow abruptly. Chapter 16 takes up how a member leaves the group treatment and returns to it.

Give the remainder an owner

The useful grouping depends on what can move together: an application family, a customer rollout, a shared dependency, a geography, or scheduled business activity. The cohort owner needs access to project-level observations so that an aggregate change can be traced to continuing members, arrivals, retirements, or a common change in behavior. Otherwise we’ve given someone responsibility for a number without a way to explain it.

Those drivers cross one another: a project sits in one geography, depends on one shared database, and belongs to one customer rollout. For ownership, I’d put each project in exactly one cohort and carry the other drivers as labels, so totals can be cut along any driver without counting anyone twice. Forecasters call this a grouped structure, and their reconciliation methods keep forecasts coherent along every grouping at once.³

Coherence doesn’t establish accuracy, and percentiles don’t add. A project that bursts in 4 percent of hours shows no burst at its 95th percentile, but two such projects bursting independently put a burst into $1-0.96^2\approx7.8$ percent of the pool’s hours, so the pool’s 95th percentile contains one. Financial risk managers know this failure from value at risk, a quantile that can be larger for a combined position than for its parts added together.⁴ When members peak at different times, the error runs the other way and the sum overstates the pool. Either way, I’d compute percentiles on the aggregated series rather than add them up.

Put an investigation threshold next to the headroom

Return to the 1,000-unit portfolio with 30 units available, and suppose we review a shared change whenever it could consume at least half the remaining headroom before we can respond. I’d use that as a rule for this case, chosen for the decisions we need to make, not a percentage every capacity team should adopt.

Half the headroom is 15 units, so the possible 20-unit increase warrants attention even though it amounts to only 2 percent of total demand. Spread across the cohort, 15 units is 0.015 per member, 7.5 percent of the average member’s 0.2. That’s the number I’d give the platform team: a proposed default, base image, or gateway change that adds more than that per member gets reviewed before rollout, whatever it does to any one project’s bill. The test tightens as the pool fills; once the 20-unit change lands, half of the remaining 10 units is 0.005 per member. A threshold fixed as a share of spend wouldn’t move.

The same 15 units can govern the chapter’s other mechanisms. Seventy-five net arrivals at 0.2 each would reach it, so an adoption plan implying that many within the supply lead time gets the same review. Persistent bias grows with $N$, so a cohort total that has missed in the same direction for several consecutive reviews gets investigated even if each miss looked small.

For the configuration change itself, the platform specialists now have a specific question: does it add about 0.02 units per member under the relevant mix, and can the rollout be limited or reversed if it adds more? Testing representative members also shows which ones differ enough in configuration or service consequence to need separate examination.

The runnable example in examples/portfolio_math.py reproduces the headroom table, the efficiency comparison, and the two aggregate standard deviations; the integrated companion repeats the separate-pool example in its review report. Neither fits a power law or estimates correlations from real data.

Chapter 16 turns these calculations into a standing process: who maintains each cohort view, when a member is promoted to individual attention, and how it is released again.

Notes and sources

1. Aaron Clauset, Cosma Rohilla Shalizi, and M. E. J. Newman, “Power-law distributions in empirical data,” SIAM Review 51(4), 661–703 (2009), https://doi.org/10.1137/070710111; preprint https://arxiv.org/abs/0706.1062, checked October 3, 2026. Supports the weakness of least-squares fits on logarithmic plots, the maximum-likelihood, goodness-of-fit and likelihood-ratio procedure, the difficulty of ruling out the log-normal, and exponents typically between 2 and 3. It says nothing about the distribution of any computing workload.↩︎

2. M. E. J. Newman, “Power laws, Pareto distributions and Zipf’s law,” Contemporary Physics 46(5), 323–351 (2005), https://doi.org/10.1080/00107510500052444; preprint https://arxiv.org/abs/cond-mat/0412004, checked October 3, 2026. Section III shows that a power law has no finite variance for exponents at or below 3 and that the expected largest value in a sample grows as a power of the sample size (his Eq. 23), the basis of the doubling figure. Applying this to computing bursts is the author’s reasoning.↩︎

3. Rob J. Hyndman and George Athanasopoulos, Forecasting: Principles and Practice, third edition, sections 11.1, https://otexts.com/fpp3/hts.html, and 11.3, https://otexts.com/fpp3/reconciliation.html, accessed October 3, 2026. Defines grouped structures, whose crossed attributes give alternative aggregation paths to one total, and reconciliation that makes forecasts coherent across them. The ownership convention is the book’s application.↩︎

4. Philippe Artzner, Freddy Delbaen, Jean-Marc Eber, and David Heath, “Coherent measures of risk,” Mathematical Finance 9(3), 203–228 (1999), https://doi.org/10.1111/1467-9965.00068; preprint https://people.math.ethz.ch/~delbaen/ftp/preprints/CoherentMF.pdf, checked October 3, 2026. Section 3.3 shows that value at risk, a quantile-based measure, is not subadditive: in its example of written digital options, the combined position’s value at risk exceeds the sum of the separate ones. The capacity example is the author’s analogy.↩︎