Volume II · Part IV · Chapter 19
Managing Risks and Anomalies
Earlier in my career, I had a small business building a search engine. We had a primary system and a backup, which was considerably less reassuring after lightning struck the building and took out both. That was effectively the end of the business. I went back to school.1
The story takes a few sentences now. What mattered then was that we had prepared for a narrower problem than the one that occurred: having a backup had not given us a recovery we could perform after losing the place that contained it.
That experience is behind the way I use the word anomaly: an event outside the ordinary operating assumptions of the business, one that changes what the capacity plan can promise. No capacity team can predict these events, and I would not ask one to try. I would ask it to know, before one arrives, which decisions will be its own, what each would cost, and how long each would take.
How to read this chapter
This chapter is about events outside the forecast, from a failed host to a pandemic, and what a capacity team does when they arrive.
- What is not an anomaly — security response and ordinary variation.
- The dimensions of an anomaly — scale and force.
- Weather, climate, and the map you inherit — forecasts read against real dependencies.
- Conflict, and knowing where things run — regions lost outside the software stack.
- The out-of-nowhere events: solar storms and meteors — warning time against recovery time.
- COVID, and the anomaly that changes demand — an early call to a provider.
- The year the water reached the supply chain — shortages, memory cycles, and contrarians.
- Not the worst-case team: the decisions that are actually yours — four decisions that belong to capacity.
- Tactical: what a capacity team actually does — destination capacity and failover arithmetic.
- Different scenarios, different frequencies — a cadence sized to the threat.
- What the capacity team is for, when it happens — the scenario record.
The lesson to carry: anomalies are managed by naming, before the event, the decisions that belong to the capacity team — not by predicting every event.
What is not an anomaly
First, a boundary, because it is the most common confusion I encounter when this subject comes up. Routine security response belongs to the security and service teams: detection, containment, rate limiting, admission control, and recovery. That is an everyday discipline, with teams, tooling, and playbooks of its own. The capacity team needs to know how those measures change resource use and whether they preserve the promised service, and it should agree with those teams beforehand on when a response would need a capacity decision.
The anomaly I mean is the event that changes the business’s operating picture: a compromise severe enough to force a temporary double bubble while credentials are rotated and environments are rebuilt; a compromise of someone else’s systems in the same data center that forces your workloads to move; a regional event that requires systems designed for one geography to fail over to another. The attack is the security team’s anomaly. The consequences — the sudden need for capacity in a different region, the doubled spend while two environments run at once, the emergency conversation with a provider about what can actually be provisioned by Friday — are the capacity team’s.
The same test applies to demand. A predictable daily peak, an annual sales event, or ordinary variation belongs in the normal capacity plan; calling it an anomaly can conceal a baseline that was never adequate. Investigate before naming: a changed workload, a broken measurement, and a loss of supply need different responses even when they produce similar charts.
The dimensions of an anomaly
It helps to be systematic about where anomalies come from, because “something bad happens” is not a plan. I use two axes: scale, meaning how much of the physical world the event takes with it, and the kind of force doing the taking.
Scale: from a host to the world
Every step up the scale changes the question the plan has to answer:
- Host. A server or an accelerator fails; Chapter 18’s failure-group arithmetic covers this case.
- Rack. Shared power or networking fails. Were supposedly separate replicas placed behind the same dependency? Volume I, Chapter 10’s seventy-two-GPU rack, with one NVLink fabric and shared power shelves, is the extreme version.
- Data center. Loss of the facility, or of access to it. Can another site carry live traffic and recovery work at the same time?
- Region. Weather, grid failure, or conflict removes a whole geographic area. Which fallback sites silently shared the affected services?
- World. A pandemic, a financial crisis, or a war that moves supply chains for everyone at once. The providers are affected too; so are their suppliers; so are your customers.
I would ask which levels the organization has actually tested. Passing host-failure tests say nothing about recovery after a facility loss, and a plan that assumes new machines will arrive during a world-level event needs evidence that suppliers can deliver them.
Force: the kinds of events that hit the business
Each type of event arrives with different warning and different consequences:
- Weather and climate. Not the daily forecast — the anomalous event: hurricane, flood, heat that overwhelms cooling, or a hard freeze in a region built for warmth.
- Energy. Constraints on generation and transmission, which can arrive independently of any storm and have their own market dynamics.
- Conflict. War reaching infrastructure that was never a target but shares geography with one.
- Geological and cosmic. Earthquakes, solar storms, and meteors.
- Regulation and policy. A rule changes which equipment, services, or locations the workload may use, even though the infrastructure still operates.
- Demand shocks. The pandemic case: the world changes its behavior overnight, and usage multiplies.
Force matters separately from scale because the responses differ: demand can exceed the service’s limit while every machine stays healthy, and a grid disruption can remove supply while demand is unchanged. Identify which assumption has failed before choosing more resources, less work, or a different place to run.
Weather, climate, and the map you inherit
A seasonal outlook is where weather planning starts, and it says little about any one place. In August 2026, with a strong El Niño under way, NOAA gave a below-normal Atlantic season a 75 percent chance and the Central Pacific a 70 percent chance of an above-normal one, while warning that its outlook is not a landfall forecast and that it only takes one hurricane to cause a disaster.23
A seasonal signal gives longer warning than any single storm, and this year’s is unusually strong. In September, NOAA’s Climate Prediction Center gave a very strong El Niño — the kind often called a super El Niño — more than a 90 percent chance this fall and winter, and a 75 percent chance that October through December would exceed every El Niño since 1950. Its outlooks lean wet across the southern Plains this winter and up the Atlantic coast through the Mid-Atlantic, though impacts are not guaranteed.4 Texas’s state climatologist expects the cool, wet conditions El Niño usually brings to the state, and cautions that an event this strong may fall outside the historical envelope.5 The last super El Niño shows what that can mean: as it built in 2015, Texas had the wettest month in its record that May, with extensive flooding, and finished the year record wet.6 A deployment split between Azure’s East US region in Virginia and its South Central US region in Texas sits inside the same seasonal signal at both ends for months.7 No storm has to reach a data center for that to matter, because a failover plan assumes the other region is having an ordinary season. Use the outlook months ahead: compare it with the provider locations you depend on and the roads, grid, and network routes behind them, schedule maintenance and migrations around the wet months, and date the assessment, because the next outlook may change it.
The February 2021 Texas freeze shows how one weather event can affect both supply and price. FERC and NERC documented the largest manually controlled load shed in U.S. history, with more than 4.5 million people losing power, some for as long as four days. Freezing and fuel problems caused three-quarters of the unplanned generator outages, and 81 percent of the freeze-related outages occurred at temperatures above the units’ stated design temperature.8 Wholesale electricity prices reached the then-applicable $9,000-per-megawatt-hour cap under emergency pricing orders,9 and some customers on wholesale-indexed retail plans faced extraordinary bills.10 A cloud customer’s exposure depends on its contract and the provider’s charges, so the capacity review should separate loss of service, operation on backup generation, and a price change the customer is actually liable to pay.
That 81 percent is the gap between a rating and a test, and it is the number I would take into a provider conversation. Are the provider’s resilience figures design ratings or tested results? How long can its backup arrangements run? Which parts of our recovery still depend on the affected region?
Whenever I have brought up solar flares and Chelyabinsk-class events in a planning discussion, I have never failed to find a room full of people laughing them away. “Well, you know, if that happens, we’ll have bigger things to worry about than the data centers.” That might be true, but it is worth asking who the laughter is for. If your business provides core infrastructure to a society — a network of grocery stores, a food-delivery platform, anything people rely on when everything else is failing — everyone else will be counting on you not to have laughed. In my own experience, that laughter disappears very quickly when the world is facing something as life-altering as a global pandemic. And if the entire cost of preparedness is having a few smug executives laugh at you a year before the event, so be it.
Geography is one decision that deserves to be made deliberately. I used to work for a company that had a primary data center in Dallas, and a secondary data center also in Dallas. Not a different suburb — the same city, close enough that the same ice storm, the same grid event, or the same substation failure could plausibly reach both. During a risk discussion I raised the Chelyabinsk meteor and other low-probability, high-consequence events, and it was not obvious to everyone in the room that there was a problem. The two facilities had different addresses, different buildings, different equipment. Nobody had asked what they shared.
My question was what “secondary” means when the event is bigger than the building: whether the two sites share power infrastructure, network paths, staffing, or simply the same regional exposure to the same weather.
Conflict, and knowing where things run
Armed conflict is another reason to know where cloud services run. In March 2026, Amazon said its AWS Bahrain region had been disrupted by the ongoing conflict and asked customers with workloads in the affected regions to keep migrating them elsewhere.11
Ask providers which independence guarantees and shared dependencies they disclose. Customers may not be told exact facility locations; that limitation belongs in the assessment rather than becoming an assumption of independence. A prepared team can name a candidate recovery region, the data it needs, its legal eligibility, and the time and cost of moving there.
A region that is physically fine but legally closed to your data is as unusable as one that is on fire; Chapter 21 treats the legal side.
The out-of-nowhere events: solar storms and meteors
Some events arrive literally from space. I have written about two of them, and they differ in the two ways that matter for planning: how much warning arrives, and what preparation can still change.
Solar storms: warning and preparation
Solar flares and coronal mass ejections are different phenomena; for the power grid, the one that matters is an Earth-directed coronal mass ejection, or CME. Forecasters can watch a CME leave the Sun. The fastest reach Earth in 15 to 18 hours, and slower ones take several days. Spacecraft near the L1 point, about 1.5 million kilometers toward the Sun, measure the disturbance as it passes, which often gives 15 to 60 minutes of warning.12 That last interval is far too short to provision and qualify replacement infrastructure. Compare the warning with the actual recovery time, and hold in advance whatever the promised response needs.
The history is concrete. In March 1989, a geomagnetic storm collapsed the Hydro-Québec grid in less than two minutes and left more than six million people without power for nine hours. Lloyd’s has modeled a storm on the scale of 1859’s Carrington Event hitting the modern North American grid: 20 to 40 million people in the United States at risk of outages lasting from 16 days to one or two years, at a cost of $0.6 to $2.6 trillion. For capacity, the line that matters is duration, which would depend largely on spare transformers; new ones would likely take at least five months to arrive.13 The Carrington storm itself set fire to some telegraph offices, and in July 2012 an eruption that researchers judged comparable missed Earth by about a week.14
The damage runs through the grid: geomagnetically induced currents stress transmission systems and can damage transformers.15 That makes a solar storm the cleanest example of an anomaly that arrives with warning, cannot be prevented, and can still be mitigated. Walking through it exposes the dependency questions: which facilities share the same grid exposure, which generators have fuel contracts that assume functioning roads, and how long a facility can run if the grid stays down longer than the fuel on site lasts.
One thing you can actually do is sign up for the warnings. NOAA’s Space Weather Prediction Center publishes them at spaceweather.gov and emails them to subscribers, and they form a ladder. A Watch can come up to 72 hours ahead, with low confidence in timing and intensity; a Warning, driven by the measurements at L1, is more certain but usually comes only minutes to a couple of hours ahead; an Alert means a storm threshold has been reached.16 Ever since I have been involved in reliability and capacity, I have made sure to sign up for those watches, warnings and alerts. They are good for capacity: the alerts are the difference between reading about a grid emergency and knowing one is inbound. They are also good if you are interested in stepping outside to look at the aurora, which the same alerts predict — and that is the little secret. If you are running capacity at scale, I hate to tell you, but you need to be tracking space weather.
The ladder also explains what a warning is for. When a Watch arrives, the storm’s timing and intensity are still uncertain; the orientation of the CME’s magnetic field, which does much to decide its severity, is measured only as it passes L1.17 So the capacity team usually does not hit the red button and tell everybody to move. It warns people that something might be happening, and it picks up the phone to tell the providers it depends on that it might need more capacity shortly.
Meteors and the loss of a place
In my 2025 essay That AWS Outage Was Nothing: When Meteors Meet Data Centers, I used a software outage to ask about a disaster that could not simply be patched: an airburst over Northern Virginia that damages a data center and the infrastructure around it.18 It is a constructed scenario for finding shared dependencies, not a probability for any site. The physical case behind it is Chelyabinsk. In February 2013, an asteroid roughly 17 to 20 meters across approached from within 15 degrees of the direction of the Sun, where asteroid-detection telescopes cannot scan, so nobody saw it coming. It exploded more than 20 kilometers up with the energy of roughly 440 to 500 kilotons of TNT, blew out windows across some 200 square miles, and injured more than 1,600 people, most of them with broken glass.19
Suppose the facility remains inaccessible for an extended period, the people who would normally restore it cannot get there, and replacement equipment cannot arrive on the usual schedule. Recovery then has to run from what exists elsewhere: the data, the credentials, the control services, the operating knowledge, the usable capacity. The reliable way to find a recovery step that still depends on the lost location is to walk the procedure with the scenario in mind. An undetected asteroid gives very little warning, yet geographic separation and recovery arrangements can still limit the damage to a service.
COVID, and the anomaly that changes demand
I tell this story not as an “I told you so” but because it shows something about the capacity role. I am the kind of person who tracks these issues, though nobody ever asked me to. In late December 2019 and early January 2020, I noticed Slack chatter on a group devoted to tracking arms-control issues about reports of a novel respiratory illness — the outbreak that became COVID-19. At the time I was responsible for capacity at a company in the middle of a migration from its own data centers to public cloud, and I managed the relationship with a large cloud provider. In January, I ended one of our regular meetings with a statement that got some surprised looks and some laughter: “Hey, this news story — I think we’re going to need to plan for 10x traffic in a few weeks. I have a bad feeling.” Someone answered, “Dude, you might be a little crazy.” I told them I felt strongly that it was going to be a big deal; I was flying back and forth to California, and people were still getting on airplanes.
Nobody got me in trouble for saying it, and the conversation started. The capacity representatives at the provider were open to raising it with their management, and told me honestly that nobody had discussed it yet. In hindsight, raising a pandemic scenario that early might have been a risk to my own credibility — the base-rate case for “10x in a few weeks” from a news story is weak, and I knew it. But our position made the warning worth the risk. Management had planned the migration as a one-year project; we were in its second year of what would turn out to be five, and the planned decommissions of private capacity had already begun. We were, in effect, in the middle of a burn-the-boats strategy: the capacity we would once have fallen back on was being retired, and the cloud was becoming the only place more could come from. A migration that retires the old capacity on schedule assumes demand will stay inside the forecast while both environments are thin. I would rather have been early and wrong than quiet and right. A month later, an early outbreak in the United States happened near one of my provider’s offices. At the next monthly meeting, someone said: “You were the first person to ask us to plan for this, and it started the conversation about prioritization. You are on that priority list.” The experience reinforced my preference for discussing a plausible surge before it becomes an urgent request, though one conversation cannot show how often an early warning secures supply.
COVID is the case in which I saw cost, capacity, and capability all move at once. Demand patterns changed, what the business needed its systems to do changed, capacity risk changed, the providers’ constraints changed, and the cost consequences followed. The early signal was visible because those could be considered together rather than escalated among separate organizations. The response went beyond raising an alarm or building a dashboard. It included calling providers directly and telling them, in substance, that we were about to be in serious trouble: “Can you help me? Can I help you? What do you need from me?”
An early warning needs a way into the capacity conversation: the continuity or risk team may already be tracking the event, and capacity adds its view of consumption, lead times, and supply constraints.
A demand shock need not involve any technical failure. DoorDash reported 816 million orders in 2020, up 210 percent, or about 3.1 times the 2019 volume.20 Annual orders reveal neither the busiest interval nor how much infrastructure consumption grew, but they show why the team needs updated business assumptions as well as machine telemetry.
The year the water reached the supply chain
Demand shocks and outages are visible from inside your own deployment; the supply-chain anomaly is quieter. I learned to watch for it when I ran a team responsible for receiving capacity configured in an internal private cloud data center — our own racks, our own hardware, ordered on a plan. That year there had been tremendous flooding in Asia: in 2011, floods in Thailand disrupted hard-drive manufacturing, and Western Digital shipped 28.5 million drives in the last quarter of 2011, against 52.2 million a year earlier, and did not expect to regain its pre-flood manufacturing capacity until the quarter ending September 2012.21 Some of our plans required that hardware land in a specific timeframe, and when it did not, somebody had to understand how supply-chain problems we did not cause were changing plans we had already made. That was one of the triggers for me saying: somebody needs to be paying attention to supply-chain issues if this company is going to be ordering that much hardware. Nobody had assigned that job. It simply had to belong to someone, and the someone was us.
The pattern did not stay in 2011. In December 2016, TrendForce expected NAND prices to keep rising through 2017, because the move to 3D NAND was cutting the supply of older planar NAND faster than the new architecture could replace it.22 By mid-2018 it was reporting oversupply, with NAND contract prices down 15 to 20 percent in a single quarter.23 In March 2026 it forecast second-quarter contract increases of 58 to 63 percent for conventional DRAM and 70 to 75 percent for NAND, with suppliers shifting capacity toward HBM, cloud providers locking in long-term agreements, and no meaningful expansion of enterprise-SSD capacity before late 2027.24 At the end of September it expected smaller fourth-quarter increases, 10 to 15 percent for conventional DRAM and 15 to 20 percent for NAND, with server DRAM still undersupplied.25 These are forecasts for particular products and periods, not every buyer’s bill, and DRAM and NAND are separate markets. Watching them should be someone’s assigned duty: the components we actually buy, current quotations, and supplier lead times, reviewed on a schedule rather than when a purchase order slips.
Cycles like these are why I think it always makes sense to look at the contrarians and model the ingenuity of someone like Michael Burry, one of the investors Michael Lewis followed in The Big Short.26 The Financial Crisis Inquiry Commission describes how Burry moved from investing in homebuilders to questioning the financing behind the boom, concluded that the reckoning would come when a wave of two-year adjustable-rate mortgages needed refinancing, and had bought credit default swaps on mortgage securities by mid-2005.27 The habit worth borrowing is the investigation: find the mechanism beneath a comfortable assumption and follow the evidence somewhere inconvenient. That is not a reason to believe every contrarian; a contrary thesis can be wrong or badly timed — a team that bought as if the 2017 NAND shortage were permanent would have paid peak prices into the 2018 oversupply — so plan for both the shortage and the case where expected growth evaporates. And you can’t short cloud capacity: even a well-timed financial position would not put a usable machine into service. An early, correct view is worth what it lets you change in time: an order placed earlier, an alternative qualified, a customer warned. Chapter 24 returns to Burry and to what financial instruments can do for a buyer of compute.
When I was responsible for physical infrastructure, receiving the hardware was only part of the job: racks might need rework, installations needed safety inspections, and equipment could arrive without being ready to serve anything. Unless the business requires owning that work, I would need a strong reason to take it on; buying a cloud service puts much of it with the provider, and the economic comparison should count it.
Not the worst-case team: the decisions that are actually yours
There is a version of this chapter that asks a capacity team to become a standing risk office, gaming out every catastrophe. That is not the job, and a team that tried it would do the analysis worse than the specialists whose actual job it is, and probably talk itself into paralysis along the way. But some decisions should be made consciously rather than by default, and four of them belong to us.
The first is knowing the geography of what you run: not “which cloud” but which region, which availability zone, and sometimes which building. Knowing this in advance removes a discovery task from the emergency.
The second is knowing which of the dimensions above have a plausible claim on your operations, and having said, out loud, what the plan is for each. A business with a Texas footprint has winter-grid questions that a purely coastal deployment does not. The point is to notice the categories with no plan at all.
The third is deciding who watches. A capacity team operating at scale is usually part of a company that has someone tracking these issues, and some organizations have teams doing nothing more than watching risks: weather, geopolitical chatter, things you would never imagine someone was paid to watch. Connect with them, but that is not sufficient, because the tracker inside your own company is watching for the events your company has already decided to name. If you are going to answer for a cloud capacity mess, you probably need one or two people on your own team, even as a rotating duty, who track this stuff just because they do. If an anomalous winter storm is going to hit Texas, the word should go out to the teams when the forecast does, not after the risk office files its report.
I have been on capacity teams where one person simply watched the Space Weather Prediction Center. It was a rare occurrence that the alerts mattered, but awareness from your own subscription is awareness you did not have to wait for anyone to grant. The same goes for somebody who follows how regulations are starting to evolve and at least knows how to talk to the lawyers when they call.
The fourth is model-provider diversity. An alternative provider is useful only if the feature can operate with its model, limits, quality, and data-handling terms, and two providers may share underlying infrastructure. I would ask the AI program what a day without its preferred model would cost, which features can pause, and whether a tested alternative has enough capacity. Switching a model name in a configuration file is not evidence that the service can recover.
Tactical: what a capacity team actually does
It is easy to read a chapter like this and conclude that the answer is a bigger disaster-recovery budget. Sometimes it is. Most of the capacity-side work is more prosaic.
Track the workloads near their limits
SRE and service teams own the failover mechanics in the arrangement assumed here. Capacity planning contributes the combined view of each destination’s qualified throughput, current demand, recovery work, and competing workloads, using the specialists’ measurements rather than a low CPU percentage before calling a destination available.
Azure makes the distinction concrete. Only a few of its services use region pairs, and many regions have none. Microsoft says a paired region does not by itself provide failover, and tells customers not to rely on Microsoft-managed failover between pairs as their primary recovery approach; for geo-redundant storage it happens only in catastrophic situations, after repeated failed recovery attempts.28 A plan that waits for it has no start time it controls. For each destination, identify which features replicate state, who initiates recovery, and what throughput it can sustain, then test the arrangement the service requires.
The headroom you hold is the failover you can do
This is Chapter 18’s failure-group arithmetic with regions as the groups. Take equal regions sharing the work evenly, each able to sustain a qualified rate , and let be the highest fraction of a survivor may carry after one region is lost. Normal utilization must then satisfy
With two regions and , the ceiling is 40 percent: two regions at send to the survivor, leaving 20 percent of its qualified capacity free. Letting the survivor run to its full tested limit raises the ceiling to 50 percent. Three regions with the same margin can each run at about 53.3 percent (); at two-thirds each, a loss leaves the survivors no headroom. These are upper bounds under stated assumptions, not utilization targets. Uneven routing, recovery work, a changed request mix, and shared failures each need their own calculation, and no percentage guarantees zero downtime.
Running at 40 percent buys room for one particular failure, and real routing layers often favor latency and proximity over even division, so test where traffic would actually go. If the headroom was never held and the survivor must double at a moment’s notice, then for a large enough workload the least you owe your provider is a conversation. If the emergency affects many companies at once — a regional disaster, a market-wide event — the capacity you are asking for may not exist at any price, because everyone else is asking too.
Multi-cloud, and what it is actually for
Plenty of thoughtful practitioners write that multi-cloud is a mistake, and for a small business they may well be right. At scale, my position is that it is not. The supply of specialized hardware is not infinite, and when one provider is short of space or equipment, a second provider is somewhere to go. Go in expecting options rather than savings. The option is real only when the workload, data, identities, network paths, and operating procedures can move with it, and maintaining it costs money and engineering time. Private infrastructure is another possibility only where the organization can operate and fund it; Volume I, Chapter 2 explains why I treat that as an exception. Compare each arrangement with a second region in the same cloud against the same requirements; an untested deployment on another provider is not a switch we can count on during an incident.
Model the failover before you need it
Compare the two-region example with a single region carrying the same total workload at 80 percent of . The two-region design buys of capacity instead of , doubling compute capacity before replication, networking, and other costs. Three regions held to the 53.3 percent ceiling worked out above can protect the same workload with less. After a loss, each survivor carries half of , or ; at no more than 80 percent, each region needs , and the three together need — a quarter less capacity than the two-region design, paid for with a third copy of the state, more replication traffic, and another region to operate. Compare prices only under equal service and failure requirements; otherwise an apparently cheaper arrangement may simply promise less.
What makes this tactical rather than theoretical is that the numbers move. A two-region design that ran at 35 percent when it was designed may run at 55 percent after two years of roughly 25 percent annual growth () that nobody connected to the failover math. At 55 percent, losing a region sends to the survivor. The design no longer survives the failure it was bought for, yet a dashboard showing 55 percent utilization looks unremarkable. The monthly review in Chapter 22 is where that drift gets caught; the escalation is what happens when it is found.
A tabletop exercise can expose missing authority, inaccessible credentials, or an unchecked supplier assumption, but it cannot certify the throughput of a surviving region; a controlled failure test can. The capacity team helps define the scenario and the evidence needed; the operators decide how to test safely.
Different scenarios, different frequencies
These scenarios do not all deserve the same meeting or a fully prepared plan, but each deserves a cadence. World-level events — the pandemic class, the war-moves-supply-chains class — are worth a discussion at least once every six months: what would happen, who would be affected, and a tabletop exercise that walks the response and finds the gaps. I will admit the mental image of a monthly standing meeting about solar flares and Chelyabinsk-style events is funny, and the joke writes itself: agenda item one, the sun; agenda item two, the sky; minutes carried forward since 2019. Nobody should run that meeting. But once or twice a year, a refresh makes sense — not because the probability moved, but because the people did, and someone has to remind a room that these things are possible before it becomes the kind of room that laughs.
Regional failures are the opposite case and should not be an annual or even quarterly topic. Talk about them month to month, and have someone maintaining a plan at the ready: which regions can absorb which workloads, what the routing would actually do, and what the current recovery times are. Underneath all of it, the applications and internal customers should already have criticality levels, so that if something requires prioritization, the order is not invented during the event. The conclusions, including the gaps you chose not to close, should be on the record before something happens.
What the capacity team is for, when it happens
The organization should assign ownership of business continuity and disaster recovery explicitly. In the arrangement described here, the capacity team contributes without owning the whole plan: when someone asks “can we handle this?”, we answer alongside the service operators, risk specialists, and business owners whose requirements define what handling it means.
That answer cannot be assembled during an event. Keep a short record for each scenario the organization has decided to plan for:
- Event and scale, with the date of the evidence.
- Affected workloads, their criticality, and the dependencies they share (Volume I, Chapter 1).
- Surviving capacity, and what it is qualified to do (Chapter 18).
- Recovery deadline, and the lead time for anything not already held.
- Permitted destinations, including legal eligibility (Chapter 21).
- Provider commitments, and what they entitle us to in a shortage (Chapter 20).
- Decision owner and trigger, and the evidence that would change the response.
- Accepted gaps: who accepted them, and when they will be reviewed.
I would write the first record for the scenario the organization is most confident it has covered, because confidence is what stops people asking what two sites share. Its findings go to the next monthly review (Chapter 22) and the next provider conversation.
Notes and sources
Tim O’Brien, Leaving the Cloud Isn’t for Everyone, October 31, 2025, “The Risk Premium You’re Paying Cloud to Carry.” The passage is the author’s recollection.↩︎
NOAA Climate Prediction Center, 2026 Atlantic Hurricane Season Outlook, August 2026 update, checked October 4, 2026: a 75 percent chance of a below-normal season amid a strong El Niño; “not a seasonal hurricane landfall forecast”; “it only takes one hurricane (or tropical storm) to cause a disaster.”↩︎
NOAA National Weather Service, 2026 Central Pacific Hurricane Season Outlook. Gives a 70 percent probability of above-normal activity; the forecast is for the basin and season, not a particular facility.↩︎
NOAA Climate Prediction Center, “Prognostic Discussion for Long-Lead Seasonal Outlooks”, September 17, 2026, checked October 4, 2026: more than a 90 percent chance of a very strong El Niño this fall and winter; a 75 percent chance that October–December 2026 exceeds every event since 1950; a wet signal across the southern Plains (up to 40–50 percent chances of above-normal precipitation) and along the Atlantic coast through the Mid-Atlantic; impacts “not guaranteed.” Replaced monthly (next issue October 15, 2026); seasonal probabilities, not a forecast for any facility.↩︎
Railroad Commission of Texas, “Winter Weather Outlook 2026”, notice to operators, checked October 3, 2026, quoting Texas State Climatologist John Nielsen-Gammon: El Niño events are “most commonly associated with cool and wet conditions across Texas,” and “past El Niño events may not be a good guide to this event.”↩︎
NOAA National Centers for Environmental Information, National Climate Reports for May 2015 and Annual 2015, checked October 3, 2026: Texas’s statewide 8.93 inches in May was “the wettest month ever” in the 1895–2015 record, with “extensive flooding”; Oklahoma and Texas were “record wet for the year.” El Niño shifted the odds toward a wet year; it does not account for each storm.↩︎
Microsoft Learn, “List of Azure regions”, updated September 23, 2026, checked October 3, 2026: East US (
eastus) in Virginia and South Central US (southcentralus) in Texas. Microsoft gives locations by state only. An illustration of a two-region layout, not a description of any customer’s deployment.↩︎FERC and NERC, Final Report on February 2021 Freeze Underscores Winterization Recommendations, November 16, 2021: the largest manually controlled load shed in U.S. history; more than 4.5 million people without power, some for up to four days; freezing and fuel issues behind 75.6 percent of unplanned generator outages; 81 percent of freeze-related outages above the units’ stated ambient design temperature.↩︎
ERCOT, Market notice M-C021521-01, February 15, 2021. Implements the PUCT order to reflect firm load shed in scarcity pricing and keep the $9,000/MWh cap in effect. Historical event, not a current price-cap claim.↩︎
Texas Attorney General, Forgiveness of $29 million in electric bills after suing Griddy Energy, March 16, 2021. About 24,000 former customers released from $29.1 million in unpaid bills. A retail-consumer case, not evidence of cloud pass-through.↩︎
Amazon, AWS Bahrain Region disrupted by ongoing conflict, March 24, 2026. Asks customers with workloads in the affected regions to continue migrating; gives no recovery timeline.↩︎
NOAA SWPC, Coronal Mass Ejections, checked October 4, 2026: the fastest Earth-directed CMEs arrive “in as little as 15-18 hours,” slower ones in several days; DSCOVR at L1 “can often provide 15 to 60 minutes advanced warning.” Typical intervals, not guarantees.↩︎
Lloyd’s, with Atmospheric and Environmental Research, Solar Storm Risk to the North American Electric Grid, May 2013: the 1989 Hydro-Québec collapse “in less than two minutes,” leaving “more than six million people” without power for nine hours; 20–40 million U.S. people at risk from a Carrington-level storm for 16 days to 1–2 years, duration depending largely on spare transformers, with a likely minimum five-month lead time for new ones; $0.6–2.6 trillion in cost. A scenario, not a probability forecast.↩︎
NASA Science, Near Miss: The Solar Superstorm of July 2012, July 23, 2014: the 1859 storms set “fire to some telegraph offices”; one week earlier, “Earth would have been in the line of fire”; Daniel Baker’s team estimated a storm comparable to Carrington’s.↩︎
NOAA/NWS Space Weather Prediction Center, “Electric Power Transmission”, accessed September 23, 2026. Describes geomagnetically induced currents, transformer effects, and grid instability; site-specific exposure requires additional evidence.↩︎
NOAA/NWS Space Weather Prediction Center, “Subscription Services”, checked October 4, 2026: Watches cover up to 72 hours with lower confidence in intensity and timing; Warnings, from upstream solar-wind data, come “minutes to a couple of hours in advance”; Alerts mark a threshold reached.↩︎
NOAA SWPC, Coronal Mass Ejections: intense storming is favored when the CME’s magnetic field turns strongly and persistently southward. Lloyd’s (2013, PDF p. 16) notes that field strength and orientation “can only be measured with a lead time of 15-30 minutes.”↩︎
Tim O’Brien, “That AWS Outage Was Nothing: When Meteors Meet Data Centers”, November 3, 2025. Its probability estimates are the author’s illustrative arithmetic, not validated site-specific risk estimates.↩︎
NASA JPL, “Additional Details on the Large Feb. 15 Fireball over Russia”, February 15, 2013, updated March 21, 2013: 17–20 meters; approached within 15 degrees of the Sun’s direction, which detection telescopes “cannot scan”; brightest at 23.3 km; nearly 500, later 440, kilotons. NASA Planetary Defense, “Remembering the Chelyabinsk Impact 10 Years Ago, and Looking to the Future”, February 15, 2023, checked October 4, 2026: windows broken over 200 square miles; more than 1,600 injured, mostly by glass. Estimates vary by method.↩︎
DoorDash, 2020 Form 10-K, checked October 4, 2026: “Total Orders increased to 816 million, or 210% growth compared to the year ended December 31, 2019.” Orders are not a measure of peak infrastructure consumption.↩︎
Western Digital, January 23, 2012 quarterly-results release filed with the SEC, checked October 4, 2026: 28.5 million drives shipped in the quarter ended December 30, 2011, against 52.2 million a year earlier; capacity expected back to pre-flood levels in the September 2012 quarter (forward-looking). The receiving-plan experience is the author’s recollection.↩︎
TrendForce, TrendForce Expects NAND Flash Prices to Keep Rising During 2017 as Industry’s Wafer Capacity Will Expand by Just 6% Annually, December 21, 2016: the 3D-NAND transition cutting planar NAND supply while 3D supply stays tight. A NAND forecast, distinct from DRAM price history.↩︎
TrendForce, Global NAND Flash Revenue Rose by 3.5% QoQ in 2Q18; TrendForce Expects Close to 10% Decline in NAND Flash Prices in 3Q18 due to Continued Oversupply, August 16, 2018: oversupply since 1Q18; NAND contract prices down 15–20 percent on average in 2Q18. Survey data.↩︎
TrendForce, AI Server Demand to Drive Memory Contract Price Increases in 2Q26, March 31, 2026, checked October 4, 2026: quarter-on-quarter forecasts of 58–63 percent (conventional DRAM) and 70–75 percent (NAND); capacity shifting toward HBM and server products; cloud providers signing long-term agreements; enterprise-SSD expansion unlikely before late 2027 or 2028. Forecasts, not realized prices.↩︎
TrendForce, AI Server Demand Sustains Memory Contract Price Increases in 4Q26, While Consumer-Side Pressure Persists, September 30, 2026, checked October 4, 2026: 4Q26 quarter-on-quarter forecasts of 10–15 percent (conventional DRAM) and 15–20 percent (NAND Flash), a moderating pace; server DRAM still undersupplied. Forecasts, not realized prices.↩︎
Michael Lewis, The Big Short: Inside the Doomsday Machine (W. W. Norton, 2010). Burry is among the investors the book follows.↩︎
Financial Crisis Inquiry Commission, The Financial Crisis Inquiry Report, January 2011, chapter 10, p. 194: Burry’s shift from homebuilder stocks (2002) to questioning “the financing that supported this booming market,” his expectation of “a final judgment on housing” when two-year adjustable-rate mortgages sought refinancing, and his credit default swaps by mid-2005. The capacity lesson is the author’s.↩︎
Microsoft Learn, Azure region pairs and nonpaired regions, checked October 3, 2026: pairing does not “provide automatic high availability, disaster recovery capabilities, or failover”; customers should not rely on Microsoft-managed failover as their primary recovery approach, which for GRS storage “is only performed in catastrophic situations and after repeated failed recovery attempts.”↩︎