Specialized Hardware

I call these devices GPUs, as most of the people I work with do; this chapter says “accelerators” only where TPUs and other designs are included. The story I have to tell about GPUs is how little most people know about them, and how often the people requesting them are trying to prove they can compete with an industry that is guaranteed to move faster than they can.

If someone wants to procure hardware to build a novel approach to generative AI, the first question may sound like a joke: are they sane? I mean it as a sanity check, and the record from 2024 makes it a quick one. Training at the frontier already cost on the order of $100 million. The Stanford AI Index estimated the compute for GPT-4 at about $78 million and for Gemini Ultra at about $191 million at cloud rental prices; Epoch AI’s amortized-hardware method puts the same final runs nearer $40 million and $30 million, finds that cost growing about 2.4 times a year, and puts research staff at 29 to 49 percent of the total cost of developing such a model.¹ In April 2024, Anthropic’s chief executive put that year’s models at “of order $100 million” to train and those then in training “closer in cost to $1 billion.”² Attempts below the frontier met their own version of the arithmetic. Databricks introduced its open DBRX model in March 2024, trained on 3,072 H100s for what it said was about $10 million; three weeks later Meta released Llama 3 70B openly, with a higher published MMLU score. In June 2026 Databricks’ chief executive said the cluster had cost about $20 million, of which about $4 million paid for the run that produced DBRX and the rest for idle time, smaller trial runs, downtime and mistakes of inexperience.³ The startups whose business was the model fared worse. Between March and August 2024, Inflection AI, Adept and Character.AI each saw its founders and some of its staff leave for Microsoft, Amazon or Google, which took licenses to its technology as part of the arrangement. Inflection had raised $1.3 billion the year before; Alphabet reports paying Character.AI $2.7 billion in cash in an arrangement that included a non-exclusive license and the hiring of some employees.⁴

That record turns the joke into three questions to ask before buying GPUs for a model of the company’s own. Which available model, hosted or open, fails the task, and by how much? What will the attempt cost through its second and third versions, with staff, trial runs and idle hours counted, not only the final run? And what is the plan when a better model appears next quarter at a lower price? If the answers hold up, the rest of this chapter sizes the fleet. If they don’t, I would look at fine-tuning or serving someone else’s model, which still needs GPUs and the same arithmetic.

How to read this chapter

If the work does need GPUs of its own, the request arrives as a device count, and the plan needs the arithmetic behind it: memory per request, goodput measured against latency objectives, the unit a supplier sells, and the failures the fleet must survive.

1. What a device decides before any arithmetic — memory, interconnect and the unit we can actually buy.
2. The arithmetic a planner owns — weights, KV cache, requests in flight and a bandwidth floor.
3. What must be measured — goodput against latency objectives, and why a utilization percentage misleads.
4. Installed, schedulable, usable — what fragmentation, failure and maintenance subtract from a fleet.
5. An experiment on shared hardware, before the fleet — the serving experiment can run on rented capacity or a research service, before any purchase.
6. Acquiring it — on-demand, capacity blocks, reservations, dedicated rental and ownership, and how I would layer them.
7. The sizing process — the loop, and the record it should leave behind.

A device-hour price is useful, but it does not establish cost per accepted task; compare the full configuration and the organization needed to operate it.

What a device decides before any arithmetic

Memory capacity decides whether the weights and working state fit. Memory bandwidth, arithmetic throughput and the interconnect then limit different phases of the work, and which one binds depends on the model, batch size and serving configuration.

Vendor specifications illustrate the differences: NVIDIA lists 80 GB and 3.35 TB/s for H100 SXM, 141 GB and 4.8 TB/s for H200 SXM, and 180 GB and up to 8 TB/s for B200 SXM.⁵ AMD lists 192 GB for MI300X and 288 GB for MI355X; Google lists 192 GiB per Ironwood chip.⁶⁷ A footprint between 80 and 141 GB fits one H200 but needs two H100s, and two devices bring the communication that comes with splitting a model. The inference doesn’t run backward: a model that fits on the larger device may fit on the smaller one at another precision or a shorter context, so compare at the configuration we would actually run.

The purchasable configuration matters as much as the chip. Azure’s ND H100 v5 VM contains eight H100 GPUs with 80 GB each: 8 × 80 = 640 GB of aggregate GPU memory, connected within the VM through NVLink. ND H200 v5 has the same shape with eight 141 GB H200s, 1,128 GB in all.⁸ Neither is one large buffer; a model too big for one GPU needs a distribution, precision or offload strategy. Write down the VM size, GPUs per VM, memory per GPU and communication path before comparing its hourly price with a single-device offer. The VM is also the unit that fails, gets maintained and gets reserved.

That is one reason GPU hours are not freely interchangeable, a subject Volume II, Chapter 24 returns to. A different chip or VM changes which model configurations fit and how much communication they need, so the experiment has to show that the alternative completes the same work within the same latency and quality requirements.

The arithmetic a planner owns

Memory per request becomes a fleet size only once we know how many requests are resident at the same time, and Chapter 5’s Little’s law, $L = \lambda W$, supplies that. Suppose the hypothetical service peaks at four requests per second; like every figure in this example, that is illustrative, not measured. Each brings up to 6,000 prompt tokens and generates up to 800, inside the 8,000-token allowance above, and the service has two objectives: the first token within two seconds and each later token within 50 milliseconds. A request that just meets both stays about 2 + 799 × 0.05 ≈ 42 seconds, so about 4 × 42 = 168 requests are in the system at the peak, and 168 ÷ 21 = 8 devices by memory alone. $L$ is an average, and random arrivals scatter around it. Counting every request at its full 8,000-token allowance leaves some room for that scatter, because paged allocation charges a request only for the tokens it actually holds.

The arithmetic also runs backward, and that direction is the one I’d want the on-call engineer to know. If decoding runs at 30 milliseconds per token, a request stays about 26 seconds and the fleet holds about 104 requests, five devices’ worth of cache. Let it slow to 50 and the same arrivals need room for 168. A latency regression arrives as a memory shortage. In vLLM, running out of cache space preempts requests and recomputes them later, which slows everything again.¹¹

Memory says whether the requests fit; bandwidth sets a floor under how fast they can run. In ordinary decoding, each step produces one token per request and reads the weights once for the whole batch plus every resident request’s cache. Dividing those bytes by the device’s bandwidth gives the fastest possible step. On a 4.8 TB/s device the weights alone take 70 GB ÷ 4,800 GB/s ≈ 14.6 ms, so a request running by itself cannot exceed about 69 tokens per second. With 21 requests holding an average of 6,400 tokens each, a step also reads 21 × 6,400 × 327,680 bytes ≈ 44 GB of cache, and (70 + 44) GB ÷ 4,800 GB/s ≈ 23.8 ms: at most about 42 tokens per second for each request and 880 for the device. Batching shares the weight read across requests; it does nothing for the cache, which grows with every resident token. Real kernels don’t reach full bandwidth, so the measured step will be slower than the floor. When a benchmark reports a faster one, it is measuring a different configuration — shorter contexts, a smaller cache precision, speculative decoding — and I want to know which.

The floor sits comfortably under the 50-millisecond objective, which makes eight devices plausible rather than proven. Prefill is the main reason for doubt. Processing a 6,000-token prompt is arithmetic-heavy where decoding is bandwidth-heavy, and long prompts arriving while other requests decode delay everyone’s next token. vLLM’s chunked prefill exposes the trade directly: a smaller per-step token budget favors inter-token latency, and a larger one favors time to first token.¹² Only a test with the real prompt lengths says where this configuration lands.

What must be measured

A serving experiment measures something narrower than the accepted work of Chapters 7 and 8. It measures latency-qualified throughput, work completed within the timing requirements, not business-accepted throughput, which also requires a result that meets the customer’s substantive criteria. A fluent incorrect answer passes the timing test.

For a token-generating service, the timing requirement usually comes in two parts. Time to first token (TTFT) covers queueing and the prefill that processes the prompt; time per output token (TPOT) is the average gap between the tokens after that. DistServe’s authors define a device’s goodput as the highest request rate it can serve while a stated share of requests — their example is 90 percent — meets both objectives.¹³ That is the number I want from a serving test, because it already contains the latency requirement that a raw tokens-per-second figure leaves out. vLLM’s benchmark measures it against a running server; for the hypothetical service, one device’s share of the peak is half a request per second:

vllm bench serve --model <served-model> \
  --dataset-name random --random-input-len 6000 --random-output-len 800 \
  --ignore-eos --num-prompts 1000 --request-rate 0.5 --burstiness 0.5 \
  --percentile-metrics ttft,tpot,e2el --metric-percentiles 50,95,99 \
  --goodput ttft:2000 tpot:50 --save-result

Here --goodput counts a request as good only if it meets every listed objective, in milliseconds; a --burstiness below 1 makes arrivals burstier than a Poisson stream; and --ignore-eos holds every response to its full length.¹⁴ A thousand prompts at half a request per second is a run of about 33 minutes. The random dataset uses fixed lengths, which suits a first pass. The run that informs a purchase should replay the customer’s real length distribution, cache states and mix, with rejected and timed-out requests in the report; GuideLLM and AIPerf are comparable harnesses for that. vLLM reports good requests per second at the offered rate, so the command has to run at several rates. The highest rate at which the stated share of requests stays good is the device’s goodput, and the fleet is the peak divided by it. At 0.5 per device, the measurement agrees with the memory estimate on eight devices; at 0.4 it says ten; if prefill interference holds it to 0.3, fourteen.

A utilization percentage can’t do that job. The figure nvidia-smi reports as GPU utilization is the share of the sample period in which at least one kernel was running, so one small kernel can hold it near 100 percent while most of the chip waits.¹⁵ DCGM’s profiling fields are more specific: dcgmi dmon -e 1001,1002,1004,1005 samples graphics-engine activity, the share of time multiprocessors have a warp active, tensor-pipe activity and memory-bandwidth activity. Even these need care: a warp waiting on memory counts as active, and GPUs that can’t collect every metric group at once multiplex them, so fast sampling can return zeros. I would use these fields the way Chapter 5 uses Gregg’s USE questions, to explain why goodput moved after a change, such as a batch policy that raised aggregate throughput while interactive requests missed their objectives. Nsight Systems timelines and PyTorch Profiler traces take the explanation down to individual kernels and, in specialist hands, show what to test next.

For training, the question becomes when a usable model will exist. The device-level efficiency measure is model FLOPs utilization (MFU): training throughput, counted in the model’s own arithmetic, as a share of the hardware’s peak. Meta reports 38 to 43 percent BF16 MFU for its Llama 3 405B configurations, slipping from 43 percent on 8K GPUs to 41 on 16K because each data-parallel group then carried a smaller batch.¹⁶ MLPerf’s benchmarks define what a time-to-quality measurement means under controlled conditions — Training for compute, Storage for the data delivery and checkpoint work that training runs depend on.¹⁷ A published result describes its benchmark and configuration, not the enterprise’s plan, which also has to schedule the allocation, the data supply, checkpoint writes, restarts after interruption and delivery to the team that will use the model, several of which cannot overlap.

Installed, schedulable, usable

Three different numbers hide behind “our GPU capacity.” Installed capacity is what the contracts and purchase orders say. Schedulable capacity is the part a scheduler can actually give the work in the shapes the work needs. Usable capacity is what remains after failures, maintenance and recovery take their share. Google’s training-goodput framing divides the losses much the same way, into scheduling, runtime and program goodput (the last being MFU); its advice to protect the first with reserved capacity comes from a vendor that sells reservations.¹⁸

Fragmentation and shape

A synchronous training job needs all its accelerators at once, which schedulers call gang scheduling, and it runs faster when they sit close together on the fastest interconnect. Those two requirements turn free capacity into stranded capacity. A 2019 study of a two-month trace from a Microsoft multi-tenant GPU cluster shows the trade. Relaxing locality shortened queueing, especially for large jobs, but scattered jobs synchronized more slowly, and allocation made the cluster look busy while the GPUs in use averaged about 52 percent hardware utilization. Around 30 percent of jobs were killed or ended unsuccessfully, mostly from programming errors that surfaced early, so the authors suggest catching them on a small pool before a job takes a large allocation.¹⁹ The hardware has changed since then; gang scheduling and locality have not. TPUs make the shape explicit: TPU7x slices come in fixed three-dimensional topologies, and slices larger than 64 chips are built from 4 × 4 × 4 cubes, so free chips that don’t form the next job’s shape don’t help it.²⁰

So I’d rather see a capacity report that counts free shapes than free GPUs: how many whole eight-GPU machines, how many complete NVLink domains, how many slices of the size the next training run needs. A report of 300 free GPUs scattered two and three to a machine tells me nothing about whether a 64-GPU job starts tomorrow.

Failure and maintenance

Return to the serving example. Eight 141 GB devices make exactly one ND H200 v5 VM. If the business requires the service to survive the loss of any one VM — the requirement Volume II, Chapter 18 builds its failure groups around — the fleet needs two VMs and sixteen devices, twice what the load requires. When the requirement is a whole number of units, the N + 1 overhead is the unit size divided by the requirement: 8 ÷ 8 = 100 percent here, against 8 ÷ 40 = 20 percent for a service that needs forty devices and buys six VMs instead of five. That is one reason a small GPU deployment looks so expensive beside a hosted API: its redundancy comes in lumps the size of a machine. A smaller VM size shrinks the lump if one has enough memory per device for the model. A replica split across several devices is itself the failure unit; lose one device and the replica stops.

Maintenance draws on the same allowance. Azure live-migrates most VMs during host updates, but not the N-series GPU families, and maintenance that needs a reboot comes with a self-maintenance window, typically 35 days for host maintenance and 14 for hardware decommissioning.²¹ With two VMs, rebooting one leaves the service without failure headroom for the duration, so I’d schedule it for the daily low, one VM at a time, and agree in advance who accepts that exposure. Google’s reservations show the same trade from the supplier’s side. For reserved GPU clusters, a managed mode replaces faulty machines but holds back part of the reserved capacity so instances can restart; an all-capacity mode delivers the whole reservation and makes the customer responsible for failures and planned maintenance.²² In comparing the modes, I would count the hold-back as capacity we pay for and cannot schedule, and the other mode’s extra capacity as usable only once our own repair process has been rehearsed.

Rack-scale systems move the failure domain up again, which is Volume II, Chapter 18’s geometry at a larger size. An NVL72 rack joins 72 GPUs on 18 compute trays through nine NVLink switch trays, with shared power shelves and liquid-cooling manifolds.²³ A job shaped to the whole NVLink domain cannot run on the 68 GPUs left after one tray fails. One answer is to shape jobs to fewer GPUs than the domain holds and keep a spare tray’s worth idle inside it. That costs those GPUs all the time, and whether it beats waiting for repair depends on the repair time, a number to get from the supplier and then verify.

Training turns every component into a shared failure domain. Because training is synchronous, Meta reports, a single GPU failure may require restarting the entire job. Over a 54-day snapshot of Llama 3 pre-training on up to 16K H100 GPUs, the job was interrupted 466 times: 47 planned, for automated maintenance and operator changes, and 419 unexpected, about one every 3.1 hours (54 × 24 ÷ 419). About 78 percent of the unexpected interruptions were attributed to confirmed or suspected hardware problems, with GPUs the largest category. Meta still reports more than 90 percent effective training time, meaning useful training over elapsed time.²⁴ Those figures describe one run, not a component failure rate. Read together, they imply a recovery speed. If the snapshot matched the 90 percent figure, its interruptions cost at most a tenth of the elapsed time, about 17 minutes each on average (0.10 × 54 × 24 × 60 ÷ 466 ≈ 16.7), to cover detection, replacement, restart and the work redone since the last checkpoint. I’d budget a first large run on our own measured recovery time, not on that one.

An experiment on shared hardware, before the fleet

The serving experiment at the center of this chapter has to run on real hardware, and a fleet proposal usually assumes the hardware is already in hand. It doesn’t have to be. A first experiment — does this model hold this many requests at this latency? — is exactly the work the shared rental services are priced for, and the cost is the meter reading, not a machine.

The first option to investigate needs no GPU at all. The National Deep Inference Fabric (NDIF) is a research service, supported by the U.S. National Science Foundation, that hosts large open models on shared GPU pods for remote execution through the NNsight Python library. The researcher writes intervention code against a model’s internal layers, adds remote=True, and the code runs where the weights live; only the saved results come back. A machine with no GPU can write and run interventions against a 405-billion-parameter model.²⁵ For the interpretability questions — what the model represents, what a circuit does, what changes when a layer is patched — this is a step before renting, and it is free to register. Its limits are the research service’s limits: shared queues, a one-hour cap per request, and a catalog of deployed models to check before submitting.

The rented options take over when the work needs a whole machine or a serving stack of its own: a vLLM benchmark like the one above, a fine-tuning run, a checkpoint schedule, anything with its own runtime state. Runpod rents dedicated GPU pods by the second, with SSH and Jupyter access, storage billed separately while the pod is stopped, and a default spend limit that contains a mistyped experiment.²⁶ Modal runs the same custom Python and PyTorch code remotely, choosing the GPU from a catalog that reaches from a T4 to a B300, eight devices to a container, and bills by the second for the GPU, the CPU cores and the memory the code actually uses — nothing while it is idle.²⁷

The published rates make the shape of the comparison visible. Modal prices an H100 at $0.001097 per second, which is $3.95 per GPU-hour; Runpod’s H100 starts near $2.89 to $3.49 per hour; both publish an A100 or an RTX-class card in the same page of the catalog.²⁸ Against an owned H100 at roughly $25,000 to $30,000 through authorized resellers, the arithmetic in Chapter 2 applies unchanged: a day of experimentation is dollars, a month of it is less than one depreciation schedule, and none of it pays for the room, the power contract or the failure allowance.²⁹ These are list prices from vendor pages, checked on the date in the footnote; they move, and the commitment prices move with them. The point is not which vendor is cheaper this week. It is that the experiment’s cost is proportional to the experiment, and a purchase’s cost is not.

Acquiring it

On-demand instances, capacity blocks, dedicated rental and ownership leave the enterprise with different obligations. Chapter 2 covers what elasticity costs at GPU scale and how capacity reservations are billed.

On-demand GPU instances charge for use, but the required type may be unavailable when requested. AWS Capacity Blocks reserve supported accelerated instances for a future window, and their terms shape the plan as much as their price. A block runs in one-day increments up to 14 days or seven-day increments up to 182, holds up to 64 instances, can start up to eight weeks out, is paid up front at a price set by supply and demand, and can’t be cancelled, moved or split. Its instances are placed close together in an EC2 UltraCluster, which helps with the locality problem above. A block ends at 11:30 UTC, and AWS begins terminating its instances at 11:00 UTC on the final day, so the last checkpoint has to land before 11:00, not at 11:29.³⁰

In Azure, check the regional vCPU quota and the selected VM-family quota before planning a GPU deployment; as Chapter 6 showed, quota and physical supply are separate checks.³¹ Two commands cover the permission side for a candidate size and region:

az vm list-skus --location <region> --size Standard_ND96isr_H200_v5 --all --output table
az vm list-usage --location <region> --output table

The first shows whether the size is offered in that region, in which zones, and with what restriction for this subscription; the second lists regional and per-family vCPU quota against current use.³² Neither shows whether the hardware will be there when we deploy. A capacity reservation would hold the size once Azure accepted the request, but as Chapter 2 noted, support is SKU-specific: the documentation lists several NC and NV families and excludes ND-series and NCads H100 v5.³³ For an ND-series plan, then, establish the supply arrangement for that exact deployment, and keep a tested alternative in case the required allocation is unavailable.

The third option is dedicated rental under a supplier contract. Compare quotes for the same usable configuration and obligations, including who holds the spares and how fast a failed node is replaced, since those set the usable fraction. The fourth is ownership, where Chapter 2’s presumption in favor of buying a service still applies and the comparison includes capital, power, cooling, staffing, maintenance and replacement. An accounting useful life is an estimate, and large operators revise it in both directions: Amazon lengthened its servers’ useful life from five years to six at the start of 2024, then shortened it back to five for a subset of servers and networking equipment a year later and retired some early, citing the pace of AI and machine-learning development.³⁴ Test the comparison across several demand and replacement scenarios rather than one depreciation schedule.

The facility can decide feasibility before price does. NVIDIA’s DGX guide puts an NVL72 rack at approximately 120 kW, with liquid carried to cold plates on the CPUs and GPUs; a room without liquid-cooling distribution can’t host it.³⁵ Large synchronous jobs also swing the load: Meta reports data-center power swings on the order of tens of megawatts when tens of thousands of GPUs pause together for checkpoints or collective communication.³⁶

A different accelerator architecture can also change the software. Google’s TPU7x documentation supports JAX and PyTorch but not TensorFlow, and JAX sees each Ironwood chip as two devices,³⁷ so the comparison has to include the model port, compiler and library support, and the debugging and operating skill the new platform needs.

This is how I would layer the arrangements for a serving fleet with some training beside it; it is judgment, not a rule. The base — the devices the service needs at its ordinary daily peak, plus the failure allowance — goes on the longest supply arrangement the evidence supports, with the take-or-pay arithmetic of Volume I, Chapter 2 and Volume II, Chapter 20 deciding how much. Training runs and predictable peaks go on time-bounded blocks sized to the job’s useful duration divided by its expected usable fraction. If rehearsal runs show 85 percent usable time (an illustrative figure), a job that needs 20 days of useful training needs about 23.5 days of block, and because blocks longer than 14 days come in whole weeks, the purchase is 28 days or a combination of shorter blocks; the idle remainder is part of the price. Overflow goes to on-demand capacity where it can be found, to a hosted model where data placement allows (Chapter 7), or to a degraded mode agreed in advance: shorter contexts, a smaller model, or a queue with an honest wait. A long GPU commitment also bets on the configuration: the model’s architecture, precision and context length decide how many requests a device holds, so a model change can strand a commitment that the volume forecast still supports.

The sizing process

The device fleet joins the portfolio as its own pool with its own ledger lines. Rented GPUs are a cloud charge and sit in the same view as the rest of the bill. Owned GPUs would join the private data center that Chapter 1’s ledger already carries as an allocated thirty-day resource cost, with the purchase cash, the power contract and the refresh obligation tracked beside that allocation rather than hidden in it. The constructed portfolio owns no GPUs, so an ownership proposal starts as a comparison against the service or rental it would replace, made as honestly as Chapter 13 treats the migration’s double bubble.

Chapter 11 looks past these devices to architectures still proving themselves. The devices themselves have begun to acquire a public price, and Volume II, Chapter 24 follows them there.

---

Notes and sources

1. Stanford Institute for Human-Centered Artificial Intelligence, AI Index Report 2024, “Research and Development”, April 2024, for estimated training compute costs of about $78 million (GPT-4) and $191 million (Gemini Ultra) at cloud rental prices; Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej and David Owen, “The Rising Costs of Training Frontier AI Models,” Epoch AI, 2024, https://arxiv.org/abs/2405.21015, for amortized hardware and energy costs (about $40 million and $30 million), growth of about 2.4 times a year since 2016, and R&D staff at 29 to 49 percent of total development cost for the models examined. Checked October 4, 2026. Both are estimates from public information, not company disclosures, and the two methods disagree for the same runs.↩︎

2. Ezra Klein, “Transcript: Ezra Klein Interviews Dario Amodei”, The Ezra Klein Show, New York Times, April 12, 2024, checked October 4, 2026, through search excerpts of the transcript and The Decoder’s report of the same quotation. A laboratory head’s order-of-magnitude estimate of rented-compute cost, not an audited figure; in the same answer he projected $5 billion to $10 billion for 2025 and 2026.↩︎

3. Databricks, “Introducing DBRX: A New State-of-the-Art Open LLM”, March 27, 2024, for the 3,072 H100s and DBRX Instruct’s 73.7 percent five-shot MMLU; Kyle Wiggers, “Databricks spent $10M on new DBRX generative AI model”, TechCrunch, March 27, 2024, for the company’s $10 million figure; Meta, Llama 3 model card, for the April 18, 2024 release date and 82.0 percent five-shot MMLU for Llama 3 70B Instruct; Eric Newcomer, “Databricks CEO Ali Ghodsi: A Costly AI Training Foray”, Newcomer, June 16, 2026, for the $20 million cluster, the $4 million final run and the account of the remainder. Checked October 4, 2026; the Newcomer article is paywalled after the opening paragraphs that carry these figures. Benchmark scores are each vendor’s own, and the costs are the company’s statements, not audited accounts.↩︎

4. Reuters, “Microsoft agreed to pay Inflection $650 mln while hiring its staff”, March 21, 2024, including Inflection’s $1.3 billion raise in 2023; CNBC, “Amazon hires execs from AI startup Adept and licenses its technology”, June 28, 2024; Alphabet Inc., Form 10-Q for the quarter ended September 30, 2024, acquisitions note, for the August 2024 non-exclusive license, the $2.7 billion cash payment and the hiring of certain employees. Checked October 4, 2026. Terms beyond these disclosures are not public. Adept’s own announcement could not be retrieved; CNBC reports its statement that continuing to build its own models would have required more capital.↩︎

5. NVIDIA, HGX AI Factory reference architecture, “Components,” Table 1, https://docs.nvidia.com/enterprise-reference-architectures/hgx-ai-factory-h100-h200-b200/latest/components.html, checked October 3, 2026. Vendor specifications for H100, H200 and B200 SXM memory and bandwidth, not measured serving throughput.↩︎

6. AMD, Instinct accelerator product pages, accessed September 26, 2026, and ROCm GPU hardware specifications, checked October 3, 2026. 192 GB HBM3 (MI300X) and 288 GB HBM3E (MI355X); the ROCm table gives the same capacities in GiB. Vendor specifications; rental rates cited by aggregators change frequently and are not quoted here.↩︎

7. Google, “Ironwood: The first Google TPU for the age of inference”, 2025, and Google Cloud, “TPU7x (Ironwood)”, last updated October 2, 2026, checked October 3, 2026. Vendor documentation: 192 GiB of HBM per chip in the specification table (the page’s prose says 192 GB); 9,216-chip pods; supported slice shapes from 2 × 2 × 1 to 8 × 16 × 16, with slices above 64 chips built from 4 × 4 × 4 cubes; JAX and PyTorch supported, TensorFlow not; each chip exposed to JAX as two devices.↩︎

8. Microsoft Learn, “ND-H100-v5 sizes series” and “ND-H200-v5 size series”, checked October 3, 2026. Eight GPUs per VM with intra-VM NVLink; specifications do not establish serving throughput or transparent pooling of device memory.↩︎

9. vLLM, “Quantization”, checked October 3, 2026. Methods and supported hardware differ by release. Raw weight arithmetic excludes scales, metadata and runtime allocations.↩︎

10. Woosuk Kwon and colleagues, “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP 2023, https://arxiv.org/abs/2309.06180. Describes the cache memory lost to reservation and fragmentation under contiguous allocation, and paged allocation as the remedy. Its OPT-13B example applies the same per-token formula: 2 × 5,120 × 40 layers × 2 bytes, about 800 KB.↩︎

11. vLLM, “Optimization and Tuning”, checked October 3, 2026. Preemption and recomputation when KV-cache space runs out; chunked prefill enabled by default in V1, with max_num_batched_tokens trading inter-token latency against time to first token. Engine documentation, not a measurement of the hypothetical configuration.↩︎

12. vLLM, “Optimization and Tuning”, checked October 3, 2026. Preemption and recomputation when KV-cache space runs out; chunked prefill enabled by default in V1, with max_num_batched_tokens trading inter-token latency against time to first token. Engine documentation, not a measurement of the hypothetical configuration.↩︎

13. Yinmin Zhong and colleagues, “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving,” OSDI 2024, https://arxiv.org/abs/2401.09670. Defines TTFT, TPOT, SLO attainment and per-GPU goodput; the 90 percent attainment goal is the paper’s example, not a standard.↩︎

14. vLLM, “bench serve”, with flag behavior checked against vllm/benchmarks/serve.py and vllm/benchmarks/datasets/datasets.py on the project’s main branch, October 3, 2026, and the flags rechecked there October 4, 2026; GuideLLM, https://github.com/vllm-project/guidellm; AIPerf, https://github.com/ai-dynamo/aiperf. The command was not run for this book, and the harnesses were not comparatively benchmarked. Flags change between releases; check vllm bench serve --help for the installed version.↩︎

15. NVIDIA, nvidia-smi documentation, for the definition of GPU utilization; NVIDIA, DCGM Feature Overview, for profiling fields 1001, 1002, 1004 and 1005 and metric multiplexing; NVIDIA, Nsight Systems User Guide; PyTorch, Profiler recipe. Checked October 3, 2026; documented capabilities, not measurements of a constructed deployment.↩︎

16. Aaron Grattafiori and colleagues, “The Llama 3 Herd of Models,” 2024, https://arxiv.org/abs/2407.21783, section 3.3. Up to 16K H100 GPUs; BF16 MFU of 38–43 percent; a 54-day snapshot with 466 job interruptions (47 planned, 419 unexpected, about 78 percent of those attributed to confirmed or suspected hardware issues, GPU issues 58.7 percent); more than 90 percent effective training time; power swings on the order of tens of megawatts. One organization’s run, not a failure-rate forecast. The per-interruption bound in the text is the author’s arithmetic and assumes the snapshot matched the run-level 90 percent.↩︎

17. MLCommons, MLPerf Training and MLPerf Storage, checked October 3, 2026. Results are vendor submissions under benchmark conditions; version differences matter and are documented per release.↩︎

18. Vaibhav Singh and Daniel Herrington, Google Cloud, “Goodput metric as measure of ML productivity”, April 10, 2024. Vendor framing of scheduling, runtime and program goodput for training, with a recommendation to reserve resources and keep hot spares.↩︎

19. Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao and Fan Yang, “Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads,” USENIX ATC 2019, https://www.usenix.org/system/files/atc19-jeon.pdf. A two-month trace from one cluster of an earlier GPU generation; the mechanisms transfer more readily than the percentages.↩︎

20. Google, “Ironwood: The first Google TPU for the age of inference”, 2025, and Google Cloud, “TPU7x (Ironwood)”, last updated October 2, 2026, checked October 3, 2026. Vendor documentation: 192 GiB of HBM per chip in the specification table (the page’s prose says 192 GB); 9,216-chip pods; supported slice shapes from 2 × 2 × 1 to 8 × 16 × 16, with slices above 64 chips built from 4 × 4 × 4 cubes; JAX and PyTorch supported, TensorFlow not; each chip exposed to JAX as two devices.↩︎

21. Microsoft Learn, “Maintenance and updates for virtual machines in Azure”, checked October 3, 2026. Live migration excludes the G, L, N and H series; reboot-requiring maintenance comes with self-maintenance windows that are typical, not guaranteed, and urgent maintenance can be shorter.↩︎

22. Google Cloud, “Cluster management capabilities”, last updated September 30, 2026, checked October 3, 2026. Vendor documentation; the size of the managed-mode hold-back is not published there. TPU reservations offer a similar choice under “TPU capacity modes”.↩︎

23. NVIDIA, DGX GB200 and GB300 rack system user guide, “Hardware”, checked October 3, 2026. NVL72 rack composition, power shelves and bus bar, liquid cooling through manifolds and cold plates, and rack power consumption of approximately 120 kW. Vendor documentation; site requirements depend on the configuration ordered and the facility’s design.↩︎

24. Aaron Grattafiori and colleagues, “The Llama 3 Herd of Models,” 2024, https://arxiv.org/abs/2407.21783, section 3.3. Up to 16K H100 GPUs; BF16 MFU of 38–43 percent; a 54-day snapshot with 466 job interruptions (47 planned, 419 unexpected, about 78 percent of those attributed to confirmed or suspected hardware issues, GPU issues 58.7 percent); more than 90 percent effective training time; power swings on the order of tens of megawatts. One organization’s run, not a failure-rate forecast. The per-interruption bound in the text is the author’s arithmetic and assumes the snapshot matched the run-level 90 percent.↩︎

25. NDIF (National Deep Inference Fabric), ndif.us, accessed October 14, 2026, and NNsight, “Remote Execution”, accessed October 14, 2026. Free account registration; models are served on shared GPU pods (NCSA Delta, including H200 nodes) for research use; each request carries a one-hour time limit and the fairness policy is described as still evolving; remote requests run only against models currently deployed on the service, so check the status page before submitting. A research service’s terms, not a commercial product; commercial terms should be verified before any production use.↩︎

26. Runpod, “Connection options” and “Pricing”, accessed October 14, 2026. Pods are billed by the second for compute and storage with no ingress or egress fees; a default spend limit of $80 per hour across all resources applies to new accounts, and account balances reaching $0 stop the pods. Vendor terms; verify current rates in the deployment console.↩︎

27. Modal, “GPU acceleration” and “Pricing”, accessed October 14, 2026. GPU types from T4 through B300, up to eight per container; billing is per second for GPU, CPU and memory, with no charge for idle resources. Vendor documentation and list pricing; regional selection and non-preemptible execution carry multipliers of 1.15–1.75x and 3x respectively.↩︎

28. Modal’s published rate of $0.001097 per second for an H100 equals $3.95 per hour (0.001097 × 3,600); Runpod lists H100 rates from about $2.89 (PCIe, Secure Cloud) to $3.49 (SXM, Secure Cloud) per hour. Rates read October 14, 2026 from each vendor’s published pages and third-party comparisons of them; rental prices change frequently, which is the reason this book quotes them only with dates.↩︎

29. Runpod, “NVIDIA H100 Price”, accessed October 14, 2026, for authorized-reseller pricing of approximately $25,000–$30,000 for a standalone H100 PCIe card. Reseller market estimates, not a quotation; Chapter 2 carries the buy-versus-rent arithmetic and its commitment terms.↩︎

30. Amazon Web Services, “Capacity Blocks for ML”, checked October 4, 2026, for instance limits, start window, placement, cancellation and termination times; “Find and purchase Capacity Blocks” and “Capacity Blocks pricing and billing”, checked October 3, 2026, for durations and pricing. Vendor terms for supported instance types; availability and 64-instance blocks vary by type and Region.↩︎

31. Microsoft, “Check vCPU quotas,” https://learn.microsoft.com/en-us/azure/virtual-machines/quotas, checked September 30, 2026. Regional and VM-family quota checks are separate from available infrastructure.↩︎

32. Microsoft Learn, “az vm” command reference, entries for az vm list-skus and az vm list-usage, checked October 3, 2026. list-skus applies subscription-level restrictions; --all includes sizes not available to the subscription, and --size accepts a partial name.↩︎

33. Microsoft, “On-demand capacity reservation,” https://learn.microsoft.com/en-us/azure/virtual-machines/capacity-reservation-overview, checked October 4, 2026. Supply reservation versus Reserved Instance discount, acceptance conditions and supported and excluded VM families. Recheck support for the specific SKU and location when procuring.↩︎

34. Amazon.com, Inc., Form 10-K for the fiscal year ended December 31, 2024, filed February 2025, critical accounting estimates. Useful-life changes effective January 1, 2024 (five to six years) and January 1, 2025 (six to five years for a subset), early retirements with about $920 million of accelerated depreciation and related charges in the fourth quarter of 2024, and the stated reason. One company’s accounting estimates, not a forecast for any other fleet.↩︎

35. NVIDIA, DGX GB200 and GB300 rack system user guide, “Hardware”, checked October 3, 2026. NVL72 rack composition, power shelves and bus bar, liquid cooling through manifolds and cold plates, and rack power consumption of approximately 120 kW. Vendor documentation; site requirements depend on the configuration ordered and the facility’s design.↩︎

36. Aaron Grattafiori and colleagues, “The Llama 3 Herd of Models,” 2024, https://arxiv.org/abs/2407.21783, section 3.3. Up to 16K H100 GPUs; BF16 MFU of 38–43 percent; a 54-day snapshot with 466 job interruptions (47 planned, 419 unexpected, about 78 percent of those attributed to confirmed or suspected hardware issues, GPU issues 58.7 percent); more than 90 percent effective training time; power swings on the order of tens of megawatts. One organization’s run, not a failure-rate forecast. The per-interruption bound in the text is the author’s arithmetic and assumes the snapshot matched the run-level 90 percent.↩︎

37. Google, “Ironwood: The first Google TPU for the age of inference”, 2025, and Google Cloud, “TPU7x (Ironwood)”, last updated October 2, 2026, checked October 3, 2026. Vendor documentation: 192 GiB of HBM per chip in the specification table (the page’s prose says 192 GB); 9,216-chip pods; supported slice shapes from 2 × 2 × 1 to 8 × 16 × 16, with slices above 64 chips built from 4 × 4 × 4 cubes; JAX and PyTorch supported, TensorFlow not; each chip exposed to JAX as two devices.↩︎