Spot Instances Were a Lie

Do you use spot instances? Most engineers say yes, for CI runners, batch jobs, checkpointed training runs. Ask how spot works and you will get the 2009 answer: you bid, and if the market price rises above your bid, the machine goes away. That answer has been wrong since 2017. Spot is seventeen years old, older than a lot of the people writing about it, and over that stretch the product has been rebuilt twice while the explanation most engineers carry around has not been updated at all.

The auction that wasn’t

Amazon launched EC2 Spot in December 2009. Jeff Barr’s launch post describes an auction in plain terms: AWS evaluates open bids, computes a new spot price for each instance type, terminates the instances whose bids fall below it, and launches the requests at or above it.¹ You set a maximum bid, you watched the price history, and a higher bid bought you longer runtime. For a few years that was the accepted mental model, and small teams built crawlers, transcoders, and CI jobs on top of it.

Here is the part almost nobody knows. Researchers who deconstructed Amazon’s spot price traces concluded that most of the time the prices were not market-driven. Their finding: prices were “likely to be generated most of the time at random from within a tight price range via a dynamic hidden reserve price mechanism,” an undisclosed algorithm moving a hidden reserve price, not a real crowd of bidders setting the number.² The auction was a story the pricing told. The mechanism behind it was an algorithm with a budget of spare capacity to dispose of. That is the first thing to unlearn: what made spot cheap was never really competition between your bid and mine.

Some of you might find that hard to believe

A price that moves but is not set by buyers sounds like forum-thread material, so here is the homework. A team at UC Santa Barbara recorded spot price histories from 2015 onward, and their 2018 technical report walks through one day in detail: January 26, 2016, when a c4.8xlarge in us-west-1b sold for about thirty cents through the morning, jumped to $2.03 for twenty minutes, then to $22.08, and sat at exactly that price for the next three and a half hours. The on-demand price for the same machine was $1.19.³ A crowd of bidders does not hold a price at eighteen times the fixed rate for hours; bidding produces movement, and a reserve algorithm produces plateaus. The team’s own read of the archive agreed with the deconstruction paper: the spot price “is not market-driven but is set by Amazon according to an undisclosed algorithm,” with real bids behind only about 2 percent of observed prices.⁴

The second check covers the years after the auction died. In May 2023, spot prices rose as much as 55 percent relative to on-demand while a research group that had launched 5.5 million spot instances watched its preemption rates nearly quadruple in a few months.⁵ Eric Pauley, who published the numbers, pointed out that this ran against the economics: spot had just lost the one thing that justified a discount, the ability to pay your way to reliability, so the discount should have grown. A former AWS engineer who worked on spot answered publicly and confirmed the mechanism: AWS raises the hourly price on heavily used instance types on purpose, “to spread out the load across their many instance types.”⁶ The algorithm did not go away in 2017. It stopped being hidden. The same UCSB team measured the 2017 switch itself: average spot prices rose 37 to 61 percent across North American regions.⁷ The smoothing was not free.

2017: the provider takes over pricing

At re:Invent 2017, AWS replaced the auction. Bidding went away, prices began adjusting “more gradually, based on longer-term trends in supply and demand,” and purchasing became a synchronous request that either gets fulfilled or does not, with no bid analysis and no standalone API.⁸ AWS’s own follow-up put the change in one line: spot prices went from being “terminated in ascending order of bids” to being “determined by supply and demand for Amazon EC2 spare capacity, not bid prices.”⁹ AWS reported typical savings of 70 to 90 percent and a typical interruption frequency under 5 percent.

Why did AWS do this? Because the auction was bad at its job, and AWS could see it. Volatile prices had turned bidding into a skill; AWS ran a whole best-practices series in 2015 teaching customers to read 90-day price histories and hunt for pools whose prices were stable.¹⁰ But the price signal was noise, not information, which the deconstruction paper had already shown. When the number you ask people to optimize against is an algorithm’s output, the honest fix is to stop pretending it is a market. So the price became a property of the capacity pool, and interruption became a capacity event rather than a pricing event.

What the last decade actually changed

The 2017 repricing was the hinge, and four more dated changes rebuilt the product around it.

In 2019, capacity replaced price as the signal. AWS added a capacity-optimized allocation strategy, and its own post states the shift plainly: “Spot pricing changes slowly over time based on long-term trends in supply and demand, but capacity fluctuates in real time.”¹¹ The skill that used to be “find the cheap pool” became “find the deep pool.” In 2021, the middle product died. Spot Blocks, launched in 2015 to sell guaranteed duration at a spot discount, closed to new customers that July and ended entirely at the end of 2022, with AWS’s advice reduced to: interruption-tolerant workloads use spot, everything else uses on-demand.¹² Between 2021 and 2023, the competitors converged on the same design. Google’s Preemptible VMs were the opposite bet, fixed pricing with a 24-hour runtime cap; in October 2021 they became Spot VMs with provider-set pricing and no cap at all.¹³ Azure publishes per-hour eviction rates computed from the last seven days of history, so you can pick a size by how often it gets evicted, not just what it costs.¹⁴ Everyone landed in the same place: provider-set prices, short notices, and interruption data published as product information.

Then 2023 turned the mechanism around. Capacity Blocks for ML let you prepay for future-dated accelerator capacity, with reservation prices “updated regularly based on trends in supply and demand.”¹⁵ That is the spare-capacity pricing logic pointed the other way. Spot sells capacity that already exists, at a discount, because you accept the risk of losing it. A Capacity Block sells the removal of that risk before the capacity exists, and AWS reprices the reservations as demand builds. When accelerator supply is loose, the guarantee is cheap. When every team in your field is targeting the same stretch of the calendar, the guarantee carries a premium. What is actually scarce is not the supply, which grows every year. It is the overlap. The product is no longer leftover capacity. The product is certainty, and certainty has its own market.

One quiet inversion tells you how far things moved. Today, if you specify a maximum price on a spot request, your instances are interrupted more frequently than if you leave it unset, per Amazon’s own documentation.¹⁶ The knob that defined spot in 2009 is now counterproductive. Alongside that, spot is deliberately not covered by Savings Plans, so the two discount products stay separate markets.¹⁷

What I expect in the next five years

Three predictions, offered as predictions. First, spot prices become the reference price. The book’s final chapter argues that compute already has markets in everything except a shared signal of what it costs, because every rate is negotiated privately and kept confidential (chapter 24). Spot is the one published price that actually moves, and if compute futures ever list on a U.S. exchange (the proposals are lined up in chapter 24), whatever index they track will be built on provider-set, spot-like pricing. Expect internal transfer prices to borrow spot as their anchor: those are the hand-built spreadsheets companies use to charge one team for another team’s usage, and the rates in them are guesses that age badly, because the underlying prices move weekly while the spreadsheet moves annually. Spot is the one public, fine-grained, moving price, and when the internal price and the market disagree too loudly, the market wins, because somebody in the room can read the spot history too.

Second, the spot/committed boundary blurs. Spot Blocks failed because it sold duration at a discount; Capacity Blocks sells time against spare-capacity repricing (chapter 20). Expect more products in that seam: interruptible accelerator capacity with reserved windows, prepaid interruption insurance, spot-like pools with committed minimums. What customers actually buy is not “spot versus committed,” it is who carries the interruption risk and at what price.

Third, interruption data becomes first-class product information. Azure already publishes eviction rates per hour per SKU; AWS has spot placement scores and a rebalance recommendation signal that fires before the two-minute notice.¹⁸ Expect richer interruption dashboards and autoscalers picking instance types by predicted interruption frequency, not just price. The skill that used to be “read the price history” becomes “read the interruption history.”

If you plan capacity

If your mental model of spot is “bid and hope,” you are planning against a product that stopped existing in 2017. The questions have changed: not “what is my maximum bid” but “which pools are deep, how often do they interrupt, and where does my interruption risk actually sit?” Borrowed capacity changes the failure calculation, and the teams that treat it as a designed-for part of the architecture rather than a coupon get real savings (chapter 18). When the autoscaler itself is picking the machines, the person who understands what spot is becomes the person who understands what the fleet is (chapter 2).

This is the territory of Scale Without Surprise, Volume II, whose market chapter looks at exactly this drift, from provider-set pricing to compute futures that do not quite exist yet (chapter 24). Go buy the book. And if you would rather read it first, the whole thing is free to read online, and the spot history here is one small corner of it.

Sources

---

1. Jeff Barr, “Amazon EC2 Spot Instances – And Now How Much Would You Pay?” AWS News Blog, December 2009: launch post — “we’ll evaluate the open bids for each Region and compute a new Spot Price for each instance type. After that we’ll terminate any Spot Instances with bids below the Spot Price, and launch instances for requests with bids higher than or at the new Spot Price” (accessed October 18, 2026).↩︎

2. “Deconstructing Amazon EC2 Spot Instance Pricing,” ACM preprint: spotprice-acmsmall.pdf — “usually prices are not market-driven, as sometimes previously assumed. Rather, they are likely to be generated most of the time at random from within a tight price range via a dynamic hidden reserve price mechanism”; the study also identifies the 14 December 2009 public announcement and a pricing-mechanism change around 8 January 2010 (accessed October 18, 2026).↩︎↩︎

3. Rich Wolski, Gareth George, Chandra Krintz, and John Brevik, “Analyzing AWS Spot Instance Pricing,” UC Santa Barbara Computer Science Technical Report 2018-02: report — the c4.8xlarge us-west-1b history for January 26, 2016: “the average price for a c4.8xlarge spot instance in the us-west-1b AZ was approximately $0.31”; “the price rose to $2.028 for approximately 20 minutes and then spiked to $22.08”; “remained fixed until approximately 2:00 PM”; on-demand “for a c4.8xlarge in us-west-1 is $1.19”; “we (like the authors of [1]) speculate that the $22.08 price is a hidden reserve price”; quoting the deconstruction paper’s finding: “the spot price is not market-driven but is set by Amazon according to an undisclosed algorithm” and “prices do not represent real client bids 98% of the time”; and on the 2017 switch: “overall the average price change is an increase of between 37% and 61%” (accessed October 18, 2026).↩︎↩︎

4. Eric Pauley, “Farewell to the Era of Cheap EC2 Spot Instances,” May 2, 2023: post — “Since the start of 2023, spot ratios have spiked as much as 55% (in us-east-1)”; “Between October 2022 and today, we’ve provisioned 5.5 million spot instances on AWS across all regions”; “We see a near quadrupling in preemption rates in the span of just a few months” (accessed October 18, 2026).↩︎

5. Cristian Măgherușan-Stanciu, “My thoughts on the current state of EC2 Spot pricing,” LeanerCloud newsletter, May 10, 2023: post — “I left AWS not so long ago so I know quite a lot about the Spot internals, capacity figures, and lots of things that I can’t share”; “Their way to spread out the load across their many instance types is by increasing the hourly price, in addition to the inherent increase in interruptions” (accessed October 18, 2026).↩︎

6. Jeff Barr, “Amazon EC2 Update – Streamlined Access to Spot Capacity, Smooth Price Changes, Instance Hibernation,” AWS News Blog, November 28, 2017: post — “moving to a model where prices adjust more gradually, based on longer-term trends in supply and demand”; “Unlike the old model which required an understanding of Spot markets, bidding, and calls to a standalone asynchronous API, the new model is synchronous and as easy to use as On-Demand” (accessed October 18, 2026).↩︎

7. AWS Compute Blog, “New Amazon EC2 Spot pricing model: Simplified purchasing without bidding and fewer interruptions,” January 2018: post — “Previously, Spot Instances were terminated in ascending order of bids, and the Spot price was set to the highest unfulfilled bid. The market prices fluctuated frequently because of this. In the new model, the Spot prices are more predictable, updated less frequently, and are determined by supply and demand for Amazon EC2 spare capacity, not bid prices”; “predictable prices that adjust slowly over days and weeks, with typical savings of 70-90%”; “typical frequency of interruption for Spot Instances in the last 30 days was less than 5% on average” (accessed October 18, 2026).↩︎

8. Jeff Barr, “Focusing on Spot Instances – Let’s Talk About Best Practices,” AWS News Blog, 2015: post — “Check the Price History – You can access historical prices on a per-pool basis going back 90 days”; “Seek out pools that have lower prices and more stable prices … to find bargains and lower interruption rates” (accessed October 18, 2026).↩︎

9. AWS Compute Blog, “Introducing the capacity-optimized allocation strategy for Amazon EC2 Spot Instances,” 2019: post — “Spot pricing changes slowly over time based on long-term trends in supply and demand, but capacity fluctuates in real time”; “The capacity-optimized strategy does account for pool capacity depth as it deploys Spot Instances, but it does not account for Spot prices” (accessed October 18, 2026).↩︎

10. Jeff Barr, “New – EC2 Spot Blocks for Defined-Duration Workloads,” AWS News Blog, 2015, with July 2021 update: post — “Spot Instances with a defined duration (also known as Spot blocks) are no longer available to new customers as of July 1, 2021. For customers that have previously used the feature, we will continue to support Spot Instances with a defined duration until December 31, 2022” (accessed October 18, 2026).↩︎

11. Google Cloud Blog, “Introducing Preemptible VMs, a new class of compute available at 70% off standard pricing,” 2015: post — “Preemptible VM pricing is fixed. You will always get low cost and financial predictability”; “they’ll receive a 30 second notice”; and “Run your fault-tolerant workloads cost-effectively with Google Cloud Spot VMs,” October 29, 2021: post — “Preemptibles will follow the same pricing as Spot VMs”; “Customers who wish to have no max duration, should switch to Spot VMs to avoid any limits” (accessed October 18, 2026).↩︎

12. Microsoft Learn, “About Azure Spot Virtual Machines”: docs — “the Azure infrastructure will evict Azure Spot Virtual Machines with 30-seconds notice”; “Eviction rates are quoted per hour. For example, an eviction rate of 10% means a VM has a 10% chance of being evicted within the next hour, based on historical eviction data of the last 7 days” (accessed October 18, 2026).↩︎

13. Amazon Web Services, Amazon EC2 Capacity Blocks for ML pricing: pricing page — “Reservation prices are updated regularly based on trends in supply and demand for EC2 Capacity Blocks”; launched at re:Invent 2023 (accessed October 18, 2026).↩︎

14. Amazon Web Services, EC2 User Guide: Spot Instance interruptions — “if you specify a maximum price, your instances will be interrupted more frequently than if you do not specify it”; Spot Instances — “Spot Instances are not covered by Savings Plans”; Spot Instance interruption notices — “a warning that is issued two minutes before Amazon EC2 stops or terminates your Spot Instance”; “Interruption notices are emitted on a best effort basis”; “EC2 instance rebalance recommendation – Amazon EC2 emits an instance rebalance recommendation signal to notify you that a Spot Instance is at an elevated risk of interruption … an opportunity to proactively rebalance your workloads without … the two-minute Spot Instance interruption notice” (accessed October 18, 2026).↩︎↩︎↩︎