Databases and Data-Bound Technologies

I once designed a transactional schema that looked perfectly reasonable to me. It was normalized, organized, and arranged around the work I understood the application needed to do. Then people wanted reports, which was also a perfectly reasonable request. The business wanted to understand the activity the system supported; it had no particular obligation to remain within the boundaries of my original design.

For a while, the database did both kinds of work. Eventually the database was spending 80 percent of its time updating an index, and by then the downtime and project constraints made changing the arrangement difficult. We started looking at faster storage.

I would like to tell you that I immediately recognized the underlying problem and supplied the obvious answer, but I was involved in the system that had the problem. We were trying to buy our way past it. The eventual response included a separate reporting path, and what stayed with me was how far the work had moved beyond the assumptions that made the original design seem sensible.¹

A user-count forecast would have told us very little about that change. The useful questions were what people wanted to do with the data, how their requests interacted with the transactional work, and what we could change while the system kept running.

This is why I get interested when somebody shows me a database cluster with low average CPU use and free disk space, like the Cassandra cluster in the Introduction. There may be a worthwhile reduction to make, but first I want to know what happens when a node fails, a customer keeps twice as much history, or the team has to rebuild something while serving traffic.

Databases are not stateless web workers. They aren’t designed to run at 80 or 90 percent CPU all the time, because the room above the daily load is where index maintenance, compaction, vacuuming, repair and recovery happen while customers keep working. Some of the biggest databases appear idle most of the time. You don’t run a large Cassandra cluster at 80 percent CPU.

When a database does become a capacity problem, the root cause is almost always one of two misunderstandings: of what the database is for, or of redundancy and multi-region replication. My schema story is the first kind. The second is the assumption that multi-region redundancy happens magically and in an instant. I can’t overstate how often it falls to the capacity team to explain consensus to a team, and to point out that the speed of light is finite. Light in optical fiber covers about 204 kilometers per millisecond, roughly two-thirds of its speed in a vacuum, so a write that has to be confirmed in a second region waits for at least one cross-country round trip before it can be acknowledged. Microsoft’s published median round trip between its East US and West US regions is 67 to 69 milliseconds.² In general, people understand database reliability even less than they understand databases. That misunderstanding shows up later in this chapter as copies, slots and recovery paths that nobody counted.

For me, predicting and managing data capacity is a defining challenge of capacity planning: understanding what a data technology must preserve, rearrange and recover is where the real fun begins. It could be an entirely separate book. Cassandra, Kafka and PostgreSQL each deserve much more space than I can give them, but following a few of their mechanisms will help us ask better questions than “How much storage do we need?” I am not proposing that a capacity team set out to master all three, much less every technology it meets. I want us to work with the specialists, follow their explanations well enough to ask the next question, and arrange qualified review of evidence that will materially affect the plan. Volume II, Chapter 17 returns to how that partnership works when the number is consequential or contested.

How to read this chapter

This chapter is about databases and streaming platforms, where capacity is decided by the choice of database, the data model, retention and query behavior as much as by machine count.

1. Plan the work as well as the space — four questions to carry through any data system.
2. The database a proof of concept outgrows — scaling past the original design, data that belongs in object storage, and the distance between application and database.
3. Cassandra: the data model is part of the capacity plan — partitioning, compaction, repair, and expansion.
4. Kafka: retention is a promise to a future reader — replay, placement, storage growth, and a managed alternative.
5. PostgreSQL: understand the query and the unfinished work — plans, index builds, vacuum, and WAL retention.
6. Recovery is work that has not happened yet — restores, rebuilds, and when capacity has to exist.
7. Build a data-capacity decision record — the artifact that carries the decision, filled in for the Cassandra expansion.

A capacity estimate that omits maintenance and recovery describes only part of the service.

Plan the work as well as the space

We can make the opposite mistake too, adding the maximum requirement of every conceivable operation and presenting the total as a necessary purchase. Before asking for that money, I would want to know which operations can coincide, which can be scheduled apart, and which failure removes our ability to postpone work.

The database a proof of concept outgrows

A familiar path starts with a proof of concept. Ten people use it, then 10,000, on Oracle or Postgres, and the database works fine. Then a million people arrive over a month or two. There are inflection points at about 10 users, 10,000, a million and a billion, and each needs a different approach. Extending a relational database past the range it was designed for usually means throwing high-powered SKUs at it. Sharding is the other option, and it feels like the answer from 15 or 20 years ago, before NoSQL. A system that depends on oddly large VMs is also hard to place, and every move, replacement or expansion has to find that size again.

Switching from a relational database to a NoSQL one takes weeks or months of engineering, and the migration runs old and new systems side by side, the double bubble whose economics Chapter 13 works through. The worst cases postpone that transition until the delay has cost millions of dollars in large SKUs.

So if you’re building a system that has to scale, my judgment is to start with something designed to scale out, such as Cosmos DB or Spanner. That isn’t a claim that relational databases can’t scale; Spanner is relational.³ Postgres is a respectable choice, and AlloyDB is a good one when the workload has to be transactional and relational.

At scale, an organization’s plan comes to depend on 10 or 12 databases at critical scale, each able to move it by tens or hundreds of millions of dollars. Over time they move to more scalable configurations, and the proof of concept that started on MySQL ends as an expensive Cassandra cluster. The conclusion is unavoidable; reaching it often takes years of trial.

Along the way the capacity team becomes a database expert, whether or not it meant to. For many engineers, databases are one of the weakest areas of knowledge: most of us learned schemas, keys and normalization in school, and much of that doesn’t hold at web scale. That is no criticism of whoever chose the proof of concept’s database, which worked when they chose it. But the capacity team is often asked to compensate, with hardware and headroom, for a selection that no longer fits, when the lasting answer is engineering work by the team that owns the system.

Petabytes that don’t belong in a database

For a database measured in petabytes, my first questions are about the data. How often is it queried? Do we need all of it at once? Are we indexing it? Could it live in object storage, such as Google Cloud Storage or Amazon S3, instead? The prices make those questions worth asking. On October 4, 2026, in Google Cloud’s Iowa region, Cloud Storage Standard listed at $0.02 per GiB-month and Spanner’s regional SSD storage at $0.30 per GiB-month, a price that covers Spanner’s three replicas.⁴ A pebibyte is 1,048,576 GiB, so it costs 1,048,576 × $0.02 ≈ $21,000 a month in object storage and 1,048,576 × $0.30 ≈ $315,000 a month in Spanner storage, fifteen times as much before Spanner’s compute. At up to 10 TiB of storage per node, a pebibyte needs at least 103 nodes, roughly another $68,000 a month at the Standard edition’s $0.90 per node-hour. Data that is rarely read can go colder still, with retrieval charges that the access pattern decides. Spanner’s price buys a transactional, replicated, queryable store; the question is whether every byte needs one.

The distance between the application and its database

I have seen many cases where the database stayed in a private data center, because that was thought to be cheaper, while the application moved to a public cloud. A couple of milliseconds between them looks harmless on a network diagram, but an application rarely makes one database call per user action. Take an illustrative page that makes 50 sequential queries: across a 2-millisecond round trip it waits 100 milliseconds, which nobody notices. An object-relational mapper that loads a 1,000-row list one row at a time makes 1,000 calls and waits 2 seconds; a report that does the same for 10,000 rows waits 20. At a quarter of a millisecond per round trip inside one data center, the list would have waited a quarter of a second. Before moving either half, count the round trips per user action, not just the latency of one.

Cassandra: the data model is part of the capacity plan

Imagine a design that groups customer events by customer identifier. Most customers are small, but one begins sending many more events and retaining years of history. The cluster average can remain reassuring while the team has to decide whether to change partitioning, change the access pattern, or establish that the existing placement can carry that customer’s growth. Buying nodes may be part of the response, but nobody has yet explained what they would change for the difficult partition.

There is also work behind the foreground request. Cassandra writes immutable files called SSTables and compacts them into new files, during which the old inputs and the replacement output coexist on disk. A delete is itself a write, a marker called a tombstone, which compaction can drop only after the table’s gc_grace_seconds has passed (ten days by default) and only alongside every file holding the older data it shadows.⁶ A customer planning to delete records has therefore supplied useful demand intelligence, but the first effect on storage is more data, and the fall comes later and unevenly.

Repair ties that grace period to a schedule. It compares replicas and streams the differences, which can consume substantial disk and network I/O, and it has to keep pace with the grace period or a replica that missed a delete can bring the data back. With the ten-day default, the documentation suggests repairing every node at least once every seven days to leave slack.⁷ That is recurring work with a deadline, competing with customer traffic. Before somebody shortens gc_grace_seconds to recover space sooner, I want the specialist to explain what repair has completed, what remains, and whether deleted data could reappear.

Enough space to change the cluster

Take a deliberately simplified six-node cluster in which one copy of retained data occupies 24 TiB after encoding and compression, and three copies are required. The baseline is 24 × 3 = 72 TiB, excluding indexes, logs, snapshots, and temporary work. Starting from the retained physical representation matters here; multiplying application payload bytes by three would not necessarily describe the same quantity.

Now add 2 TiB of retained data per month before replication. Over two months, growth contributes 2 × 2 × 3 = 12 TiB, giving an 84 TiB baseline, or 84 ÷ 6 = 14 TiB per node if placement were perfectly even. We have assumed linear growth and unchanged retention, compression, and behavior, and still need to check how those bytes are actually placed. The existence of room somewhere in the cluster does not establish room on the node that needs it.

Expanding the cluster also requires resources. A joining Cassandra node takes responsibility for token ranges and streams their data from current replicas.⁸ If a seventh node joins at that point, an even share is 84 ÷ 7 = 12 TiB. At an illustrative sustained effective rate of 200 MiB/s, transfer alone takes:

$$\frac{12\times1{,}048{,}576\text{ MiB}}{200\text{ MiB/s}\times3{,}600\text{ s/hour}}\approx17.5\text{ hours}.$$

Finishing that transfer in four hours instead would require about 874 MiB/s: 12 × 1,048,576 ÷ (4 × 3,600). These are invented effective rates, not Cassandra benchmarks, and they leave out preparation, validation and interruptions. What they expose is the relationship between the deadline and the movement we can sustain while the service is busy, which a network interface’s nominal speed leaves unanswered.

Nor does the join free space by itself. Cassandra leaves the transferred data on the original nodes, still counted against their load, until nodetool cleanup runs on each of them, and cleanup is another compaction.⁹ Each original node drops from about 14 TiB to 12 TiB only when that work finishes.

Because expansion consumes resources on the existing nodes, it has to begin while those resources are still available. I would want a trigger based on growth, measured movement time, and usable reserve together. A universal disk-percentage threshold would conceal exactly the conditions that determine whether this particular expansion can finish.

Turn the growth estimate into an experiment

Cassandra’s native table metrics separate live SSTable disk use from total use, which includes obsolete files awaiting removal, and report pending compactions and local latency.¹⁰ The gap between live and total is space compaction has not yet returned, although a count of pending compactions still needs interpretation before it becomes a completion-time estimate.

For our growing customer, the experiment needs to change the data distribution as well as the request rate. I would want the schema, replication and consistency choices, and maintenance state attached to the result, followed by an account of what happened on the busiest replicas during compaction or controlled expansion. A cluster average taken after everything settles does not tell us whether the customer can tolerate the transition.

The project’s cassandra-stress tool runs user-defined profiles against a team’s own tables and queries, although its current documentation marks it deprecated and recommends NoSQLBench instead.¹¹ Either gives the team a way to represent the difficult partition instead of congratulating itself on a default workload that avoids it.

If finishing the expansion in time requires slowing a customer’s work, the customer’s agreement belongs in the plan. Knowing how to carry out the technical procedure does not give us permission to change the service along the way.

Kafka: retention is a promise to a future reader

Kafka’s configuration reference describes retention.ms as “an SLA on how soon consumers must read their data.” The settings that keep that promise deserve as much scrutiny as the stated intention: time and size limits can each trigger deletion, retention.bytes applies per partition, and cleanup removes whole log segments rather than each record exactly on its birthday.¹³ A request for a number of days of history therefore needs to be checked against the actual settings and workload before we turn it into an assured replay window.

For a delete-policy example without tiered storage, assume an aggregate append rate of 10 MiB/s measured in its stored representation, seven days of retention, and three replicas. The simplified retained-log estimate is:

$$10\text{ MiB/s}\times(7\times86{,}400\text{ s})\times3 =18{,}144{,}000\text{ MiB}\approx17.3\text{ TiB}.$$

We divide by 1,048,576 to convert to TiB. The estimate assumes stable traffic and no earlier size-based deletion; it leaves out indexes, segment cleanup delay, operational logs, movement, and reserve. It is a baseline for investigating a disk allocation, not the allocation.

In that same model, retaining fourteen days instead of seven doubles the baseline to about 34.6 TiB once the additional history accumulates. A seemingly modest request to give the team another week to recover has changed capacity without changing the live event rate at all.

Catching up is a different workload

Retention lets a future reader return, but reading the accumulated work still takes time. Consider a separate replay example with 2 TiB left to process. New work arrives at 20 MiB/s, and representative testing establishes effective processing of 60 MiB/s in the same input-byte units. The net catch-up rate is 60 − 20 = 40 MiB/s, giving:

$$\frac{2\times1{,}048{,}576}{40\times3{,}600}\approx14.6\text{ hours}.$$

Using 60 MiB/s as the denominator would make the answer more appealing by forgetting the work that arrives during recovery. If processing falls to the arrival rate, the constant-rate model never clears the backlog; below that rate, the backlog grows. We also need the age of the oldest required record and its remaining retention, because comparing recovery time with a nominal retention duration ignores how much of that duration has already passed.

Adding consumers is not an answer we can assume will work. In a traditional consumer group, each partition has exactly one reader at a time, so extra consumers sit idle and a hot partition stays with one reader. Share groups, production-ready since Kafka 4.2, let several consumers take records from the same partition, but the proposal that introduced them says records can then arrive out of order.¹⁴ That suits independent tasks and not a consumer rebuilding a database from a change log, so the team needs to know which model it is using before claiming any concurrency.

If the consumer is rebuilding a database, our catch-up rate matters only if that database can accept the work while carrying its existing customers. I would want an end-to-end rate, an agreed impact on live service, and a deadline, with someone able to slow competing work if the plan requires it.

Observe the whole delivery path

Kafka’s monitoring reference separates broker measurements, such as the count of under-replicated partitions and request-handler idle time, from client measurements such as records-lag-max, which the consumer publishes rather than the broker.¹⁵ Those distinctions help us follow the work: a producer acknowledgment, a consumer position, and a completed operation in the destination describe different stages. Blending them into one throughput number makes a slow destination look like successful delivery.

A representative experiment should preserve the record-size and key distributions, acknowledgment requirements, replication, and destination behavior. OpenMessaging Benchmark is one possible harness; its workload files set message size, partitions, producer rate and a consumer backlog in gigabytes, which is close to the replay case above.¹⁶

Placement raises the Cassandra question again. Cruise Control models Kafka resources and rebalance goals, including capacity and replica placement,¹⁷ and I would ask the specialist to explain the movement required to reach its proposed arrangement as well as the arrangement itself; the move may be exactly what an already constrained cluster cannot accommodate.

In our portfolio, the web workers’ headroom cannot be spent on the brokers or the destination database, so a staged product launch may need its own schedule for replay or migration.

Azure Event Hubs shows how a managed service changes that boundary. It offers a Kafka-protocol endpoint on its Standard, Premium and Dedicated tiers while Microsoft operates the brokers, so the tier’s limits replace the three-replica disk estimate.¹⁸ Standard keeps events for at most seven days, which rules it out for the fourteen-day request, and on any tier the catch-up rate, like each Kafka feature the application uses, has to be checked against the tier’s limits or tested.

PostgreSQL: understand the query and the unfinished work

PostgreSQL’s query plans let us examine scans, joins, sorts, and other operations. EXPLAIN ANALYZE runs the statement and reports what actually happened, and in PostgreSQL 18 it includes buffer activity by default; planner costs are arbitrary units, conventionally scaled so that one sequential page fetch costs 1.0, not dollars or promised elapsed time.¹⁹ For a consequential query, I would much rather start there than infer its behavior from a machine’s average CPU use.

Suppose an application adds a search across several years of customer history, the kind of request whose rising share could erase the web service’s margin in Chapter 5. Estimating the new demand takes representative parameters, realistic data volumes and distributions, and an explanation of how much work the plan performs. Testing the smallest customer’s history would tell us something, but it would not settle the launch for the largest customer.

Getting the feature ready is work too. A search across years of history is likely to need a new index on a large table. An ordinary CREATE INDEX blocks inserts, updates and deletes until it finishes, which on a very large table can take many hours; CREATE INDEX CONCURRENTLY lets writes continue but scans the table twice, waits for existing transactions, and adds CPU and I/O load while it runs.²⁰ Afterward the index must be maintained under the live workload, which is where my own schema story went wrong. Chapter 5 asked who holds a lock before anyone buys another core; here, the holder may be our own migration.

Find the expensive work before inventing a remedy

Where pg_stat_statements is already configured, it exposes execution counts, aggregate execution time, block activity, and WAL generation grouped by statement identity.²¹ This read-only starting query is written for PostgreSQL 18 (stats_since needs 17 or later), assumes the extension is already loaded, and needs pg_read_all_stats or equivalent rights to show other users’ statements. It does not install or configure anything.

SELECT dbid, userid, queryid, toplevel, calls,
       total_exec_time, mean_exec_time,
       rows, wal_bytes, stats_since,
       left(query, 60) AS query_start
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

The result identifies statements with the most accumulated execution time, which is a different question from which customer experienced the worst delay. If nested statements are tracked, a function call and the statements inside it both appear, distinguished by toplevel; adding their times together counts the same work twice. A cheap statement executed constantly and an expensive one executed rarely can both deserve attention. I would keep the statistics interval and reset history with the result, then investigate representative parameters with the application owner; the reported mean cannot supply a tail-latency percentile.²²

EXPLAIN (ANALYZE, BUFFERS) can then help explain actual behavior for small and large customer histories (the explicit BUFFERS matters only before PostgreSQL 18), but ANALYZE executes the statement, so costly or modifying work belongs in an appropriate controlled environment.²³ The reassuring word explain is not a reason to run something casually against production.

pgbench supports custom transaction scripts and rate-controlled workloads; under rate control, its measurements account for scheduled-start delay and can identify skipped late transactions.²⁴ Those capabilities let a specialist investigate the application’s mix and arrival model rather than rely on a default benchmark whose principal virtue is being easy to repeat.

What I want back is an argument about the mechanism: why this operation becomes expensive under these conditions, what the proposed change addresses, and which obligations have been checked. A list of ten expensive queries starts that investigation; it does not say which change the organization should fund or what the change might disturb.

On Azure Database for PostgreSQL flexible server, Query Store can keep query runtime history and, with wait sampling switched on separately, sampled waits. It is opt-in, Microsoft advises against it on the Burstable tier because of its performance impact, and it keeps seven days by default.²⁵ For the history-search launch, it can show which statements changed and what they waited for, if collection starts early enough that the baseline has not aged out.

A deletion does not settle the storage question

Those mechanisms matter when somebody sets a storage saving against a deletion campaign. Routine vacuuming aims at a steady state rather than a minimum size, and reusable room inside the existing allocation may be exactly what we need if it postpones the next increase. A smaller bill is a different promise. Azure Database for PostgreSQL flexible server can grow provisioned storage but not shrink it; a smaller allocation means a new server and a dump and restore.²⁷ The team can remove obsolete records successfully and keep paying for the same disk.

I would treat unfinished maintenance as work that needs capacity under the expected foreground demand, with evidence showing whether it is keeping up. An automatic maintenance process tells us where to start looking, not whether it can finish under the workload we are planning.

Replication can turn an outage into a space problem

Replication slots keep PostgreSQL’s write-ahead log, or WAL, until their consumers have received it, and the documentation warns that they can retain enough to fill the space allocated for it. Capping retention with max_slot_wal_keep_size, or with PostgreSQL 18’s idle_replication_slot_timeout, protects the primary’s disk at the cost of any consumer that still needed the removed WAL. The consumer need not be a standby: a logical slot feeding change data capture, perhaps into the Kafka cluster above, holds WAL the same way and also keeps vacuum from removing old system-catalog rows, which in the extreme case leads to the wraparound protection described above.²⁸

Take a constructed scenario generating 40 MiB/s of WAL while a slot makes no progress for six hours. Potential additional retention is 40 × (6 × 3,600) = 864,000 MiB, or about 844 GiB, before pre-existing retained WAL and other files. The rate here is measured WAL generation; application payload and net table growth would be different inputs.

Now the capacity decision needs the owner of the lagging consumer. Can it resume, how quickly can it catch up, and what does the tested rebuild path require if the history it needs is gone?

Recovery is work that has not happened yet

Across these technologies, ordinary customer work is visible every day while maintenance and recovery may be postponed, interrupted, or needed only after an event. That makes the latter easy to omit from an average, especially if the event has been absent long enough for everybody to become comfortable with the graph.

Before reducing a cluster, I would want to follow the relevant recovery with the operating specialists. Where is the usable source data? How long do transfer, replay or rebuild, and validation take, and at what point can customer traffic return? Which steps must run while foreground work continues? A successful copy gives us one observation along that path; a restore into an otherwise empty test system may still leave the shared-resource case unresolved.

A managed-service provider can carry much of that path. We still need to know what the purchased service performs, what the customer must prepare, and what evidence supports the recovery commitment. I have no interest in rebuilding a self-managed operations team just to demonstrate that we take recovery seriously, but I do want to know what service we are relying on.

For example, Azure Database for PostgreSQL flexible server automates backups and supports point-in-time recovery within a retention period of seven to thirty-five days. The restore creates a new server with the source’s compute and storage configuration rather than overwriting the original, and Microsoft’s post-restore tasks include redirecting clients, re-creating server parameters and firewall and network rules that are not carried over, and configuring high availability, which the restored server lacks.²⁹ The service performs substantial recovery work, while the application team still needs a tested route back to useful service: validation, connection changes and the period of paying for two servers belong in the recovery plan.

These questions also set the date by which capacity has to exist. The Cassandra expansion, the Kafka replay and the rebuilt replication consumer each need room to finish, and growth can consume that room while the utilization report still looks comfortable. If growth slows, transfer improves, or the event moves, the date may change. I would rather defend a schedule with those dependencies exposed than defend a percentage whose original assumptions nobody can find.

Build a data-capacity decision record

Filled in for the growing Cassandra customer, the record asks for more than a storage ticket would. The obligation is that customer’s history and latency, and the account team’s view of whether the growth continues. The footprint is 84 TiB replicated in two months, with the busiest replicas measured. The work is a seventh node’s 12 TiB join at a measured rate, then cleanup on six nodes, both kept clear of the weekly repairs. The failure to plan for is a replica unavailable during the join, which fails a consistent bootstrap unless the team accepts the documented risk of overriding it. The decision names a start date, the cost of carrying a seventh node from then on, an owner, and the triggers that would move the date: faster growth, a slower transfer, or repairs that stop finishing inside the grace period.

The storage, transfer, replay and WAL arithmetic in the Cassandra, Kafka and PostgreSQL sections is reproducible with python3 examples/data_capacity_math.py; the script does not inspect a cluster or recommend production settings. The portfolio companion likewise leaves its stateful-data-services pool unresolved until service, maintenance and recovery evidence arrives. Adapting either requires measured inputs and dated demand intelligence.

Before removing nodes from the cluster that began this discussion, I want to know whether the proposed configuration can carry the expected work, finish necessary maintenance, and recover within its commitments. We may have to preserve capacity while an assigned investigation resolves an uncertainty, or fund more capacity early enough for the transition to remain feasible. Neither response is especially satisfying to someone who expected low CPU and free disk to settle the matter. But those are the decisions we can explain, and the next chapter follows their economic consequences when keeping the service viable makes its efficiency metric look worse.

---

Notes and sources

1. Timothy O’Brien, “Fail First: The Only Way I Actually Learn New Tech,” Medium, September 29, 2025, republished as “If You’ve Never Broken It, You Don’t Really Know It”, O’Reilly Radar, December 17, 2025. The author’s recollection, retold without identifying details; not evidence that all relational systems face the same limit.↩︎

2. Corning Incorporated, “Corning SMF-28 Ultra Optical Fiber”, product information sheet PI-1424-AEN, which gives an effective group index of refraction of 1.4676 at 1310 nm and 1.4682 at 1550 nm; National Institute of Standards and Technology, “CODATA Value: Speed of Light in Vacuum”, 299,792,458 m/s. Together these give about 204,000 km per second, or 4.9 microseconds per kilometer one way. Microsoft, “Azure Network Round-Trip Latency Statistics”, Microsoft Learn, median round trips for the 30 days ending July 30, 2026. All checked October 4, 2026.↩︎

3. Google Cloud, “Spanner overview” and “AlloyDB overview” (last updated September 30, 2026); Microsoft Learn, “Partitioning and horizontal scaling in Azure Cosmos DB”. Checked October 4, 2026. Vendor descriptions: Spanner as a relational, multi-model database; Cosmos DB scaling a container across physical partitions by partition key; AlloyDB as PostgreSQL-compatible, with one read-write primary instance and read pools. Not a performance comparison; the preference among them is the author’s judgment.↩︎

4. Google Cloud, “Cloud Storage pricing” and “Spanner pricing”, checked October 4, 2026, us-central1 (Iowa) list prices: Cloud Storage Standard $0.02, Nearline $0.01, Coldline $0.004 and Archive $0.0012 per GiB-month; Spanner regional SSD storage $0.30 per GiB-month including replication, and Standard-edition compute $0.90 per node-hour. Google Cloud, “Quotas and limits” for Spanner, last updated October 2, 2026, for 10 TiB of storage per node. List prices before discounts and free tiers, excluding operations, network, retrieval and backup charges; other regions and editions differ. The arithmetic is the author’s.↩︎

5. Apache Cassandra documentation, “RDBMS design” and “Data Definition”, checked October 3, 2026. Query-first modeling and partitions as undivided units of storage; the customer example is hypothetical.↩︎

6. Apache Cassandra 5.0 documentation, “Compaction overview”, checked October 3, 2026. SSTables, tombstones, the 864,000-second default gc_grace_seconds, and when compaction may drop a tombstone.↩︎

7. Apache Cassandra 5.0 documentation, “Repair”, checked October 3, 2026. Resource cost, the reappearance risk, and the seven-day suggestion; follow the deployed version’s guidance.↩︎

8. Apache Cassandra documentation, “Adding, replacing, moving and removing nodes”, checked October 3, 2026. Range streaming, consistent-bootstrap failure when a replica is unavailable, and moved data counting against the old nodes until cleanup, a compaction type in “Compaction overview.” The node count, sizes and rates here are invented.↩︎

9. Apache Cassandra documentation, “Adding, replacing, moving and removing nodes”, checked October 3, 2026. Range streaming, consistent-bootstrap failure when a replica is unavailable, and moved data counting against the old nodes until cleanup, a compaction type in “Compaction overview.” The node count, sizes and rates here are invented.↩︎

10. Apache Cassandra documentation, “Monitoring”, checked October 3, 2026. Live versus total SSTable disk use, pending compactions and latency; use the deployed version’s definitions.↩︎

11. Apache Cassandra documentation, “Cassandra Stress”, checked October 3, 2026: deprecated in favor of NoSQLBench, with user-mode profiles for custom schemas. No benchmark was run.↩︎

12. Apache Kafka 4.3 documentation, “Design”, checked October 3, 2026; the downloads page still listed 4.3.1 (June 25, 2026) as the newest release when rechecked October 4, 2026. Replay of retained records; consumption versus deletion and key compaction.↩︎

13. Apache Kafka 4.3 documentation, “Topic Configs”, checked October 3, 2026. The quoted retention.ms description, per-partition retention.bytes, and segment-based deletion; tiered storage is not modeled, and the figures are not sizing recommendations.↩︎

14. Apache Kafka 4.3 documentation, “Design”; “Apache Kafka 4.2.0 Release Announcement”, February 17, 2026 (share groups production-ready); KIP-932, “Queues for Kafka”, “Ordering”; all checked October 3, 2026.↩︎

15. Apache Kafka 4.3 documentation, “Monitoring”, checked October 3, 2026. UnderReplicatedPartitions, RequestHandlerAvgIdlePercent, and the consumer-published records-lag-max.↩︎

16. OpenMessaging, “The OpenMessaging Benchmark Framework”, checked October 3, 2026. Workload parameters, including consumer backlog size; validate the release and driver. No comparative performance claim is made.↩︎

17. Cruise Control for Apache Kafka, checked October 3, 2026. Resource modeling and rebalance goals; not connected to a cluster for this chapter.↩︎

18. Microsoft, “What is Azure Event Hubs for Apache Kafka?” and “Compare Azure Event Hubs tiers”, checked October 3, 2026. Kafka support by tier, retention limits and per-tier throughput limits.↩︎

19. PostgreSQL 18 documentation, “Using EXPLAIN”, checked October 3, 2026. Arbitrary cost units, ANALYZE executing the statement, and ANALYZE implicitly enabling BUFFERS.↩︎

20. PostgreSQL 18 documentation, “CREATE INDEX” and “Explicit Locking”, checked October 3, 2026. Write blocking, many-hour builds on very large tables, and the concurrent build’s extra scan, waits and load; the capacity interpretation is the author’s.↩︎

21. PostgreSQL 18 documentation, “pg_stat_statements”, checked October 3, 2026. Setup, visibility, toplevel, timing and WAL fields; stats_since arrived in 17. The query, minus that column, ran on a disposable PostgreSQL 14 instance, not on 18 or production.↩︎

22. PostgreSQL 18 documentation, “pg_stat_statements”, checked October 3, 2026. Setup, visibility, toplevel, timing and WAL fields; stats_since arrived in 17. The query, minus that column, ran on a disposable PostgreSQL 14 instance, not on 18 or production.↩︎

23. PostgreSQL 18 documentation, “Using EXPLAIN”, checked October 3, 2026. Arbitrary cost units, ANALYZE executing the statement, and ANALYZE implicitly enabling BUFFERS.↩︎

24. PostgreSQL 18 documentation, “pgbench”, checked October 3, 2026. Custom scripts, target-rate scheduling, latency and skipped transactions.↩︎

25. Microsoft, “Query Store in Azure Database for PostgreSQL flexible server”, checked October 3, 2026. Opt-in capture, separate wait sampling, seven-day default retention and the Burstable-tier warning; no live measurement is claimed.↩︎

26. PostgreSQL 18 documentation, “Routine Vacuuming”, checked October 3, 2026. Reuse versus release, the steady-state goal, VACUUM FULL’s copy and ACCESS EXCLUSIVE lock, and read-only operation near wraparound.↩︎

27. Microsoft, “Limits in Azure Database for PostgreSQL flexible server”, checked October 3, 2026. Storage cannot be reduced without a dump and restore to a new server.↩︎

28. PostgreSQL 18 documentation, “Log-Shipping Standby Servers”, “Replication” and “Logical Decoding Concepts”, checked October 3, 2026. Slots filling pg_wal, max_slot_wal_keep_size (unlimited by default), idle_replication_slot_timeout (new in 18, off by default), and slots retaining WAL and catalog rows.↩︎

29. Microsoft, “Backup and restore in Azure Database for PostgreSQL flexible server” and “Limits in Azure Database for PostgreSQL flexible server”, checked October 3, 2026. 7–35-day retention, restore to a new server of the same size, and settings to re-create; recovery time needs an application-specific test.↩︎