These are my notes from hours of reading Designing Data-Intensive Applications, working through its ideas from first principles, and building implementations to see what holds up.
roynotes-preview.mp4
Site: roynotes.vercel.app | All chapters: roynotes.vercel.app/chapters
A fault is one component deviating from its specification. A failure is the system as a whole no longer keeping its promise to the user. Reliability engineering exists entirely to make sure the first never turns into the second, whether the underlying cause is a dead disk, a software bug, or a human typing the wrong config value at 2am.
Load has to be described with a specific parameter (requests per second, read/write ratio, concurrent users), because "how much traffic can this handle" is meaningless without picking what you're actually measuring. And performance should never be reported as a single average. It's a distribution, and the tail matters more than the center: if a page depends on twenty parallel backend calls, it only takes one slow p99 outlier among them to make the whole page feel slow to the user.
Maintainability is the quiet third pillar. A system that's reliable and scalable today but impossible to safely modify six months from now has just moved the failure into the future instead of preventing it.
- 1.1 Chapter Thesis
- 1.2 First Principles
- 1.3 Mechanism
- 1.4 Build It Yourself
- 1.5 Failure Modes
- 1.6 Tradeoffs
- 1.7 Production Perspective
- 1.8 Related Concepts
Every data model, relational, document, or graph, is a bet about which relationships in your data should be cheap to query and which ones you're willing to pay for later. You cannot judge a schema by reading it. You judge it by pricing one specific query against it and seeing what that query actually costs to run.
The relational model defers the cost to query time: any relationship can be joined later, as long as you're willing to index it. The document model pays the cost up front, at write time, by embedding related data directly, which makes reads cheap but duplicates anything that appears in more than one place. The graph model makes deep, irregular traversal the default access pattern, at the cost of losing the uniform, tabular structure that makes aggregate queries fast.
None of these three is simply "better." Each one is optimized for a different shape of query, and picking the wrong one doesn't show up as a design flaw on day one, it shows up months later as a query that should be simple but requires an application-level join the database was never built to make cheap.
- 2.1 Chapter Thesis
- 2.2 First Principles
- 2.3 Mechanism
- 2.4 Build It Yourself
- 2.5 Failure Modes
- 2.6 Tradeoffs
- 2.7 Production Perspective
- 2.8 Related Concepts
A database engine isn't a black box with a query language attached to it. It's one specific data structure, chosen deliberately, and that choice determines exactly which operations end up fast and which end up slow. The fastest possible write is a sequential append to the end of a file. The fastest possible read is a direct lookup by key. No single structure gives you both of those for free at the same time.
A hash index over an append-only log gets you O(1) reads and writes, as long as the entire index fits in memory, which is exactly the constraint that breaks the moment your key space outgrows RAM. B-trees solve this by keeping the index itself on disk in a form that makes lookups a small, bounded number of page reads. LSM-trees solve it differently, by buffering writes in memory and periodically flushing sorted files to disk, trading read complexity for sustained write throughput.
Compaction is the unglamorous mechanism that makes any of this sustainable over time: reclaiming space from an append-only log by rewriting it sequentially, never by mutating it in place.
- 3.1 Chapter Thesis
- 3.2 First Principles
- 3.3 Mechanism
- 3.4 Build It Yourself
- 3.5 Failure Modes
- 3.6 Tradeoffs
- 3.7 Production Perspective
- 3.8 Related Concepts
Data written today will be read by code that doesn't exist yet. Encoding is the contract that makes that work anyway: it defines exactly what a future reader is allowed to assume about a writer it has never met and never will meet, because in any system with more than one process, deploys are staggered and old and new code run side by side for the entire rollout window.
That means a schema has to support both backward compatibility (new code reading old data) and forward compatibility (old code reading new data) at the same time. The mechanism that makes forward compatibility possible is simple once you see it: identify every field by a permanent numeric tag instead of by name or position, so a reader that doesn't recognize a tag can just skip past it safely instead of breaking.
The failure mode that trips people up is treating a new field as required. You cannot force every already-running writer to start supplying a field simultaneously, so anything added during evolution has to be optional by default, or the rollout itself becomes the outage.
- 4.1 Chapter Thesis
- 4.2 First Principles
- 4.3 Mechanism
- 4.4 Build It Yourself
- 4.5 Failure Modes
- 4.6 Tradeoffs
- 4.7 Production Perspective
- 4.8 Related Concepts
Copying data onto multiple machines is how you survive one of them disappearing. But it introduces a problem a single machine never has to think about: the copies can, and eventually will, briefly disagree with each other. Every replication scheme is really a specific, named answer to exactly how much disagreement is tolerable, for how long, and who's the first to notice.
Leader-based replication solves the ordering problem by construction: if only one node ever accepts writes, there's only ever one possible order for those writes, so replicas can't receive conflicting versions of the same key independently. What's left is a pure latency versus durability tradeoff. Synchronous replication means no acknowledged write is ever lost, at the cost of write latency being bound by the slowest follower. Asynchronous replication is fast, but a leader crash before a write replicates loses that write permanently, even though the client was already told it succeeded.
Replication lag isn't an edge case or a bug report. It's the direct, unavoidable consequence of choosing asynchronous replication for its speed, and any system that reads from a follower has to design around it explicitly.
- 5.1 Chapter Thesis
- 5.2 First Principles
- 5.3 Mechanism
- 5.4 Build It Yourself
- 5.5 Failure Modes
- 5.6 Tradeoffs
- 5.7 Production Perspective
- 5.8 Related Concepts
Once a dataset is too large or too hot for one machine, it has to be split, and partitioning is the decision of which key lives on which machine. At its core it's just a hashing algorithm, but the moment machines start joining or leaving the cluster, it becomes an operational problem: how much data has to move, and how disruptive is that movement to the rest of the system.
Naive modulo hashing couples every key's location directly to the current node count, so adding a single machine reshuffles almost the entire keyspace at once. Consistent hashing fixes this by placing both nodes and keys on the same hash ring, so that adding or removing one node only affects the small arc of keys immediately next to it, leaving everything else untouched.
Partitioning distributes keys evenly, but it says nothing about load. A single viral key can still overload the one partition it happens to land on, no matter how uniform the hash function is.
- 6.1 Chapter Thesis
- 6.2 First Principles
- 6.3 Mechanism
- 6.4 Build It Yourself
- 6.5 Failure Modes
- 6.6 Tradeoffs
- 6.7 Production Perspective
- 6.8 Related Concepts
A transaction is a promise: a group of reads and writes will appear to every other observer as if it happened all at once, or not at all. That promise is never free. The isolation level you choose is the dial that trades correctness guarantees directly for concurrency, and most "impossible" production data bugs trace back to that dial being set lower than the application actually assumed it was.
Snapshot isolation, implemented through multi-version concurrency control, gives every transaction a consistent view of the database as of when it started, and lets writes proceed by creating new versions instead of blocking readers. This eliminates dirty reads and lost updates without ever making a reader wait on a writer. What it does not eliminate is write skew: two transactions can each read a perfectly valid snapshot, each make a decision that's individually correct given what they saw, and the combination of both decisions can still silently violate an invariant that neither transaction could see on its own.
That gap between snapshot isolation and true serializability is exactly where a lot of real-world "how did the database let this happen" incidents live.
- 7.1 Chapter Thesis
- 7.2 First Principles
- 7.3 Mechanism
- 7.4 Build It Yourself
- 7.5 Failure Modes
- 7.6 Tradeoffs
- 7.7 Production Perspective
- 7.8 Related Concepts
A single machine either works or it doesn't, and when it fails, it fails all at once and tells you immediately with a crash or a stack trace. A distributed system doesn't offer that courtesy. A remote node can be slow, partially failed, disconnected but still running, or genuinely gone, and from where you're standing, every one of those states looks identical for an unbounded amount of time.
A timeout is not a failure detection mechanism. It's a calibrated guess, and every choice of timeout duration embeds a real tradeoff: too short and normal jitter or a garbage collection pause triggers false failure detections; too long and a node that's actually dead stays "alive" in the system's eyes for longer than it should. Clocks make this worse, because even NTP-synchronized machines can disagree by tens or hundreds of milliseconds, so using wall-clock timestamps to decide which of two writes happened first is really just comparing two independently drifting guesses.
Nothing in this chapter is solvable in the sense of making the ambiguity disappear. It can only be bounded, worked around, or made explicit, which is exactly what every technique in the following chapters is built to do.
- 8.1 Chapter Thesis
- 8.2 First Principles
- 8.3 Mechanism
- 8.4 Build It Yourself
- 8.5 Failure Modes
- 8.6 Tradeoffs
- 8.7 Production Perspective
- 8.8 Related Concepts
Consensus is what a group of machines needs whenever exactly one outcome has to be chosen among several possibilities, such as which node is the leader or what the next entry in a shared log is, and every machine has to end up agreeing on that single outcome despite the network and the clocks lying to all of them. It's provably impossible to solve in full generality on an asynchronous network where nodes can fail, which is precisely why every real consensus protocol makes a specific, named compromise with that impossibility rather than a clever way around it.
Total order broadcast reframes the whole problem usefully: if every node delivers the same sequence of messages in the same order, you can build a replicated state machine trivially, just apply the messages in that order everywhere. Quorums make a related tradeoff explicit. Requiring writes to reach w replicas and reads to check r replicas guarantees at least one overlapping replica whenever w plus r is greater than n, which is what lets a quorum read see the most recent write, but that guarantee only covers the single most recent write, not concurrent ones.
Systems like etcd and ZooKeeper exist specifically so the rest of the ecosystem can outsource this exact problem to one well-tested implementation instead of everyone reinventing a fault-tolerant sequencer badly.
- 9.1 Chapter Thesis
- 9.2 First Principles
- 9.3 Mechanism
- 9.4 Build It Yourself
- 9.5 Failure Modes
- 9.6 Tradeoffs
- 9.7 Production Perspective
- 9.8 Related Concepts
A dataset too large for one machine's memory, or a job that takes too long for one machine's CPU, still needs to be processed completely and correctly. Batch processing solves this by breaking the job into small, independent, re-runnable pieces spread across many machines, and the real design problem isn't achieving parallelism itself, it's making sure the failure of any single piece only costs that piece's work, not the entire job.
That guarantee depends entirely on one constraint: the map function must be a pure function of its input, with no reliance on external mutable state and no side effects beyond its declared output. If that holds, a failed task can simply be re-run somewhere else with confidence it will produce the same result. The expensive part of the whole pipeline is the shuffle, the step that groups every emitted value by key across every mapper, because that's where data actually has to move across the network.
Sorting is what makes joins scale past what fits in memory in the first place. A sort-merge join turns an unbounded-memory problem into a sequential-scan problem by using disk to hold what RAM can't.
- 10.1 Chapter Thesis
- 10.2 First Principles
- 10.3 Mechanism
- 10.4 Build It Yourself
- 10.5 Failure Modes
- 10.6 Tradeoffs
- 10.7 Production Perspective
- 10.8 Related Concepts
Batch processing assumes the input has an end, a file you can eventually finish reading. A stream has no end, only a continuously growing history, and that one difference invalidates almost everything batch processing was allowed to assume. There's no final answer to compute anymore, only a continuously updated one, and "done" has to mean "caught up to right now," which is a moving target by definition.
An event log makes this workable by being an append-only sequence with a durable position: every consumer tracks its own offset into the log, so multiple independent consumers can each replay from wherever they need to without disrupting each other. Delivery semantics turn out to be a spectrum rather than a binary choice. At-most-once can silently drop messages. At-least-once can redeliver the same message after a failure. Exactly-once processing, which is different from exactly-once delivery, is really just at-least-once delivery combined with idempotent processing, so a redelivered message produces the same end state whether it's handled once or five times.
Event time and processing time are also not the same clock, and any windowed computation has to explicitly choose which one it's windowing by, because the two will give different, equally valid answers.
- 11.1 Chapter Thesis
- 11.2 First Principles
- 11.3 Mechanism
- 11.4 Build It Yourself
- 11.5 Failure Modes
- 11.6 Tradeoffs
- 11.7 Production Perspective
- 11.8 Related Concepts
Almost no real application runs on one database. A typical system has a system of record, a search index, a cache, an analytics warehouse, and several denormalized read views, each holding a different shape of the same underlying facts. The actual engineering problem is keeping all of those derived copies in sync with the source of truth, and in practice the dominant failure mode isn't any single store being wrong, it's the copies quietly disagreeing with each other.
Writing to two stores directly from application code cannot be made atomic without a distributed transaction across systems that mostly don't support one, so any crash between the two writes leaves them permanently out of sync with no record that it even happened. Change data capture solves this by treating every write to the system of record as an event on a stream, and letting every derived store subscribe to that stream independently instead of being written to directly. Because that stream can always be replayed from the beginning, any derived view becomes disposable and rebuildable by design, which is what makes this approach fundamentally more resilient than a hand-rolled dual write.
This is also the natural endpoint of the whole book: correctness stops being a property you can check one database at a time, and becomes an end-to-end property of the entire system.
- 12.1 Chapter Thesis
- 12.2 First Principles
- 12.3 Mechanism
- 12.4 Build It Yourself
- 12.5 Failure Modes
- 12.6 Tradeoffs
- 12.7 Production Perspective
- 12.8 Related Concepts
This site, its content, and its code are my own work. I used AI a little, mostly for grammar checks and occasionally to talk through a bug. It didn't write the code or the descriptions.
Designing Data-Intensive Applications and its original content are the work of Martin Kleppmann and the respective copyright holders. These notes are independent commentary and do not reproduce the book.
~ This Side Roy, Signing Of , Bye....