hasina.dev
EN
menu
← All posts

My simple mental model for system design

13 min read

A hand holding System Design Interview, Volume 1, by Alex Xu, on a train

Coding is comfortable! Very comfortable in fact that it's sometimes hard to get over it.

But we need to come to terms with the fact that you will need to grow up at some point. Growing up means taking more responsibilities, learning new things and stretching behind what we feel comfortable with.

Let's face it: Coding, as we know it, is a dying craft. And at some point, you won't care much about what's under the hood because you will be busy dealing with other problems. We are software engineers first, developers second.

One of those problems we will have will be about system designs. AI can automate many things but judgement and accountability should always be human.

There are many books on the topic of System Design. Three of them are System Design Interview Volume 1 & 2 by Alex Xu and Designing Data-Intensive Applications by Martin Kleppmann.

If you prefer courses, you'd only need "Grokking the System Design Interview" by Arslan Ahmad.

There is a free resource for Ankidroid11 I'm a big fan of Ankidroid on github but I'll be honest, I downloaded it years ago. Check on it once or twice and I never got back to it!22 One of the reasons is that I find that, though Ankidroid is a very very good learning tool, its default interface is off-putting. I'm creating my own flashcard with better claude-designed style.

It's seriously a big topic and you can easily burn out trying to learn everything about it. But luckily, you won't necessarily need to learn everything. That's why, I built a simple mental model for me to decide what I should be doing at which scale. And this is what I'm sharing with you here.

The basis

The very first step is always one server and one database. I'm fairly confident that a CCX13 on Hetzner for 40€/month is usually enough for most companies in most use cases, especially if your company is a B2B. That's 2vCPU with 8Gb of RAM and 20TB of traffic. You can have millions of requests a day and it will still hold on.

I worked at a small B2B company in my first years and we never had a capacity issue despite having hundreds of customers. The issue was more on how to grow customers than how to scale the infrastructure.

The classic solution

When the server begins to reach its capacity33 That's the importance of observability tools like Grafana and NewRelic connected to Slack. You may also use Uptime services like UptimeRobot and Uptime Kuma as an added bonus to check on your app every 10 minutes or so. I may address my preferred observability strategy on another blog entry if I have enough material to cover., it's a good time for you to scale. Maybe CCX33 or CCX43 (I'm not sponsored by Hetzner by the way, if you are from the US or Canada, it may not be the best deal for you).

Of course, you won't scale vertically forever. No machine can take that and you may run out of strong enough servers to handle it. And that's your sign to scale horizontally.

Scaling horizontally

When you are close to max out your server. It's a good time to add servers to distribute the load. That's what we call scaling horizontally.

Add a load balancer to distribute the traffic evenly among the servers. And a backup loadbalancer in case the primary one fails which may put down all your system since it's the sole access to the servers.

Beyond the issue about load balancer going down, we can face a couple of issues we wouldn't have in one server.

In our initial case, your reflex would be to put your session your server. If you have multiple servers, that will be problematic. Suppose your login session was created in one server but in your next request, the load balancer routed your request to another server, effectively losing your session. That's why, in an horizontal scale, you should move your session from the servers to a session store like Redis and make your servers stateless. Of course, we're adding some latency there since we moved the sessions but that's a tradeoff we have to take.

Scaling database

limit connection

So far we talked about servers. Databases actually scale a bit differently. First, they have a limited number of connections they can take simultaneously. Postgres defaults to 100. You can raise it, but each connection costs memory, so you can't raise it forever.

If you're to the point where you need multiple servers to serve your app, chances are, you may reach database limit connection really fast.

So, don't let the servers open and close connections to the database by themselves, let them borrow connections through a connection pool which can maintain a set number of connections permanently open to the database. Each request that doesn't get a connection immediately will wait until a slot is free.

But careful: frameworks like Rails or Prisma already keep a pool inside each server. So 10 servers with a pool of 20 connections each can still open 200 connections and blow past the limit. Either size each pool so the total stays under the database limit, or put a shared pooler like PgBouncer between your servers and the database. With a shared pooler, all servers go through the same gate, so the max number of connections to the database is never reached.

index frequently used data

I think my life changed when I understood database indexation years ago. Why make your database suffer going through 8 millions lines if you can index the whole thing and get what you want in milliseconds?

Imagine a table with the list of pets :

| Location | chip_number     | name     | species |
| -------- | --------------- | -------- | ------- |
| row 1    | 250269812345671 | Rex      | dog     |
| row 2    | 250268700012938 | Whiskers | cat     |
| row 3    | 250269900456120 | Biscuit  | dog     |
| row 4    | 250268500998877 | Clover   | rabbit  |
| ...      | ...             | ...      | ...     |

you do

SELECT * FROM pets WHERE chip_number = '250269900456120';

And the whole table gets scanned. Yet if you index it:

CREATE INDEX idx_pets_chip ON pets (chip_number);

Just like that, your database builds a separate list containing only the chip numbers, sorted! And the database would read that when it needs it!

| chip_number (sorted) | → location |
| -------------------- | ---------- |
| 250268500998877      | row 4      |
| 250268700012938      | row 2      |
| 250269812345671      | row 1      |
| 250269900456120      | row 3      |
| ...                  | ...        |

Just make sure to not over-index as it will take disk size and your database backup will suffer. Also, too many indexes makes the RAM works more on write.

Okay moving on!

scaling reads

This is getting advanced in the database territory! You will need to judge depending on the business needs.

You may have seen most tutorials out there about getting a primary database to serve writes and multiple read replicas for the read. The reason is to distribute the load fairly among databases. Those mainly assume business cases like social media or video streaming. Yours might not necessarily that use case. Well, I concede that most use cases have more reads than writes but beware of some use cases!

There's a time gap (called "Replication lag") between the writing in the primary database and the data update to the replicas. Social medias don't really care about that if followers only see your post 5 seconds or 10 seconds late. If you're a finance app, uhm, maybe it's a different story. If you really need the data right after saving, you can specifically route the read to the primary database for this.

In Ruby on Rails we do this:

ActiveRecord::Base.connected_to(role: :writing) do 
	user = User.find(42)
end

In Prisma, you use the extension 'readReplicas' and you can do this:44 I prefer Drizzle over Prisma though, I just put it as it's more popular. In Drizzle, we do this: const [user] = await db.$primary.select().from(users).where(eq(users.id, 42));

const user = await prisma.$primary().user.findUnique({
  where: { id: 42 },
});

Sharding

Well, there is a easy way and a hard way about it, and thank God, I only encountered the easy way.

Sharding the easy way is like in the first company I work in. You have a B2B company. you just have one database per customer. Life is good!

At my mission at L'Equipe, we had multiple databases each for each sports. Same! a scale multiple steps above my first company but still manageable.

Then there's the hard way!

I must admit, here, it's a bit out of my depth since I never had encountered it. I knew about it during a presentation in a tech meetup. Then I read about it. Not something I would be comfortable doing in a production app without having it done multiple times in my own server labs first.

The concept is that, your database become so big that you need to run it on multiple machines.

Sharding is really when you have multiple databases for the same service. Pennylane has this need as they handle a database where some tables handle billions of lines and growing everyday.55 they explained this during an epic meetup: https://www.meetup.com/fr-fr/paris_rb/events/315975122

The question is where do you put each data? How do you divide the database?

The common approach is to take the data id and divide it by the number of databases and the remainder is where it goes.

For example, you have the user_id 43 and 4 databases. you have 43 % 4 = 3 so it goes into database #3.66 if you're extra with your security, chances are your id is some kind of string like uuid and not incremental integer, in which case you need to hash the uuid to obtain a fixed number. It does not really need to be cryptographic so you can just google a function or tell AI to do it for you

There are actually a whole set of problems about sharding, and the most benign of which is that you can't just do SELECT COUNT(*) FROM pets anymore. You need to run that query on every database and add up the results (that's called "scatter-gather").

I can imagine how it is if you need to make a complex query across multiple databases. They actually called that "cross shard queries". And don't forget latency!

Caching

Well! sharding or no sharding, there's one thing that will make it so that you may not need to send much read into it: caching!

It spares your database from multiple, repetitive queries.

Need to know the number of likes, views and followers? cache

Need to know product lists? cache

Need to know Apple's current stock price? cache

And last, queues and background jobs!

You need queuing if the operation the client triggered needs other requests done by external parties or other APIs so that you can get back to the client fast. Your app would put the job in a queue and workers will take the jobs from the queue and execute it.

Take into account to implement retries when the worker fails but also limits of retries to avoid "poison message" that eats your CPU usage. A monitoring alert will make a human check into it.

Claude's comments

Since we live in the age of AI, I asked Claude to fact-check me on these points. Since I'm too lazy to mix my points with his (or hers or I don't know), you can check the fact checks below:

On scaling horizontally: capacity isn't the only reason to add servers. A single server is also a single point of failure, so availability is often the first reason to go from one machine to two.

On stateless servers: the cheap alternative to a session store is "sticky sessions", where the load balancer always sends a user to the same server. It works, but if that server dies, its users lose their sessions, and the load can end up uneven. Sessions are also not the only state to move out: files uploaded to a server's local disk have the same problem. The usual fix is object storage like S3.

On indexes: the main cost of too many indexes is slower writes, because every insert or update has to update every index on the table. They also take disk space and compete for memory with the data you actually want cached. For backups, a logical dump like pg_dump only stores the index definitions, so the dump stays small, but restoring takes longer because every index gets rebuilt. Also, an index on a column with only a few distinct values, like species, rarely helps: the database will often scan the table anyway.

On read replicas: routing a read to the primary right after a write has a name: "read-your-own-writes". Rails can do it for you: with automatic role switching, after a write it keeps sending that user's reads to the primary for a short delay (2 seconds by default). For the Drizzle snippet, db has to be created with withReplicas(...) for $primary to exist.

On the order of things: in practice, caching usually comes before read replicas, and long before sharding. It's the cheapest way to take reads off the database.

On sharding: the "easy way" (one database per customer, or per sport) is still sharding: it's called directory-based or lookup-based sharding, where a lookup tells you which database holds which customer. It's easy because the shard key matches how the data is used, so most requests stay inside one database, and adding a customer or a sport never moves existing data. It still has its own problems: shards can be very uneven (football is much bigger than handball), and anything that spans all of them, like a "latest news, all sports" feed, has to query every database or read from a separate combined store. The "hard way" with id % N has a bigger catch: going from 4 to 5 databases changes the result for almost every row, so almost all the data has to move. That's why people use consistent hashing, range-based sharding or a lookup table instead. Choosing the shard key (customer, user, region...) is usually the most important decision.

On UUIDs: UUIDs mainly stop people from guessing ids, which is useful but not really a security measure. Hashing them for sharding is correct, and fast non-cryptographic hashes like xxHash or MurmurHash are the usual choice.

On caching: every cache needs an answer to "how stale can this be?". That's usually a TTL (time to live): the stock price might be cached for a second, a product list for minutes. When the data changes, you either wait for the TTL to expire or delete the cached entry yourself (cache invalidation, famously one of the two hard things in computer science).

On queues: queues aren't only for calls to external services. Any slow work fits: sending emails, resizing images, generating PDFs. Jobs that fail after the retry limit usually go to a "dead letter queue" where a human can look at them. Two more things go with retries: wait longer between each attempt (exponential backoff), and make jobs idempotent, meaning running the same job twice has the same effect as running it once. With retries, a job will run twice sooner or later.

What do we do moving forward?

There are so many things you can learn on System design but those are already solid foundations.

So what's next?

We don't have much illustrations nor hands on approach here do we?

I'm planning to evolve this blog entry. I think it's best to take a real app we can test out all those concepts together! in the meantime, take care!

Notes

  1. I'm a big fan of Ankidroid

    ↩
  2. One of the reasons is that I find that, though Ankidroid is a very very good learning tool, its default interface is off-putting. I'm creating my own flashcard with better claude-designed style

    ↩
  3. That's the importance of observability tools like Grafana and NewRelic connected to Slack. You may also use Uptime services like UptimeRobot and Uptime Kuma as an added bonus to check on your app every 10 minutes or so. I may address my preferred observability strategy on another blog entry if I have enough material to cover.

    ↩
  4. I prefer Drizzle over Prisma though, I just put it as it's more popular. In Drizzle, we do this: const [user] = await db.$primary.select().from(users).where(eq(users.id, 42));

    ↩
  5. they explained this during an epic meetup: https://www.meetup.com/fr-fr/paris_rb/events/315975122

    ↩
  6. if you're extra with your security, chances are your id is some kind of string like uuid and not incremental integer, in which case you need to hash the uuid to obtain a fixed number. It does not really need to be cryptographic so you can just google a function or tell AI to do it for you

    ↩

Did this post speak to you?