EP 3: What is Sharding? How it is different from Partitioning?

When it comes to build applications that handle millions of users, scaling databases efficiently is the most crucial part. As traffic on our application grows, the amount of data stored in our database increases significantly thus our databases need to handle increasing read and write requests without being jammed. If the database is not properly scaled, it can become a bottleneck, slowing down the application and negatively affecting the user experience.

Let’s discover in this blog, how to shard the database in order to overcome these bottlenecks and handle millions of users concurrently.

How Databases Scale over time?

When you start building an application, you typically have a single database running on a small server. This setup works fine initially, but as more users join and data grows, performance issues start showing up like CPU usage spikes, memory runs low, and query times increase.

Let’s understand this through an example,

Let’s say you launch a new application, and your database starts handling 100 writes per second. Everything is running smoothly on a single server with MySQL installed. Your database process listens on port 3306, executes queries, and stores data on local disk.

Vertical Scaling (Scaling Up)

Suddenly, your app goes viral after a famous influencer tweets about it. Your traffic doubles to 200 writes per second, and your server starts slowing down. At this stage, you scale up by upgrading your server which means adding more CPU, RAM, and storage.

Now, your database can handle higher throughput, but there's a limit to how much you can scale up. For example, AWS and other cloud providers have hardware limits which means you can’t keep increasing server capacity indefinitely.

Read Replicas for Scaling Reads

While writes are increasing, reads also increase as users fetch profiles, view content, and perform complex queries. Instead of overloading the main database, you introduce read replicas. These are identical copies of the main database that only serve read requests.

In this setup:

• Writes go to the main database (primary/master)

• Reads are handled by replicas (followers/slaves)

This is called leader-follower architecture (or master-slave replication).

Horizontal Scaling (Scaling Out) with Sharding

Now, imagine your app becomes even more popular, and traffic spikes to 1,500 writes per second. You check AWS and realize you cannot upgrade the server anymore beyond 1200 writes since this is the limit of vertical scaling.

So, you switch to horizontal scaling (scaling out) by sharding the database.

What is Database Sharding?

Database sharding is the process of dividing a large dataset into multiple smaller datasets (shards), which are stored on separate servers (nodes).

Each shard contains unique rows of data, and all shards share the same schema (table structure). Instead of one large, overloaded database, you now have multiple smaller databases, working together to handle more users and data efficiently.

Each database node (physical shard) can store one or more logical shards, and an application will use a shard key to determine which shard should store or retrieve a particular piece of data.

Sharding follows a shared-nothing architecture, meaning each shard operates independently and is unaware of other shards.

Let’s also try to understand with an example

Let's say you run a food delivery app like Uber Eats. You store millions of orders in a single database.

Without sharding, your database looks like this:

If millions of users place orders, queries will slow down.

Sharding Solution: Instead of storing all orders in one big table, we split the database based on a sharding key (e.g., City).

• Shard 1 (New York Orders)

• Shard 2 (Los Angeles Orders)

• Shard 3 (Miami Orders)

Now, when a user in NYC searches for an order, the system only looks at "Shard 1", not the entire database.

How does Database Sharding Work?

A traditional (unsharded) database stores all data in a single system, which means every query must scan a large number of rows.

With sharding, the dataset is split into logical partitions (shards), which are stored across multiple physical servers (nodes). Each shard handles a portion of the data independently, reducing query load and improving performance.

To determine which shard to store or retrieve data from, software developers use a shard key, a specific column in the dataset that determines how data is distributed.

Before going into depth of sharding, let me clear one thing for you, don’t get confused between sharding and partitioning!

What is Database Replication?

This involves creating multiple copies also called replicas of a database and then distributing them across multiple servers.

• It ensures high availability since if one of the databases go down, the application can still switch to another replica.

• It also scales the read capacity of the database because now we have multiple databases that serve the data.

Two common replication methods are:

1. Leader - Follower Replication

In this setup we have one database which serves as the leader also called master and we have other databases which act as followers also called slaves.

In this case, write operations are directed to the leader and then it propagates the changes to the follower databases and read operations are distributed across both leader and followers. Thus enhancing the read capacity.

2. Leader - Leader Replication

Here multiple databases act as leaders and each of them can accept both read and write operations.

However in this case we may have conflicts between leader databases and we need to set up conflict resolution mechanisms (Timestamp-bases resolution, Last-write wins, Custom conflict resolution) to make sure we are handling these conflicts properly.

These replications can be Synchronous or Asynchronous replications.

How Sharding and Partitioning are different?

Sharding and partitioning help distribute the data load across multiple servers, preventing any single database from becoming a bottleneck.

1. Partitioning: Splitting data into smaller chunks

2. Sharding: Splitting the database itself into smaller instances

Sharding (Database-Level Scaling)

Sharding means splitting the database across multiple servers (shards). Each shard contains a subset of the data, allowing multiple servers to share the load.

For example:

• Instead of one database handling 1,500 writes per second, you create two shards

• Shard 1 handles 750 writes/sec, and Shard 2 handles the other 750 writes/sec

This ensures that each server stays within its capacity, preventing slowdowns and crashes.

Partitioning (Data-Level Scaling)

Partitioning is about logically dividing the data into smaller chunks, but all data might still be stored on a single database.

For example:

• You have a 100GB database but want to break it into five smaller partitions (40GB, 10GB, 20GB, 20GB, 10GB)

• The partitions are mutually exclusive (no data overlaps)

• All partitions can still exist in one database, or they can be distributed across multiple shards

Types of Partitioning

Partitioning strategies determine how the data is divided. There are two main types:

1. Horizontal Partitioning (Row-Level)

It operates at document level or we can say at row-level. Data is split row-wise, meaning different rows belong to different partitions. Mainly used when different segments of data need to be processed separately.

Example: Users A–M in Partition 1, Users N–Z in Partition 2

2. Vertical Partitioning (Column-Level)

It operates at table level or we can say at column-level. Data is split column-wise, meaning different attributes of the same record exist in separate partitions. Mainly useful for optimizing query performance and security.

Example: A User Table where:

• Basic details (ID, name, email) go in Partition 1

• Sensitive info (password, payment details) goes in Partition 2

Lets try to understand Sharding and Partitioning via some use-cases

When should you use Sharding & Partitioning?

You should consider sharding and partitioning when:

• You need to handle high write and read throughput

• Your storage requirements exceed a single server’s limit

• You want higher availability, if one shard fails, others keep running

• You need to distribute load evenly to prevent slowdowns

Benefits of Database Sharding

Sharding provides several benefits for applications dealing with large-scale data:

1. Improved Performance & Faster Response Time

By splitting data into smaller chunks, queries only search a portion of the dataset, rather than scanning an entire monolithic database. This leads to lower latency and faster query execution.

2.  Prevents Total Service Outages

Sharding helps distribute the load across multiple servers. If one shard fails, others can still operate. Combined with replication, data can also be recovered from another shard.

3. Efficient Scalability

A single database server has limited storage and computing resources. With sharding, instead of upgrading a single machine (vertical scaling), you can add more shards dynamically (horizontal scaling) without downtime.

There’s a reason 400,000 professionals read this daily.

Join The AI Report, trusted by 400,000+ professionals at Google, Microsoft, and OpenAI. Get daily insights, tools, and strategies to master practical AI skills that drive results.

Sign up now for free and work smarter, not harder.

Types of Database Sharding

There are different ways to divide your database into shards. Let’s break them down with examples.

1. Range-Based Sharding (Dynamic Sharding)

Example: Sharding by Customer Name

Imagine a customer database where names are stored alphabetically:

We split the customers into three shards:

• Shard 1 → Names A to I

• Shard 2 → Names J to S

• Shard 3 → Names T to Z

Pros: Simple & easy to implement.
Cons: Uneven distribution (More names may start with "A-I" than "T-Z").

2. Hash-Based Sharding

Example: Sharding by User ID

A social media app like Twitter assigns each user an ID number. Instead of storing them in order, we apply a hash function to distribute them evenly.

Suppose we have three shards, and the hash function returns:

HASH(User ID) % 3 = Shard Number

Pros: Data is evenly distributed across shards.
Cons: Hard to add new shards (you must recalculate all hashes).

3. Directory-Based Sharding

Example: Sharding by Product Category

An e-commerce store like Amazon stores products in a lookup table:

If a user searches for laptops, the system looks at Shard A (electronics) instead of searching the entire database.

Pros: Flexible & easy to manage.
Cons: If the lookup table crashes, the entire system fails.

4. Geo-Based Sharding

Example: Sharding by Country

A food delivery app like Uber Eats assigns each country its own shard:

Now, a customer in Canada searches orders only in "Canada-Shard", reducing query load.

Pros: Fast queries (data is closer to users).
Cons: Uneven distribution (some countries may have more users than others).

How to Choose a Good Shard Key?

Picking the right shard key is important. A bad shard key can cause hotspots, where one shard gets overloaded while others remain underused.

1. High Cardinality (Many unique values)

Bad Example: Using "Gender" (Male/Female) as a shard key → Only two shards possible!
Good Example: Using "User ID" (millions of unique values) → More balanced shards.

2. Even Data Distribution

Bad Example: Using "Age" for a fitness app → Most users are 30-45, causing imbalance.
 Good Example: Using "Randomized User ID", ensuring equal distribution.

3. Avoid Monotonic Change

Bad Example: Using "Total Purchases" in an e-commerce app → Frequent buyers end up in the same shard.
Good Example: Using "Order ID (Random)", so new orders are evenly distributed.

I hope this gives you a clear picture of Sharding, Why it is needed and How it is used.

In a nutshell, Sharding splits a database into smaller, independent shards to improve scalability and performance, while partitioning divides data within a single database for better organization and efficiency.

Keep learning. You’ve got this!