EP 5: What is Caching?

In partnership with

Optimize global IT operations with our World at Work Guide

Explore this ready-to-go guide to support your IT operations in 130+ countries. Discover how:

• Standardizing global IT operations enhances efficiency and reduces overhead

• Ensuring compliance with local IT legislation to safeguard your operations

• Integrating Deel IT with EOR, global payroll, and contractor management optimizes your tech stack

Leverage Deel IT to manage your global operations with ease.

Download free guide

Have you ever wondered why some websites or apps seem to load instantly, while others take forever? Or why your computer feels super fast sometimes and slower at other times? A big part of the answer lies in something called caching.

In simple terms, caching is like keeping the things you use most often in a quick-access drawer so you don’t have to keep searching for them. 

Let’s discover what caching is, how it works, and why it’s so important in this blog today.

What is Caching?

Caching involves temporarily storing copies of data in a special storage space, called a cache, which is much faster to access than the original data source (like a server or a database). When you access the same data again, instead of fetching it from the original, slower source, the system retrieves it from the cache, speeding up the entire process.

Common Use Cases for Caching

1. Reducing Network Calls

Imagine you have a client, server, and database, where the client sends requests to the server, and the server retrieves data from the database. Every time the client makes a request, it involves network calls that can slow down the system. By caching the results at the client or server level, the system avoids making unnecessary network calls, leading to faster data retrieval.

For example, if you’re browsing a website and repeatedly request the same data (like your profile or homepage), caching stores that data locally so the server doesn’t have to process the same request multiple times.

2. Speeding Up Computationally Expensive Operations

Another scenario where caching is beneficial is when you're dealing with computationally heavy operations. Imagine a server has to run a complex algorithm or process a large dataset before returning the result. Instead of running the same algorithm every time a request is made, caching the result prevents the server from repeating the expensive operation. This can drastically reduce response times and improve system efficiency.

3. Preventing Data Hotspots

Sometimes, specific data becomes very popular, leading to a "data hotspot".

For example, imagine millions of users trying to access the Instagram profile of a popular celebrity. Without caching, every user request would hit the database, overwhelming it. By storing the profile data in a cache, all requests can be served from the cache instead of querying the database, preventing the database from crashing due to too many requests.

Levels of Caching

Caching can happen at various levels in a system:

1. Client-Side Cache: The client (such as a web browser or app) stores the data so that it doesn't need to request it from the server every time.

2. Server-Side Cache: The server caches data retrieved from the database, preventing repeated requests to the database for the same data.

3. Database Cache: A cache can also sit between the server and the database, minimizing the number of times the server queries the database.

4. Hardware Cache: On a hardware level, modern CPUs use caches (like L1, L2, and L3 caches) to quickly access frequently used data from memory.

The gold standard of business news

Morning Brew is transforming the way working professionals consume business news.

They skip the jargon and lengthy stories, and instead serve up the news impacting your life and career with a hint of wit and humor. This way, you’ll actually enjoy reading the news—and the information sticks.

Best part? Morning Brew’s newsletter is completely free. Sign up in just 10 seconds and if you realize that you prefer long, dense, and boring business news—you can always go back to it.

Join 4.3 Million Readers Now

How Caching Works in Hardware

Let’s first dive into how caching works inside a computer, starting with the hardware level.

1. CPU Caches (L1, L2, L3)

The brain of every computer is the CPU (Central Processing Unit), which handles most of the processing work. To make the CPU faster, it uses caches at three levels: L1, L2, and L3.

• L1 Cache: This is the fastest and smallest cache, located directly on the CPU chip. It stores the most frequently used data and instructions that the CPU needs quickly.

• L2 Cache: Larger than L1 but slower, L2 is usually found on the CPU chip or very close to it. It acts as a secondary cache to store data not found in L1.

• L3 Cache: This cache is the largest and slowest of the three, and it’s shared between multiple CPU cores. It helps store data that’s needed less frequently but still needs faster access than the main memory (RAM).

These different levels of caches help the CPU access data much faster than it would if it had to go all the way to the slower main memory.

2. Translation Lookaside Buffer (TLB)

Another important hardware cache is the Translation Lookaside Buffer (TLB). When the CPU works with virtual memory, it needs to translate virtual memory addresses to physical memory addresses. The TLB stores these translations so that the CPU doesn’t have to look them up every time, speeding up the entire process.

3. Operating System Caches

At the operating system level, there are several caches designed to speed up access to files and memory. The page cache stores frequently accessed disk data in RAM so that when an application needs the data again, the OS can quickly retrieve it without accessing the slower disk.

There are other file system caches as well, like the inode cache, which helps speed up file system operations by reducing the number of disk accesses required to access files and directories.

Caching in Application Systems

Now that we’ve covered how caching works at the hardware level, let's explore how it’s used in everyday applications. Caching plays a significant role in making websites, apps, and services fast and efficient.

1. Web Browsers

When we request data over HTTP for the first time, and it is returned with an expiration policy in the HTTP header. When we request the same data again, the browser returns it from the cache if available.

For example, When you visit a website for the first time, your browser caches parts of the page (like images, scripts, or even the entire page) to speed up future visits. your browser stores these assets locally. The next time you visit the same site, it will load the page faster because your browser retrieves the cached data instead of downloading it again.

2. Content Delivery Networks (CDNs)

For websites and applications that deliver lots of static content (like images, videos, or files), CDNs are used. A CDN is a network of servers distributed around the world. When a user requests content, the CDN checks if it has cached the data. If it does, it delivers the content directly from the cache, making the response time much faster than if it had to pull the data from the original server, sometimes halfway across the globe.

Fact-based news without bias awaits. Make 1440 your choice today.

Overwhelmed by biased news? Cut through the clutter and get straight facts with your daily 1440 digest. From politics to sports, join millions who start their day informed.

Sign up now!

3. Load Balancers

In large systems with multiple servers, a load balancer is responsible for distributing requests evenly across all servers. Some load balancers also cache responses from the back-end servers. This means that when a user requests the same data again, the load balancer can deliver it quickly from its own cache, reducing the load on the back-end servers and speeding up the response time.

4. Message Brokers (e.g., Kafka)

In distributed systems, caching is also used in messaging systems like Kafka. Kafka stores massive amounts of messages on disk, where they can be retrieved at any time by consumers. This “caching” on disk allows consumers to access messages at their own pace without the need to continually query the producer.

5. Distributed Caches (e.g., Redis)

In some applications, caching is done using distributed systems, where data is stored in memory for incredibly fast access. Redis is a popular example of this kind of system. It allows for caching data like user sessions, frequently accessed database results, or any other information that needs to be quickly retrieved.

6. Full-text Search Engines (e.g., Elasticsearch)

Elasticsearch is a search engine that indexes data, making it easy and fast to search through large amounts of documents or logs. Caching plays an important role in Elasticsearch because it helps improve search performance.

When you search for something in Elasticsearch, the system has to go through a lot of data. To make this faster, Elasticsearch caches search results. The first time you run a search, it might take longer, but after that, the results are stored in memory, so if you search the same thing again, it can give you the results much faster from the cache instead of recalculating them.

Caching in Databases

Even in databases, caching is very critical to improve performance.

Write-ahead Log (WAL): Before any data is written to the database, it is first recorded in a write-ahead log, which acts as a form of caching to ensure that no data is lost.

Buffer Pool: Databases use a buffer pool, a section of memory that stores recently accessed data or query results. If the same query is run again, the database can return the result from the buffer pool, reducing the need to execute the query again.

Materialized Views: These are precomputed results of a query stored in the database to speed up future access.

Transaction Log: It logs all the transactions and updates to the database.

Replication Log: It tracks the replication state in a database cluster.

Caching Strategies

1. Cache-Aside

How It Works: The application manages the cache. When data is requested, it first checks the cache. If the data isn't found, the application fetches it from the database and stores it in the cache for future use.

Advantages: Simple and flexible, as it only stores data in the cache when needed.

Challenges: The application must manage the cache, which can get tricky. It also needs to ensure the cache is updated properly when the data in the database changes.

2. Write-Through

How It Works: When data is updated, it is written to both the cache and the database at the same time. This ensures that the cache always contains the most current data.

Advantages: Ensures data consistency between the cache and the database, as both are updated simultaneously.

Challenges: Write operations can be slower because both the database and cache need to be updated at once.

3. Write-Behind

How It Works: Data is first written to the cache, and then it is written to the database later, typically in batches. This allows write operations to happen faster.

Advantages: Faster writes because the database update is delayed, reducing immediate write time.

Challenges: This can lead to data inconsistencies between the cache and the database if the cache is not properly managed. If the system crashes before the data is written to the database, the changes may be lost.

4. Read-Through

How It Works: The cache is used as the main data source. When data is requested, the cache is checked first. If the data is not found, it’s retrieved from the database and added to the cache.

Advantages: Great for situations where data is frequently read but rarely updated, and it can reduce the load on the database, especially if the database is slow.

Challenges: The cache can become stale if not managed correctly, and it may not work as well for data that changes often.

Why is Caching Important?

Caching improves system performance by reducing latency (the time it takes to fetch data). Instead of constantly fetching data from slower sources, such as hard drives or databases, systems can fetch it from a faster cache. This results in faster load times, reduced load on databases and servers, and overall better user experience.

Caching Challenges

However, caching isn’t always a perfect solution. Caching comes with a lot of challenges too, like,

• Cache Invalidation: When data changes, caches need to be updated or invalidated to ensure that outdated information isn’t served.

• Memory Management: Caches use memory, and if not properly managed, they can grow too large and consume resources unnecessarily.

• Consistency: In distributed systems, keeping caches synchronized across multiple servers can be complex.

A lot of people often confuse between Cache Invalidation and Cache Eviction, let’s simplify it for you!

Cache Invalidation

Cache invalidation is when we make sure that old or outdated data in the cache is removed when the main database is updated. This ensures that the cache has the most current and fresh data, so users always see the latest information.

When does it happen?
It happens when the data in the main database is updated, and the cache needs to reflect that change.

Advantages:

• Always gives fresh and accurate data.

• Keeps the cache in sync with the main database.

Disadvantages:

• It can be complicated to manage and set rules for invalidation.

• May cause slight delays if the data is frequently updated.

Cache Eviction

Cache eviction is when older or less frequently used data is removed from the cache to make room for new data. This is done automatically based on certain rules or policies (like removing the least recently used data first).

When does it happen?
It happens when the cache is full, or when certain rules (like Least Recently Used) trigger the removal of old data.

Advantages:

• Helps manage cache size effectively.

• Optimizes memory usage by keeping only the data that is used most frequently.

Disadvantages:

• Important data may be removed if the eviction policy isn’t ideal.

• If important data gets evicted, it might take longer to fetch it from the main database.

I hope this gives you a clear picture of Caching, Why it is needed and How it is used. Will cover Cache Eviction and Invalidation Strategies in the next blog.

In a nutshell, Caching speeds up data retrieval by storing frequently accessed data in faster storage, reducing the need to repeatedly fetch it from slower sources. It's crucial for improving performance but comes with challenges like cache invalidation and eviction to ensure data remains fresh and optimized.

Keep learning. You’ve got this!