Caching

A product page needs the product, its price, stock, reviews and recommendations — five queries, perhaps 60 milliseconds of database time. Ten thousand users look at the same popular products every minute. Computing the same answer ten thousand times is waste. A cache stores the answer after the first time and serves copies until they are no longer valid.

Hits, misses and why they matter

When requested data is in the cache, it is a hit — served from memory, often in about a millisecond. When it isn’t, it’s a miss: the system does the slow work and usually stores the result. The hit rate — hits divided by all requests — decides almost everything:

• Latency. At a 90% hit rate with 1 ms hits and 21 ms misses, the average is 3 ms. At 99%, it’s about 1.2 ms.
• Load. At 99%, the database sees only one request in a hundred. A cache often lets a modest database support very large traffic.

Cache-aside, step by step

The most common pattern is cache-aside (lazy loading): the application checks the cache, and on a miss reads the database and fills the cache.

def get_product(product_id):
    key = f"product:{product_id}"
    cached = cache.get(key)
    if cached is not None:
        return cached                          # hit
    product = db.fetch_product(product_id)     # miss: go to the source of truth
    cache.set(key, product, ttl=300)           # keep it for five minutes
    return product

Try it below. The cache holds only four entries and evicts the least recently used (LRU) one when full. Request keys by hand, then switch to realistic traffic, where a few keys are much more popular than others:

With realistic, skewed traffic the hit rate climbs well above what a four-slot cache might suggest. Most real workloads look like this: a small fraction of items — top products, active users, trending posts — receives most requests. That is why a cache holding a few percent of the data can serve the majority of reads.

How big should the cache be? Compare uniform traffic with increasingly skewed traffic across 10,000 items:

The curve flattens quickly: going from 1% to 10% of the items helps a lot, but each further doubling of the cache buys fewer extra hits. Size a cache from measured hit rates, not guesses.

Keeping data fresh

A cache is a copy, and copies go stale. Common tools:

TTL (time to live)

Entries expire after a fixed time. Simple, and bounds how stale data can get.

Invalidation on write

When data changes, delete its cache entry so the next read fetches the new value — as with “Update B” above.

Write-through

Writes go to the cache and the database together, so the cache is always current — at the cost of slower writes.

Versioned keys

Include a version in the key (product:42:v7); a change simply creates a new key, and old versions expire on their own.

Choose staleness deliberately per data type: a product description can be minutes old; a bank balance can’t.

Caches at every layer

Caching appears throughout a system: the browser caches static files; a CDN caches pages and images near users; an application cache such as Redis or Memcached holds query results and sessions; the database itself keeps hot pages in memory. Caching as close to the user as possible saves the most work.

Expiry has a dangerous side effect on popular keys. Watch what happens when a hot key’s TTL runs out under heavy traffic:

Caches reduce reads. When the data itself outgrows one database, we need to split it up — the subject of the next lecture.