A cache stores a copy of data in a faster access layer so future requests can avoid repeating an expensive database query, remote API call, or computation. Cached data should normally be reconstructable from an authoritative source; the cache improves retrieval, but it should not quietly become the only copy of important business state.

Caching can improve:

• Response latency
• Read throughput
• Database capacity
• External API usage
• Application scalability
• Resilience during temporary dependency degradation

It also introduces new problems:

• Stale data
• Invalidation races
• Memory limits
• Cache stampedes
• Additional infrastructure
• Failure amplification
• Harder debugging

The most important principle is:

Caching is not only a performance decision. It is a consistency policy expressed through data placement.

Start with the latency budget

Do not add a cache only because a database or downstream API appears slow.

First understand:

• Required response time
• Current P50, P95, and P99 latency
• Frequency of repeated reads
• Read-to-write ratio
• Cost of retrieving or computing the value
• Value size
• Acceptable staleness
• Consequence of serving old data
• Expected cache-hit rate

Caching works best when the same result can be reused across multiple requests. When almost every request is unique, the cost of checking and populating a cache may exceed the benefit. The source’s rate of change and the product’s tolerance for eventual consistency should also influence the decision.

A 40-millisecond database query may not need caching when the complete request already meets its latency objective. A two-second remote computation requested thousands of times may be a strong candidate.

Cache the actual bottleneck—not the architecture diagram.

What should be cached?

Good candidates usually include:

• Frequently requested reference data
• Product descriptions
• Public configuration
• Computed summaries
• Search results
• Expensive API responses
• Feature metadata
• Session-independent page fragments
• Data that changes less frequently than it is read

Riskier candidates include:

• Account balances
• Authorization decisions
• Final payment state
• Inventory quantities
• Access permissions
• Frequently changing data
• Highly personalized sensitive information

The question is not merely whether data can be cached.

Ask:

How wrong could the system become if this value remained stale for five seconds, five minutes, or one hour?

Cache-aside

Cache-aside, also called lazy loading, is one of the most widely used caching patterns.

The application explicitly manages the cache.

Read process

1. The application checks the cache.
2. If the value exists, it returns the cached value.
3. If the value is missing, it queries the authoritative source.
4. It stores the retrieved value in the cache.
5. It returns the value to the caller.

Microsoft and AWS describe this as an on-demand approach: the cache contains data only after the application has requested it.

Figure 1: In cache-aside, the application owns cache lookup, source fallback, cache population, and response handling.

Place Figure 1 immediately after the numbered cache-aside read process above.

Conceptually:

Read cache
    |
    +-- Hit --> Return cached value
    |
    +-- Miss
          |
          v
      Read source
          |
          v
      Populate cache
          |
          v
      Return value

Advantages

• Straightforward to introduce
• Cache contains only requested data
• Application controls keys and expiration
• Source remains authoritative
• Cache failure can be handled explicitly
• Works with local and distributed caches

Trade-offs

• The first request after expiry is slower
• Cache logic exists in application code
• Cached values can become stale
• Concurrent misses can overload the source
• Every service may implement caching differently

Cache-aside works well when demand is unpredictable because values are loaded only when requested.

Read-through caching

With read-through caching, the application requests data through a cache abstraction.

On a miss, the cache layer or cache-enabled data-access component loads the value from the source.

Application
    |
    v
Cache abstraction
    |
    +-- Hit --> Return
    |
    +-- Miss --> Load from source

Advantages

• Cache-loading logic is centralized
• Application code is simpler
• Consumers use one consistent data-access interface
• Cache behaviour can change without modifying every caller

Trade-offs

• The cache layer must understand how to load the data
• Cache availability may become part of the dependency’s availability
• Application-specific fallback decisions may be harder
• Debugging can cross several abstraction layers

Inline read-through caches hide cache management behind the data-access API, while side caches such as Redis or Memcached are manipulated explicitly by application code.

Write strategies

Reading through a cache is only half the problem.

The system must also decide what happens when authoritative data changes.

Delete on write

A common cache-aside write process is:

1. Update the authoritative source.
2. Commit the transaction.
3. Delete the corresponding cache key.
4. Let the next read repopulate the cache.

Microsoft and Redis guidance describe this database-first, invalidate-after-write approach for cache-aside.

This is often safer than trying to update two systems with the same value because the next reader rebuilds the cache from authoritative state.

Update on write

The application:

1. Updates the source.
2. Writes the new representation into the cache.

This can improve immediate read performance, but concurrent writers can update the cache in a different order from the source.

Write-through

The write path updates both the backing store and the cache as part of the write operation.

Write-through improves the likelihood that subsequent reads find the current value, but it adds write work and may cache data that is rarely read.

Write-behind

The application writes to the cache first. The cache or a synchronization pipeline updates the backing store asynchronously.

This can:

• Reduce write latency
• Batch writes
• Absorb bursts
• Coalesce repeated updates

But it also means the cache participates in durability. A cache failure before persistence can lose accepted writes. Write-behind therefore requires explicit ordering, retry, recovery, and durability guarantees. Redis documents write-behind as a pattern in which cache changes are propagated to a downstream database asynchronously.

Figure 2: Delete-on-write, update-on-write, write-through, and write-behind make different freshness and durability trade-offs.

The cache-aside write race

Deleting the cache after a database write is simple, but a race is still possible.

Consider:

1. Reader misses the cache.
2. Reader loads the old database value.
3. Writer updates the database.
4. Writer deletes the cache key.
5. Reader writes the old value into the cache.

The cache now contains stale data even though invalidation already ran.

This is a concurrency scenario inferred from the separate cache-miss and invalidate-on-write operations described by the cache-aside pattern. The pattern does not guarantee perfect cache/source consistency.

Possible mitigations include:

• Short TTL
• Store a version with the value
• Compare-and-set updates
• Reject older versions
• Delayed second invalidation
• Serialize updates for critical keys
• Use write-through for stricter freshness
• Avoid caching the value where correctness risk is too high

There is no universal invalidation mechanism that eliminates every race without cost.

TTL and expiration

A time to live defines how long a cached value may remain before it expires.

Example:

product:482
TTL: 10 minutes

The TTL represents a freshness decision.

A very short TTL:

• Reduces the staleness window
• Increases cache misses
• Adds source traffic
• May reduce cache value

A very long TTL:

• Improves hit rate
• Reduces source traffic
• Allows stale values to remain visible longer

Official cache-aside guidance recommends aligning expiration with access patterns and avoiding both premature expiry and excessively long retention.

TTL by data type

These are examples, not universal values. The correct TTL follows the business freshness contract.

TTL jitter

Suppose one million product keys are populated during a deployment with the same 15-minute TTL.

Fifteen minutes later, they may expire together and create a large database spike.

Add randomness:

effective TTL = base TTL ± random jitter

For example:

Base TTL: 15 minutes
Jitter: ±2 minutes

Expiration is now spread across a wider period.

TTL jitter reduces synchronized expiry. It does not solve every stampede because one extremely popular key can still receive many concurrent misses.

Versioned keys

A version can be included in the key:

product:482:v17

When product version 18 is published:

product:482:v18

The old key becomes unreachable through the current version.

Benefits include:

• Avoiding broad key deletion
• Preventing stale updates from replacing newer data
• Supporting representation migrations
• Separating old and new schemas
• Enabling group invalidation through namespace versions

The old value should still expire eventually so unreachable versions do not consume memory indefinitely.

Figure 3: TTL bounds staleness over time, while version-aware caching prevents older generations from replacing newer state.

Cache stampede

A cache stampede occurs when many requests simultaneously miss the same popular key and all query the source.

Hot key expires
      |
      +--> Request 1 queries database
      +--> Request 2 queries database
      +--> Request 3 queries database
      +--> Request 4 queries database

This can turn one expired key into hundreds or thousands of identical source requests.

Cold application instances can create a similar effect: a deployment starts several servers with empty local caches, causing each one to refill from the downstream service. AWS describes cold-cache bursts and request coalescing as important caching concerns.

Request coalescing

Only one request loads the missing value.

Other requests wait for the result.

First request --> Load from source
Other requests --> Wait

This is also called single-flight processing.

Distributed locking

One application instance obtains a short lease and rebuilds the value.

Important safeguards include:

• Short lock timeout
• Unique lock token
• Safe lock release
• Fallback when the lock holder fails
• Avoiding indefinite waits

Stale-while-revalidate

The system serves a slightly stale value while one process refreshes it in the background.

This works well when:

• Availability matters more than perfect freshness
• A stale value remains safe
• Refresh is expensive
• The source needs protection from bursts

Refresh ahead

Popular values are refreshed before hard expiry.

This is useful when:

• Access patterns are predictable
• Misses are expensive
• A small hot set dominates traffic

Avoid refreshing every cached value. Refreshing cold entries wastes source capacity and cache memory.

Figure 4: Request coalescing, stale-while-revalidate, TTL jitter, and early refresh protect the source from synchronized misses.

Local caches

A local cache lives inside one application process.

Examples include:

• In-memory maps
• Java Caffeine
• Guava Cache
• Python process memory
• Application-level memoization

Advantages

• Lowest latency
• No network call
• Easy initial implementation
• No separate cache cluster
• Useful for small, frequently reused objects

Trade-offs

• Every application instance has its own copy
• Values may differ across instances
• Memory competes with the application
• Deployments create cold caches
• Total source traffic may grow with fleet size
• Invalidation across instances is difficult

AWS notes that local caches can develop coherence problems between servers and can create cold-start traffic during deployments.

Distributed caches

A distributed cache runs outside the application and is shared by multiple instances.

Examples include:

• Redis
• Memcached
• Managed distributed caching services

Advantages

• Shared values across instances
• Better global hit rate
• Larger shared capacity
• Reduced per-instance duplication
• Cache survives normal application deployments
• More consistent responses across application nodes

Trade-offs

• Network latency
• Another infrastructure dependency
• Cluster scaling and monitoring
• Connection management
• Failover complexity
• Cache-cluster outages
• Hot-key concentration

Distributed caches reduce some local-cache problems but introduce their own operational and availability characteristics.

Multi-level caching

A system can combine both:

L1: Local in-process cache
      |
      v
L2: Distributed cache
      |
      v
L3: Authoritative database

Read flow:

1. Check the local cache.
2. On miss, check the distributed cache.
3. On another miss, query the source.
4. Populate L2.
5. Populate L1.
6. Return the value.

Benefits:

• Very low latency for hot local values
• Shared distributed cache for wider reuse
• Reduced source load

Costs:

• More invalidation paths
• Different TTLs at different layers
• Possibility of L1 being older than L2
• Harder observability
• More complex failure handling

Figure 5: Local caches minimize latency, distributed caches improve sharing, and the source remains authoritative.

A practical approach is:

• Very short TTL in L1
• Longer TTL in L2
• Event-driven or version-based invalidation
• Source fallback only after both miss

Cache-key design

A cache key is part of the application’s data model.

Include every dimension that changes the result.

Example:

product:v3:482:IN:INR:en

Possible dimensions include:

• Entity ID
• Tenant
• Country
• Currency
• Language
• User segment
• Permission scope
• Feature variant
• Schema version

Missing dimensions cause incorrect reuse.

Example:

product:482

Too many dimensions reduce reuse and lower hit rate.

Do not place secrets or raw sensitive data into visible cache keys.

Value design

Large cached values increase:

• Network transfer
• Serialization time
• Deserialization time
• Memory consumption
• Eviction pressure
• Tail latency

Prefer task-shaped representations.

Instead of caching an entire domain graph, cache the fields needed for the specific query.

Also include a representation version:

{
  "schemaVersion": 3,
  "productId": "482",
  "name": "Example",
  "price": 4250,
  "currency": "INR"
}

If the application encounters an unsupported version, it can discard the entry and reload it safely.

Be careful: invalidating an entire fleet of incompatible cached values during deployment can create a mass refresh and overload the source. AWS highlights format changes and fleet-wide cache refreshes as potential causes of downstream brownouts.

Eviction

A cache has finite memory.

When memory reaches its configured limit, the cache must:

• Remove existing keys
• Reject new writes
• Or apply another configured policy

Redis supports policies including LRU, LFU, random eviction, TTL-based choices, and no-eviction behaviour. The chosen policy should match the workload’s access patterns.

LRU

Least Recently Used removes values that have not been accessed recently.

Useful when recent access predicts future access.

LFU

Least Frequently Used removes values accessed less often.

Useful when a stable hot set dominates traffic.

TTL-based eviction

Values nearest expiry are removed first.

Useful when the application intentionally assigns shorter lifetimes to less valuable entries.

No eviction

New cache writes fail when memory is full.

This may be appropriate when the instance stores authoritative data, but it is often unsuitable for a disposable cache unless the application handles write failures.

Eviction is normal cache behaviour—not necessarily an incident.

The source must remain capable of serving misses.

Negative caching

Negative caching stores the fact that a value was not found.

Example:

customer:9999 --> NOT_FOUND
TTL: 20 seconds

This can prevent repeated expensive lookups for:

• Invalid IDs
• Deleted objects
• Non-existent usernames
• Repeated abusive requests

Use negative caching carefully:

• Keep the TTL short
• Do not cache temporary errors as absence
• Include tenant and permission scope
• Invalidate when the entity is created
• Distinguish NOT_FOUND from DEPENDENCY_TIMEOUT

Caching a timeout as “not found” converts a temporary infrastructure failure into incorrect business data.

Failure handling

A cache must be treated as a fallible dependency.

Cache unavailable

Possible responses include:

• Fall back to the source
• Fall back with strict rate limits
• Serve stale local data
• Fail closed for correctness-sensitive data
• Shed non-essential traffic
• Return a degraded response

Blindly sending every request to the database can turn a cache outage into a database outage.

Cold cache

A restart or flush may remove most values.

Protect the source through:

• Gradual warm-up
• Request coalescing
• Rate limiting
• Preloading only high-value keys
• Stale replicas where safe
• Controlled traffic ramp-up

Poisoned value

A cached value may contain:

• Invalid serialization
• Old schema
• Corrupt data
• Incorrect tenant scope
• Oversized payload

Validate cached values before trusting them.

Hot key

One key may receive a disproportionate share of requests.

Possible approaches include:

• Small local L1 copies
• Read replicas
• Controlled sharding when semantics permit
• Precomputation
• Request coalescing
• Longer TTL
• Stale-while-revalidate

Observability

Caching should be measured by key family and endpoint—not only as one global service.

Track:

• Cache hits
• Cache misses
• Hit rate
• P50, P95, and P99 cache latency
• Miss latency
• Source latency after misses
• Evictions
• Memory usage
• Value size
• Connection failures
• Fallback rate
• Stampede suppression
• Age of served values
• Invalidation delay
• Hot-key distribution
• Database load caused by misses

Redis exposes cache hit and miss counters and eviction metrics that can help evaluate cache effectiveness and policy choice. AWS also recommends monitoring cache utilization, hit rate, CPU, memory, and downstream requests.

Figure 6: Failure strategies protect the source, while observability shows whether the cache is actually useful.

A high global hit rate can still hide:

• One critical endpoint with no reuse
• One tenant creating hot keys
• High miss latency
• Excessive stale responses
• Constant eviction churn
• A fallback path overloading the database

Consistency by data type

Not every value should use the same caching strategy.

Strong correctness

Examples:

• Current balance
• Payment authorization
• Final settlement status
• Permission decision
• Security configuration

Prefer:

• Authoritative reads
• Very short-lived caching
• Version validation
• Fail-closed behaviour where necessary

Bounded staleness

Examples:

• Product description
• Delivery estimate
• Availability summary
• Operational dashboard

Prefer:

• TTL
• Event-driven invalidation
• Version-aware cache entries
• Defined freshness objective

Best effort

Examples:

• Recommendations
• Trend counts
• Popular products
• Non-critical analytics

Prefer:

• Longer TTL
• Stale-while-revalidate
• Graceful degradation
• Approximate values

State the consistency contract in product language.

Instead of:

This value is eventually consistent.

Say:

Product-description changes appear within five minutes.

When not to cache

Avoid caching when:

• Most requests are unique
• The source is already fast enough
• Data changes almost as frequently as it is read
• Stale values create unacceptable risk
• Sensitive data would have weaker controls
• Invalidation is not understood
• The cache would provide little measurable reuse
• The system cannot survive cold-cache conditions

Microsoft specifically warns that cache-aside may not help when most requests miss and may be inappropriate for sensitive or security-related data.

Sometimes a better solution is:

• A database index
• Query optimization
• Read replica
• Materialized view
• Better partitioning
• Precomputed table
• API redesign
• More appropriate data model

A cache can hide a poor query temporarily without fixing its underlying cause.

Key trade-offs

Design review checklist

Before adding a cache, ask:

• What latency or capacity problem does it solve?
• Which source remains authoritative?
• What cache-hit rate is expected?
• How stale may the value become?
• Which caching pattern is used?
• What is the TTL?
• How is invalidation performed?
• How are concurrent writers handled?
• How are stampedes prevented?
• What happens when the cache is unavailable?
• Can the source survive a cold cache?
• How are keys scoped by tenant and permissions?
• How are old value formats detected?
• Which eviction policy matches the workload?
• Which metrics prove that the cache adds value?

Final perspective

A cache is a copy placed closer to the request.

That copy creates performance value only when the system clearly defines:

• What it contains
• Who owns it
• How long it remains valid
• How it is invalidated
• How it fails
• How it is measured
• How the source is protected

The strongest design principle is:

Cache for speed, but derive correctness from the authoritative system.

The mature question is not:

Where can we add Redis?

It is:

Which copy may be stale, for how long, under whose control, and what happens when that copy disappears?

Back to Architecture Lab