Cache-Database Consistency in High-Concurrency Systems: A Practical Guide
This article explains why caching introduces consistency risks into high-concurrency systems and how to manage them in practice. It compares four cache read/write strategies (cache aside, write through, write behind, and read through) and their trade-offs, walks through common concurrency failure scenarios with timeline analysis, and presents six engineering mitigations including delayed double-delete, change-log-driven invalidation, retry and compensation, version checking, locking, and a cache-bypass fallback. It also covers hot keys, cache penetration, and cache avalanche, and shows how to turn inconsistency into something observable through reconciliation jobs and metrics. A decision checklist at the end helps you choose the right approach for your workload.
Cache-Database Consistency in High-Concurrency Systems: A Practical Guide
1. Why the Cache Is the Weak Link in Reliability
Putting data into a cache is essentially introducing an extra copy of that data, while the database remains the authoritative source. When the business demands that data read from outside the source of truth stay correct (for example, amounts, inventory, and account balances must never be read as stale), the cache stops being a “performance optimization” and becomes a correctness risk.
There are three common failure modes:
- Readers see stale values: A request hits an old copy that has not yet been invalidated, so users see data that disagrees with the database.
- Reading a stale value right after a write (dirty read): A query that immediately follows a completed update returns the pre-update content, generating support tickets and complaints.
- Lost writes: The update lands only in the cache layer; if the asynchronous flush to the database fails or the process crashes, the change never reaches the authoritative source.
One often-misunderstood point is worth stressing: cache consistency is not solved once and for all by some “correct pattern.” It converges only through a combination of design trade-offs, fallback mechanisms, and observability. Any approach that claims to eliminate inconsistency completely should first be asked where its cost actually lies.
2. Trade-offs of the Four Cache Read/Write Strategies
1. Cache Aside
The application checks the cache first; on a miss, it reads from the database and back-fills the cache. On the write path, it writes the database first, then deletes the cache entry.
This is the most widely used strategy. Its strengths are low implementation cost, the ability to fall back to the database when the cache fails, and minimal intrusion into the database. Its weaknesses are a brief inconsistency window after a write and the possibility that the delete operation fails.
public UserDTO getUser(long userId) {CacheEntry hit = cache.get(userId);if (hit != null) return hit.value();UserRow row = userMapper.selectById(userId);cache.put(userId, new CacheEntry(row.toDto(), row.getVersion()), 30);return row.toDto();}
2. Write Through
The write request is written synchronously to both the cache and the database, with the cache layer responsible for persisting to the database. The benefit is that the cache is always up to date and the read path needs no fallback logic. The cost is higher write latency, and the cache layer becomes a hard dependency on the write path, so it must be able to persist data.
3. Write Behind
Writes land only in the cache, and an asynchronous task then flushes them to the database in batches. This gives the best throughput and suits write-heavy, read-light scenarios (such as counters and like counts). However, a process crash or a backlog in the asynchronous queue can cause data loss or delay, so it is generally used only for businesses that tolerate eventual consistency and short-term loss.
4. Read Through
The read path is handled by the cache component, which reads from the database and back-fills on a miss, so the business code no longer contains “check cache, then check database” branches. This is semantically cleaner but demands a more mature cache component.
Selection conclusion: The vast majority of business systems should treat cache aside as the default. Consider write through or write behind only when write latency becomes the bottleneck and you can accept updating the cache before the database.
3. Common Concurrency Anomalies and Timeline Analysis
Scenario A: Delete the cache first, then update the database
| Time | Thread 1 (read) | Thread 2 (write) |
|---|---|---|
| T1 | Cache miss, reads old value A from the database | — |
| T2 | — | Deletes the cache, updates the database to B |
| T3 | Writes old value A back into the cache | — |
| T4 | All subsequent reads get A | — |
The inconsistency window is roughly “one database read plus the back-fill delay.” It grows significantly during slow database queries or when cache back-fills are rate-limited.
Scenario B: Update the database first, then delete the cache
Before the write transaction commits at T1, the reading thread reads the old value from the database; at T2 the write transaction commits and the cache is deleted; at T3 the reading thread back-fills the old value—leaving the same stale copy. This is less likely than Scenario A, but the slower the read path, the higher the chance of hitting it.
Scenario C: The delete operation itself fails
Network jitter, a cache node failover, or a rate-limit rejection can all make the “delete the cache” step fail silently. Dirty data will then persist until the TTL expires, potentially for hours.
4. Six Engineering Mitigations
1. Delayed Double Delete
On the write path, execute “delete cache → write database → wait for Δ → delete cache again.” This compresses the “reading thread back-fills a stale value” action from Scenario A into the Δ window, greatly reducing the chance that an old copy survives. But Δ cannot cover every anomaly: a suspended reading thread, a GC pause, or cross-datacenter network jitter can all break through the window. Conclusion: delayed double delete reduces probability; it is not a guarantee.
2. Change-Log-Driven Asynchronous Invalidation
Use the change log committed by the database (such as the transaction log) as the trigger, turning “cache invalidation” into a reliable asynchronous pipeline:
write transaction commits → change log → change subscription → message queue → invalidation consumer → evict cache
Compared with manually calling the cache client inside business code, this pipeline offers three advantages: the invalidation action is decoupled from the business transaction, it is naturally ordered, and failures can be replayed.
3. Retry and Compensation for Invalidation Failures
@EventHandlerpublic void onUserChanged(UserChangedEvent e) {for (int attempt = 1; attempt <= 3; attempt++) {try {cache.evict(e.userId());return;} catch (RuntimeException ex) {backoff(attempt);}}retryQueue.submit(e); // enter the compensation queue, replayed by a scheduled taskmetrics.count("cache_evict_failed");}
Companion practices: track the compensation queue backlog as an alerting metric, and keep a fallback TTL so that the system can still self-heal in extreme cases.
4. Version Checking to Prevent Old Values from Overwriting New Ones
Carry the data version in the cache entry and compare before back-filling; write only when the version is not lower than the current value:
CacheEntry cur = cache.get(userId);if (cur == null || row.getVersion() >= cur.version()) {cache.put(userId, new CacheEntry(row.toDto(), row.getVersion()), 30);}
Combined with optimistic locking on the database side (UPDATE ... WHERE version = ?), this also resolves concurrent write overwrites.
5. Locking for Strong-Consistency Scenarios
For fields such as amounts and inventory that cannot tolerate any stale value, you can acquire a short-lived distributed lock on the write path, or use “logical expiration + mutex rebuild” on the read path so that only one thread reads from the database at a time. The cost is a new availability dependency on the lock and hotspot contention, so you must evaluate lock timeouts, lease renewal, and deadlock protection.
6. A Fallback Switch to Disable the Cache
Keep a configuration switch that can turn off the cache at per-endpoint granularity. During consistency complaints or data correction, quickly bypassing the cache is the most practical and often most overlooked capability.
5. Hot Keys, Penetration, and Avalanche: The Other Half of the Problem
- Cache penetration: Many requests query keys that do not exist and all fall through to the database. Solve this with short-TTL null placeholders plus up-front request validation.
- Cache breakdown: A hot key happens to expire and a burst of concurrent requests all hit the database. Use mutex rebuild or logical expiration.
- Cache avalanche: A large batch of keys expires at the same time, or the cache becomes entirely unavailable. The remedy is to add random jitter to TTLs, use multi-level caching, and apply rate limiting and circuit breaking on the database side.
- Large keys and hot keys: A single oversized key sharply increases network and deserialization overhead, while a hot key causes load skew on a single node. Split keys, or use a two-level structure combining a local cache with a centralized cache.
6. Observability: Making Inconsistency Measurable
Without metrics, inconsistency can only be discovered through “user complaints.” Build at least three capabilities:
- Scheduled reconciliation jobs: Sample by primary key and compare cache and database versions, outputting an inconsistency-rate curve.
- Core metrics: cache hit rate, database read QPS and latency, number of invalidation failures, compensation queue length, and reconciliation inconsistency rate.
- Distributed tracing: Tag write requests so that if a subsequent read for the same user returns an old version, you can pinpoint the exact back-fill time in the trace.
7. Decision Checklist
| Scenario characteristics | Recommended approach |
|---|---|
| Read-heavy, write-light, tolerates second-level inconsistency | Cache aside + fallback TTL |
| Must read the new value immediately after a write | Cache aside + short-lived mutex rebuild on the read path, or route reads directly to the database |
| Inconsistency window must be reduced to milliseconds | Change-log-driven async invalidation + version checking |
| Write throughput is the bottleneck and loss is tolerable | Write behind + asynchronous batch flush |
| Strong-consistency fields such as funds and inventory | Keep out of the cache, or use logical expiration + distributed lock |
| Need an emergency capability | Per-endpoint cache switch + reconciliation jobs |
8. Summary
At its core, the cache consistency problem is “using a non-authoritative copy to speed up reads of authoritative data.” So the right engineering posture is not to chase some universal pattern, but to: default to cache aside with a fallback TTL; make invalidation reliable with a change log and a compensation queue; eliminate old-value overwrites with version checking; turn inconsistency from an “occasional failure” into an “observable metric” with reconciliation and metrics; and finally, provide a direct-to-database path for strong-consistency fields.
