How to stop a dead Redis cache from flooding your logs with the same error thousands of times

Most posts about circuit breakers start the same way: “your dependency goes down, every request waits for a timeout, your app grinds to a halt.”

I went to write that post. Then I measured it, and it was not true for my own cache. My app did not get slower. Not by a millisecond.

Something else broke instead, and it took me longer to notice because it was quieter.

The setup: two layers of cache

A common pattern. Two caches stacked on top of each other:

• L1 — a plain Map in memory. Instant. Dies when the process restarts.
• L2 — Redis. Slower, but it survives restarts and all your servers share it.

Ask for a key, you check L1 first. If L1 has nothing, you go ask L2.

This is what createLayeredCacheStore does in @developerehsan/api-client. You give it an L1 and an L2, and you get back one cache that uses both.

const cache = createLayeredCacheStore(memoryCache, redisStore)

Now unplug Redis and see what happens.

The surprise: nothing slows down

I made a fake L2 that hangs for five full seconds before failing. Then I asked the cache for a key that L1 does not have — so the code has no choice but to reach for L2:

const slowL2 = {
  get: () => new Promise((_, reject) =>
    setTimeout(() => reject(new Error('timeout')), 5000)
  ),
  // set / delete / clear the same
}

const cache = createLayeredCacheStore(memoryCache, slowL2)

const t0 = performance.now()
const result = cache.get('missing')
console.log(result, performance.now() - t0)

Five second timeout on the backend. Here is what it printed:

undefined 0.046 ms

Not five seconds. 0.046 milliseconds.

The reason is one word in the implementation: void.

void run().then(onL2Success, (error) => onL2Failure(error, key, op))

void here means “start this, do not wait for it.” The cache never awaits Redis. It asks L1, gets nothing, kicks off a background request to L2, and immediately returns undefined to your code. When the L2 answer arrives later, it quietly warms L1 so the next read is a hit.

This is called fire-and-forget, and it is a good design. Your request path never waits on a network call. But it means the usual circuit breaker sales pitch — “stop paying the timeout on every request” — simply does not apply here. There is no timeout to pay. Nobody is waiting.

So what does go wrong?

What actually breaks: the noise

Every one of those background calls still fails. And every failure gets reported to the error hook.

I ran ten cache writes against a completely dead L2, with no breaker configured, and counted how many times the dead backend was called:

const cache = createLayeredCacheStore(memoryCache, deadRedis, {})

for (let i = 0; i < 10; i++) {
  cache.set('k' + i, entry)
}

L2 calls for 10 ops, breaker unconfigured: 10

Ten out of ten. Every single operation goes to the dead backend, fails, and fires your onStoreError handler.

Now scale that up. If that hook writes to your logging service and your app does 500 cache operations a second, a dead Redis means 500 identical error lines every second. Half a million lines in twenty minutes. All saying the same thing.

That is the real damage, and it comes in three parts:

1. Your log bill. Most hosted logging charges by volume. This is the kind of incident that shows up on an invoice.
2. Your real errors get buried. A genuine unrelated bug is now one line in a wall of l2 down. Nobody will find it.
3. Redis never gets a break. It is already struggling, maybe mid-failover, and you keep opening connections to it at full speed. You are making the recovery harder.

Your dashboards look fine. Latency is flat. Meanwhile you are paying to write the same sentence a million times.

The fix: a circuit breaker

A circuit breaker is a small piece of memory that sits in front of the failing thing and remembers that it is broken, so you stop asking.

The name comes from house wiring, and the three states borrow that language:

• Closed — normal. Electricity flows. Calls go through to Redis.
• Open — tripped. Calls do not go through at all. L1 only.
• Half-open — the cooldown has passed, so let a call through and see what happens.

You turn it on with two numbers:

const cache = createLayeredCacheStore(memoryCache, redisStore, {
  circuitBreaker: { failureThreshold: 2, cooldownMs: 10_000 },
  onStoreError: (error, context) => {
    if (context.op === 'circuit-open') alertOncall('L2 cache circuit opened')
  },
})

failureThreshold: 2 means “after 2 failures in a row, stop trying.” cooldownMs: 10_000 means “wait 10 seconds before testing it again.”

The breaker is off unless you pass this option. That is what the ten-out-of-ten test above was showing. No circuitBreaker key means every call keeps hitting the dead backend forever. If you have never set it, you have the loud version.

Once it trips, L2 is not called at all. Not called-and-caught — actually skipped:

if (circuitBlocking()) return

The package’s own test suite proves it. It opens the circuit, then does a write, a read, and a delete, and asserts the call count on the dead backend never moves:

layered.set('b', makeEntry('b'))
layered.get('c')
layered.delete('a')
await Promise.resolve()
expect(l2.calls).toBe(callsAtOpen)   // never went up

Silence. That is the whole point.

Why “in a row” matters

The counter tracks consecutive failures, and one success resets it to zero.

That sounds like a detail. It is the difference between a breaker that helps and one that annoys you.

Imagine Redis is not down, just imperfect — one request in fifty times out. If you counted total failures over the lifetime of the process, you would eventually cross any threshold you set. Not because anything is wrong, but because the app has been running for a week. You would trip the breaker on a perfectly healthy cache.

Counting in a row asks a better question: not “has this ever failed?” but “is it failing right now?”

Gotcha 1: the probe is not one call, it is a burst

Here is the part the docs and the concept diagrams gloss over, and I only found it because I tested it.

Every explanation of half-open says the same thing: after the cooldown, one call is allowed through as a probe. That is the theory. Let me check the practice.

I opened the circuit, waited out the cooldown, then fired five operations in the same tick:

layered.set('a', makeEntry('a'))            // trips the breaker
await vi.advanceTimersByTimeAsync(1100)     // cooldown elapses

const before = l2.calls
layered.set('b', makeEntry('b'))
layered.set('c', makeEntry('c'))
layered.set('d', makeEntry('d'))
layered.get('e')
layered.delete('f')
await vi.advanceTimersByTimeAsync(0)

console.log(l2.calls - before)

Expected 1. Got:

L2 calls during a 5-op burst after cooldown: 5

All five went through.

There is no lock. The gate is just an arithmetic check on the clock:

const circuitBlocking = () => {
  if (!breakerConfig || circuitOpenedAt === undefined) return false
  return Date.now() - circuitOpenedAt < breakerConfig.cooldownMs
}

The moment the cooldown expires, that returns false for everybody who asks. The circuit does not close again until a call fails and re-stamps the clock — and that failure arrives asynchronously, later. Every call that squeezes into the gap gets through.

So the honest description is not “one probe.” It is: the first call after the cooldown is the probe, and anything arriving in the same instant rides along with it.

Does this matter? Depends entirely on your traffic:

• Low traffic — cache operations arrive one at a time. The gap is empty. You genuinely get one probe, and the theory holds.
• High traffic — hundreds of operations per second means the gap is never empty, and every cooldown expiry sends a small burst at a backend that may still be down.

If you are in the second group, the fix is not clever code, it is a bigger cooldownMs. Ten probes per minute is fine. Ten probes a second is you re-creating the problem you installed the breaker to solve.

I am flagging this rather than calling it a bug, because a hard single-flight lock costs real complexity, and for most apps the burst is a handful of calls once per cooldown window. It is just worth knowing which group you are in before you trust the diagram.

Gotcha 2: “circuit closed” can fire when it was never open

The breaker reports itself through the same onStoreError hook, using an op field: circuit-open when it trips, circuit-close when it recovers.

Natural reading: circuit-close means a probe just succeeded and we are back. So you might log it as “cache recovered”, or use it to resolve an alert.

Look at the recovery code:

const onL2Success = () => {
  if (consecutiveFailures > 0 || circuitOpenedAt !== undefined) {
    report(undefined, { op: 'circuit-close' })
  }
  consecutiveFailures = 0
  circuitOpenedAt = undefined
}

consecutiveFailures > 0. Not “the circuit was open” — just “something failed at some point recently.”

So one hiccup that never came close to the threshold, followed by a success, fires circuit-close. I set the threshold to 5, caused exactly one failure, then let the next call succeed:

ops seen (threshold=5, only 1 failure): ["set", "circuit-close"]

There it is. A circuit-close with no circuit-open anywhere in sight.

Practical version: circuit-open is a real event. circuit-close is “a failure was followed by a success.” If you build alerting on this, pair them — only treat a close as recovery if you saw an open first. Otherwise a single blip on a healthy cache will page someone about a recovery from an outage that never happened.

What the breaker does not touch

Two boundaries worth knowing, because “cache is broken” is a scarier sentence than what is actually happening.

L1 keeps working, completely. The breaker only ever guards L2. Reads and writes to memory behave identically whether the circuit is open or closed. During a Redis outage your app still gets cache hits for anything in local memory. You lose the things L2 specifically gave you — surviving restarts, sharing between servers — and nothing else. Degraded, not broken. That is usually a ticket, not a 3am page.

Most of the cache never involved L2 anyway. has, size, invalidate, isStale, and keysMatching are L1-only by design. I called all five with the circuit wide open and counted the L2 calls:

extra L2 calls from has/size/invalidate/isStale/keysMatching: 0

Zero, as expected. Only get, set, delete, and clear ever cross the network, so those are the only four the breaker needs to guard.

Pick your two numbers

There are no correct defaults, but there are bad ones.

failureThreshold too low (like 1) means one dropped packet or one garbage-collection pause on the Redis side puts you in L1-only mode for a whole cooldown window, over something that would have fixed itself on the very next call. Too high and you keep generating the log storm for longer before the breaker steps in. Somewhere around 2 to 5 is a sane starting point: fast enough to catch a real outage, patient enough to shrug off normal network noise.

cooldownMs too short and you are back to Gotcha 1 — probing a struggling backend far more often than it can recover. Too long and you stay degraded well after Redis came back, losing shared cache across your servers for no reason. Base it on how long your backend actually takes to come back. A managed Redis failover can be seconds. A network partition is not.

Start at 2 failures and 10 seconds, then watch how often circuit-open fires. Firing constantly means your threshold is too low. Never firing during an incident you know happened means it is too high.

None of this is about caching

The shape here — count failures in a row, stop calling, test again after a wait — fits anything that can fail slowly: a third-party API, a read replica, an email provider.

And the two traps travel with it. Count failures in a row, not forever, or uptime alone will trip your breaker. And check whether your “single probe” is actually single, or just a probe-shaped hole that a burst of traffic pours through.

The takeaway

Before you reach for a circuit breaker, measure what a dead dependency actually costs you. The answer is not always latency.

If the call is awaited, you pay in slow requests, and the breaker buys you speed. If it is fire-and-forget like this cache, your latency graph stays perfectly flat while you quietly burn your logging budget writing the same error a million times, and hammer a backend that is trying to recover.

Same pattern, same three states, completely different reason to install it. Knowing which one you have is the difference between fixing your problem and fixing the one from the blog post.