The Quiet Failure Mode of AI Features: When Your Cached Outputs Go Stale

There is a famous line in software, attributed to the engineer Phil Karlton, that there are only two hard things in computer science: cache invalidation and naming things. It gets quoted as a joke, but the first half is one of the most honest statements in the field. Filling a cache is easy. Knowing the exact moment a cached value has become a lie, and acting on it, is genuinely hard. And the moment you put AI into production, you create caches everywhere, usually without realising it.

Most writing about production AI worries about the model. Is it accurate, is it hallucinating, is it fast enough. Those are the right questions, and they hide a less glamorous one that bites you long after the model is working fine. The instant you store the output of an AI step instead of computing it live, you have created a cache, and Karlton’s hard problem is now yours.

This affects far more features than people expect. If you embed documents for semantic search, those vectors are a cache. If you generate a summary of a record and save it, that summary is a cache. Auto-tagging, sentiment scores, extracted entities, classification labels, all of it is a derived value that was correct for one version of the source and silently becomes wrong the moment the source changes underneath it.

The bug that taught me this

I run semantic search over a few hundred records. Each record carries a cached vector, computed from its text, so that search does not have to re-embed everything on every query. To keep those vectors honest, I had a hook that fired whenever a record was saved. If the searchable text had changed, recompute the vector. Clean, automatic, and I stopped thinking about it.

Then I renamed a category. That rename touched around sixty records in a single bulk update, the kind of operation you run for efficiency. One query, sixty rows changed, done. Search quality on those sixty records quietly degraded, and it took me a while to work out why.

The bulk update had bypassed the save hook entirely. The hook fires on individual document saves. A bulk update writes straight to the database without ever loading and saving each document, so it skips every hook attached to the save lifecycle. Sixty records now had text that no longer matched their cached vectors, and nothing in the system knew. Worse, no future save would fix them, because the only thing that would have triggered a recompute was the very save path the bulk update had stepped around. Those vectors were stale forever, by design, and the design was mine.

The dangerous part was not that the cache went stale. Caches go stale. The dangerous part was that the one mechanism meant to refresh it could not see the change that made it stale.

One refresh mechanism is never enough

The instinct after a bug like this is to find the one correct place to put the refresh logic. There is no such place. Hooks miss bulk writes. A queue can drop a job. A server can restart in the half second between changing the text and computing the new vector. Any single mechanism has a gap, and the gap is where your data rots. The pattern that holds up is layered. You accept that each layer has holes, and you arrange them so the holes do not line up. I settled on three.

Layer one, mark on change. When a record’s searchable text changes through the normal save path, null out its cached vector and flag it for recompute. This handles the common case the instant it happens.

// in the save lifecycle
if (searchableFieldChanged) {
  this.embedding = null;            // the cache is now known-invalid
  this.scheduleRecompute = true;
}

Layer two, recompute in the background. Do not make the user wait for an AI call to finish their save. After the save commits, schedule the embedding to compute a moment later, and swallow its errors so a flaky model call can never break an unrelated save.

// after save commits, do not block the response
setImmediate(() => embedInBackground(this.id).catch(() => {}));

Layer three, the reconciliation sweep. This is the layer that saves you. On a schedule, scan for any record whose cached value is missing or marked invalid, and recompute it. This is the net under the trapeze. It catches the bulk update that skipped the hook, the job that got dropped, the restart that happened at the wrong moment. It does not care how the cache became stale. It only cares that it is stale, and it fixes it.

// runs on a timer, for example once an hour
const stale = await Record.find({ embedding: null });
for (const r of stale) await embedInBackground(r.id);

Layer three would have caught my sixty records within the hour. The reason the bug persisted is that I had built layers one and two, felt clever, and skipped the boring safety net. The boring safety net is the one that matters.

Do not recompute what has not changed

There is a cost trap on the other side. If your refresh logic is too eager, you will re-run the AI step on every save, including saves that did not touch the relevant text, and if your model is a paid API that is money set on fire. The fix is a content hash. Store a hash of the exact text the cached value was derived from. Before recomputing, compare hashes. If they match, the cached value is still valid and you skip the work entirely.

const currentHash = hash(record.searchableText);
if (currentHash === record.embeddingHash) return;   // nothing changed, skip
// otherwise recompute, then store both the new value and the new hash

This turns refresh whenever something might have changed into refresh only when the input genuinely changed, which is the difference between a feature that is cheap to run and one that quietly drains a budget.

How to know your cache is rotting

A reconciliation sweep fixes staleness, but you also want to know when staleness is happening, because a rising tide of stale records is usually a symptom of something else going wrong. The sweep gives you a free metric: the number of records it finds needing recompute on each run. In normal operation that number is small and steady. If it starts climbing, something is generating invalidations faster than they are being cleared, perhaps a bulk operation running often, or the background recompute failing silently. Track that number, and alert if the backlog grows beyond a threshold. The sweep stops being just a repair mechanism and becomes your early warning that the refresh path is unhealthy.

Track the lag too. For each record you can compare when its source text last changed against when its cached value was last computed. If that gap is growing, your caches are drifting behind reality even when the counts look fine. A stale cache that you can see is a manageable problem. A stale cache you cannot see is the one that quietly degrades a feature for months before anyone connects the complaints to the cause.

The same trap, beyond embeddings

Embeddings are the example I hit, but the pattern is everywhere AI output gets stored. If you generate and save a summary of a document, the summary is stale the moment the document is edited, and unless something recomputes it, users read a summary of a version that no longer exists. If you classify support tickets and store the label, a ticket that gets updated may now belong in a different category while still wearing the old one. If you extract entities or tags once and cache them, they freeze at the moment of extraction while the source moves on without them.

Every one of these wants the same three-layer treatment. Invalidate on change, recompute off the critical path, and run a sweep that does not trust the other two. The specifics differ, the discipline does not. The question to ask of any AI feature you ship is simple: this output was computed from some input, so what happens to the output when the input changes? If you do not have a confident answer, you have a stale cache waiting to happen.

The takeaway

Every AI output you persist is a cache, and every cache needs an invalidation story you can defend. Mark values invalid the moment their input changes. Recompute off the critical path so users never wait on a model. Hash the input so you never pay to recompute something that did not change. And above all, run a reconciliation sweep that does not trust any of the other layers to have worked, because the failure that hurts most is the one no other mechanism in your system can see. Phil Karlton was right that cache invalidation is hard. The way you survive it is to stop trusting any single mechanism to get it right, and to build the boring sweep that catches what the clever parts miss.

Sources
The phrase on cache invalidation and naming things is widely attributed to Phil Karlton.