Can't I just normalize the prompt and key on that?

It buys a few more hits and no more than that. Normalizing fixes casing and spacing, but the part that varies is the content, and that's the part that never repeats.

Is semantic caching ever worth it?

Sometimes, for questions where near enough really is enough. Not for anything a client will act on, because a wrong hit is indistinguishable from a right one.

What if the fact changes before the TTL is up?

Then it was the wrong TTL. Shorten it, or add a way to clear that one key when something upstream tells you it moved.

Blog / AI agents

Cache the Thing, Not the Prompt

The reason your automation pays twice for the same answer is the key, not the cache.

Kershey Cariño · Aug 15, 2026 · 4 min read

In short

Caching an AI automation only works when the key names the thing being looked up, not the prompt that asked for it. Free text rarely repeats, so a prompt-keyed cache almost never hits. Split each fuzzy step into the part about a reusable entity and the part about this one request, and cache only the first.

An enrichment step runs on every inbound lead. Look up the company, work out the industry, the rough size, the funding stage. It's the slowest thing in the pipeline and the most expensive.

Then two leads land from the same company in the same week and it does the whole lookup twice. So you add a cache. The bill doesn't move.

the key is the whole design

A cache is only as good as what you key it on, and the instinct is to key on the prompt. That's the one choice that guarantees it never hits.

Prompts to an AI step are mostly free text, and free text rarely repeats byte for byte. Every lead carries a slightly different message, so every key is new, and you've built a store that only ever writes.

the whole prompt      "Enrich acme.com and classify: <their message>"
                      never repeats. hit rate near zero.

a normalized prompt   lowercased, trimmed, punctuation stripped
                      a few more hits. still mostly misses.

semantic similarity   embed the input, accept anything close enough
                      many hits, and some of them wrong.

the entity itself     acme.com -> { industry, size, funding }
                      the one that works.

The best cache key is almost never the prompt. It's the thing the prompt is about.

split the step before you cache it

Most fuzzy steps are two jobs wearing one coat. One is about something reusable, the other is about this exact request, and they need opposite treatment.

enrich("acme.com")      about a COMPANY, repeats on every
                        lead from there              -> CACHE

classify(thisMessage)   about THIS REQUEST, asked once
                        and never again              -> DON'T

Once the step is split, the caching question answers itself. Anything that names a thing the world already knows about is worth keeping. Anything that names this particular moment isn't.

Kershey and a small robot at a prep counter lined with ready-made bowls, shelves of labelled containers behind them, a clock and a refresh icon overhead. — Prep the parts that get used again. The clock on the wall is the part everyone forgets.

The cache is the easy part. Deciding where the step splits is the actual work, and it's the same cut that makes the model's half more reliable anyway.

three different things get called caching

Three separate mechanisms share the word, and they are not interchangeable. Knowing which one you're reaching for is most of the decision.

which one	what it skips	what it saves	what it risks
result cache	the call, entirely	the whole cost and the wait	a stale answer
provider prompt cache	nothing, it discounts the repeated part	some of the input cost	very little
semantic cache	the call, on a similar input	the most, on the most requests	a confidently wrong answer

The result cache is the one you own and the one worth building first. Prompt caching is close to free if your prompts share a long stable opening. Semantic caching is the one to be suspicious of, because a near-enough match returns an answer about a different thing, in the same fluent voice as a correct one.

a TTL is a promise, not a savings knob

Set the expiry from how fast the underlying truth moves, never from how much you'd like to save. The two have nothing to do with each other.

industry, rough size: moves in months. thirty days is fine.
funding stage, headcount: moves in weeks. seven days.
is this person still there: moves any day. keep it short, or don't cache it.
this lead's classification: never repeats. not cacheable at all.

The test that settles it: how long would you be comfortable telling the client this answer is still current? That's your TTL, and it was never a budget decision.

Can't I just normalize the prompt and key on that?: It buys a few more hits and no more than that. Normalizing fixes casing and spacing, but the part that varies is the content, and that's the part that never repeats.
Is semantic caching ever worth it?: Sometimes, for questions where near enough really is enough. Not for anything a client will act on, because a wrong hit is indistinguishable from a right one.
What if the fact changes before the TTL is up?: Then it was the wrong TTL. Shorten it, or add a way to clear that one key when something upstream tells you it moved.

None of this is really about the bill. It's about noticing that half of what you keep asking is a question about the world, already answered and unlikely to change today. Only the other half is about the thing actually in front of you.

Seen in a real build

Want it in a real build? A shortlist that answers in seconds because the expensive part about each site was already looked up, and re-checked on a schedule. See how MARA · Client Shortlisting is built.

Got something to build?

Got an automation paying twice for the same lookup? Cache It Once! — contact Kershey, or See the Case Files.