CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 4 of 12 · 2 min read

Exact and semantic caches: safe hits vs risky ones

One kind of cache can only return exactly what was asked before; the other guesses, and sometimes guesses wrong.

Think of a vending machine versus a helpful but occasionally mistaken store clerk. The machine gives you exactly what you pressed for, every time. The clerk might recognise your request as "basically the same thing" as an earlier one. Then the clerk hands you the wrong item.

In plain words: an exact cache only ever returns an answer for a byte-identical repeat request, so it cannot be wrong. A semantic cache returns an answer for a merely similar request, which means it can occasionally hand back the wrong answer.

A worked example. With a repeating query stream, an exact cache gets a 92 percent hit rate — but that number is an artefact of how often the simulated queries happen to repeat, not a general result. A semantic cache is a sharper trade-off. At a similarity threshold of 0.95 it gets 16 percent useful hits and serves a wrong answer only 1 percent of the time. Loosen the threshold to 0.85, and useful hits rise to 84 percent but wrong answers rise to 16 percent. Loosen it to 0.75, and hits reach 100 percent while 66 percent of those "hits" are wrong.

The key lines.

# from code/mini-cost/costlab.py
def semantic_cache_sim(tau, n=20000, seed=5):
    rng = random.Random(seed)
    same = sum(rng.gauss(0.90, 0.05) >= tau for _ in range(n)) / n
    diff = sum(rng.gauss(0.78, 0.07) >= tau for _ in range(n)) / n
    return same, diff

Read it as: same-intent pairs cluster around a high similarity score, and different-intent pairs around a lower one. The threshold is the single line that decides how much overlap between the two gets treated as a hit. No threshold fully separates the two groups. So a looser threshold always buys more hits by accepting more wrong answers.

Exact cache, semantic cache

An exact cache returns a stored answer only for an identical request; a semantic cache returns one for a merely similar request, judged by embedding similarity.

Example: threshold 0.75 above gives 100 percent useful hits, but 66 percent of matches at that looseness are actually wrong.