LLM Best Practices

Embeddings: Semantic Cache

Last updated 3 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

A semantic cache returns a stored LLM answer when a new query’s embedding is close enough to a cached query’s. It fits high-volume, repetitive intent (support bots, FAQs) where a near-match answer is acceptable. Savings equal the measured hit rate times the per-call cost, minus the embedding and vector lookup on every request. For the concept, see semantic-cache. For re-embedding costs, see embeddings-cost-control.

Provider prompt caching is a different tool: it makes re-reading a long prefix cheaper but still generates a fresh answer. See prompt-caching-strategies. Use both when the prefix is long and the questions repeat.

Tune the threshold on labeled pairs, starting strict

The threshold is a cosine similarity cutoff, and its useful value depends on the embedding model, so a number copied from another system is not evidence.

  1. Collect 100 or more real query pairs and label each pair “same answer” or “different answer”.
  2. Score every pair with your model and pick the lowest threshold with no false positives you cannot tolerate.
  3. Start strict, loosen as the labeled false-positive rate allows, and re-run when you change the embedding model.
hit = nearest and nearest.score >= THRESHOLD   # cosine similarity; tuned on labeled pairs

Raise the threshold for legal, medical, and financial answers, or skip the cache where freshness matters more than cost.

Scope every entry

A cache entry is valid only for the context that produced it. Store, and filter on, these fields:

  • the query embedding (normalized; see embeddings-dimensionality),
  • the response text,
  • tenant_id or user scope whenever the answer uses account data,
  • prompt_version, retrieval-corpus version, and the generating model ID (for example claude-sonnet-5-5),
  • created_at and an expiry.

Never serve an entry across tenants when the answer reflects account state.

Use pgvector when Postgres is already in the stack

CREATE TABLE semantic_cache (
  id BIGSERIAL PRIMARY KEY,
  embedding vector(1024) NOT NULL,
  response TEXT NOT NULL,
  tenant_id TEXT NOT NULL,
  prompt_version TEXT NOT NULL,
  created_at TIMESTAMPTZ DEFAULT now(),
  last_hit TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX ON semantic_cache USING hnsw (embedding vector_cosine_ops);
 
SET hnsw.iterative_scan = strict_order;   -- pgvector 0.8+: keep scanning when the filter drops rows
SELECT response, 1 - (embedding <=> $1) AS score
FROM semantic_cache
WHERE tenant_id = $2 AND prompt_version = $3
ORDER BY embedding <=> $1
LIMIT 1;

An approximate index applies the WHERE filter after the scan, so without iterative scans a selective filter can return no row even when a match exists. A vector HNSW index supports up to 2,000 dimensions (see embeddings-dimensionality). chromadb supports the same filtered query for non-Postgres stacks.

Invalidate on every change that alters the answer

TTL alone is not invalidation. Delete or ignore entries when the prompt template, retrieval corpus, or model changes.

  1. Tag entries with prompt_version and filter on it; delete old tags on deploy.
  2. Or namespace by a hash of prompt plus model and let old entries expire.
  3. Bound the table: evict by last_hit or created_at once it passes a row cap.
DELETE FROM semantic_cache WHERE id IN (
  SELECT id FROM semantic_cache ORDER BY last_hit ASC LIMIT 10000);

Track hit rate and false-positive rate

Report hit rate, false-positive rate (cache hits whose answer was wrong, from user feedback or a judge), and net cost saved. A high hit rate with false positives is a correctness bug. See rag-eval for the eval setup.

Backlinks