LLM Best Practices

ChromaDB: Scale Limits and Migration

Last updated 3 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

Single-node ChromaDB is built for ease of use at moderate scale, and it degrades gradually rather than failing loudly. Plan the exit from measured numbers, not a vector-count folklore threshold. This page gives the sizing formula, the signals to watch, and the migration path.

Size the node from Chroma’s own formula

Chroma’s single-node performance guide gives N = R * 0.245, where R is RAM in GB and N is the maximum collection size in millions of 1024-dimension vectors (with three small metadata fields). Reserve at least 1 GB for the system on top of that, and run with at least 2 GB of RAM. A 16 GB host therefore holds roughly 3.9 million such vectors. Disk should be at least the size of RAM plus overhead.

Other documented limits: queries parallelize up to the number of vCPUs and then queue, so latency rises linearly under load; insert in batches of 50 to 250, with throughput plateauing near 150; Chroma’s tests stayed stable up to about 7 million embeddings.

Watch four signals, and plan the move before the first breach

  • Memory: resident memory approaching the formula’s limit for your dimension.
  • Latency: p99 on filtered queries climbing as concurrency passes your vCPU count.
  • Recall: recall@10 on a golden query set falling as the collection grows. HNSW approximate search degrades without errors.
  • Restart time: a long reload after a crash means you cannot recover quickly.

Record a baseline for all four now, so a regression is visible as a change, not an opinion.

Move to pgvector when you already run Postgres

pgvector lets you join vectors to relational data and reuse your backups and monitoring. Index limits matter: an HNSW index supports vector up to 2,000 dimensions and halfvec up to 4,000, so a 3,072-dimension embedding needs halfvec.

CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE docs_support (
  id text PRIMARY KEY, document text, tenant_id text, created_at bigint,
  embedding vector(1024)
);
CREATE INDEX ON docs_support USING hnsw (embedding vector_cosine_ops);  -- m=16, ef_construction=64 by default
 
SET hnsw.iterative_scan = relaxed_order;  -- keep scanning when a filter removes rows
SELECT id, document, 1 - (embedding <=> $1::vector) AS score
FROM docs_support WHERE tenant_id = 'acme'
ORDER BY embedding <=> $1::vector LIMIT 10;

Approximate indexes filter after the index scan, so a selective WHERE can return fewer than LIMIT rows; iterative scans fix that. See postgres-indexes and pinecone-vs-pgvector.

Choose Qdrant or Chroma Cloud for the other cases

Move to Qdrant when filtered nearest-neighbor search at scale, horizontal scaling, or quantization is the bottleneck; see rag-vector-databases. Stay on the Chroma API with Chroma Cloud when you only need more capacity: chroma copy --from-local --to-cloud moves collections. Check its quotas first; the documented limits include 5,000,000 records per collection and 300 records per write.

Migrate with a parallel write period

  1. Export the collection with paged get calls (below).
  2. Bulk-load into the target and rebuild its index.
  3. Dual-write new documents to both stores.
  4. Run the golden query set against both and compare recall@10.
  5. Move reads when the target matches or beats Chroma, then stop writing to Chroma and archive the directory.
rows, offset = [], 0
while True:
    page = collection.get(limit=1000, offset=offset, include=["documents", "metadatas", "embeddings"])
    if not page["ids"]:
        break
    rows += zip(page["ids"], page["documents"], page["metadatas"], page["embeddings"])
    offset += 1000

Migrate vectors as stored; re-embedding in the same step mixes two failures into one. Stage the migration on a snapshot first (chromadb-persistence), and budget testing time because filter syntax and tuning knobs differ between stores.

Backlinks