When your vector database gets slow

p99 query latency has gone from 40ms to 300ms over three weeks. Nothing was deployed. The corpus grew, but not dramatically. CPU looks normal.

There are maybe six things this is, and they’re distinguishable in a fairly reliable order. Working through them in that order matters, because the expensive investigations are at the bottom and most incidents resolve near the top.

Step 0: read the percentile shape

Before touching anything, look at p50, p95 and p99 over the same window. The shape narrows the search dramatically.

Shape	Usually means
All three rose together	Systemic: memory pressure, segment growth, resource contention
p50 flat, p99 rose	A subset of queries got harder — filters, or a few large result sets
p99 rose then plateaued at a round number	You’re hitting a timeout, not a slowdown
Sawtooth, spiking periodically	Background work: compaction, snapshots, or a rebuild
Step change on a specific date	Something changed. Find the deploy or config edit

That table resolves a good fraction of cases before any investigation. A step change means go read the change log, not the metrics.

Step 1: memory

Check this first every time, because it’s the most common cause and the cheapest to rule out.

The specific thing to look for: is the index still fully resident? Vector search assumes memory-speed random access. Once part of the structure is paging from disk, each query traversal hits storage repeatedly and latency degrades by an order of magnitude rather than a few percent.

Signals: resident memory pinned at the limit, page faults or swap activity, latency degradation that is disproportionate to the growth in data.

This is why “the corpus grew, but not dramatically” is a misleading observation. Growth doesn’t degrade performance smoothly — it’s fine, then it crosses the line where the working set stops fitting, and then it’s a cliff. A 5% corpus increase can be the 5% that doesn’t fit.

Fix: more memory, quantization, or shard. Rollback: all three are configuration changes; snapshot before a quantization change since it requires a rebuild. Prevention is sizing and watching the capacity projection.

Step 2: filtered queries

If p50 is flat and p99 rose, look at filters next.

Combining metadata filters with vector search is genuinely hard, and engines take different approaches: filter first then search the survivors, search first then discard non-matching results, or maintain filter-aware structures. Each has a workload where it degrades badly.

The problematic case is a selective filter over a large collection. Post-filtering means the search returns k results, most get discarded, and it must go back for more — repeatedly. A filter matching 0.1% of documents can turn one traversal into many.

Signals: latency correlates with filter selectivity rather than corpus size; a specific tenant, category or date range is disproportionately slow; the same query without its filter is fast.

Fix: depends on the engine — a filter-aware index configuration, partitioning the collection so the common filter becomes a collection boundary, or pre-filtering into a candidate set. Partitioning by tenant is the usual answer for multi-tenant workloads and solves several other problems at once.

Rollback: if you partitioned, the original collection still exists until you drop it. Don’t drop it for a full cycle.

Step 3: segment and shard count

Engines that write new segments on ingest and compact them in the background can fall behind. Every query fans out across every segment, so the count is a direct multiplier on work.

Signals: segment count trending up over the same window as latency; ingest rate recently increased; compaction lagging or disabled.

Fix: trigger compaction, or fix why it isn’t keeping up — usually it’s competing with ingest for IO or CPU, or it’s throttled by a setting nobody has looked at since install.

Rollback: compaction is generally safe and resumable, but it is IO-heavy. Run it off-peak the first time and watch latency while it runs.

Step 4: tombstones

If the collection has high churn, check the deleted fraction.

Deleted vectors typically stay in the graph, marked, until compaction or a rebuild reclaims them. The traversal still walks them. A collection where 40% of entries are tombstones is doing substantially more work per query than its live count suggests — and recall can suffer too, because the search budget is being spent on dead nodes.

Signals: deleted-as-fraction-of-total is high; memory didn’t drop after a large delete; latency rose gradually alongside churn rather than alongside growth.

Fix: compaction if the engine reclaims that way, otherwise a rebuild. Rollback: snapshot first; a rebuild is covered in reindexing without downtime.

Step 5: search parameters

Only reach here having ruled out the above, because this is the knob people reach for first and it trades away quality.

Graph indexes expose a runtime parameter controlling how much of the graph a query explores — the ef-style search-breadth setting. Higher means better recall and more latency. Cluster-based indexes expose an equivalent in how many partitions to probe.

The reason this is step 5, not step 1: if nobody changed it, it isn’t your regression. A parameter that was fine at 2M vectors is usually still fine at 3M. Turning it down will make latency better and results worse, which looks like a fix on the dashboard and isn’t one.

Two cases where it genuinely is the cause: someone raised it recently and didn’t mention it, or your result quality requirements changed and it was raised deliberately without re-checking the latency budget.

If you do change it: run the recall probe from what to monitor before and after. Never adjust this without measuring what it cost. Rollback: it’s a runtime parameter — revert it immediately if recall drops.

Step 6: everything else

Rarer, but real:

Noisy neighbours. Co-tenanted infrastructure, another workload’s IO.
Embedding latency counted as query latency. If you measure end to end, a slow embedding model or a rate-limited API looks exactly like a slow database. Instrument the two separately — this is worth doing before you need it.
Result payload size. If queries return large stored text, you may be measuring serialization and network transfer. Compare latency with payloads excluded.
Connection pool exhaustion. Client-side queuing presenting as server-side latency. Check whether the engine’s own timing agrees with your client’s.

That last one is worth a standing habit: compare client-measured latency with engine-reported latency. When they diverge, the problem is not in the database and you can stop looking there.

The order, condensed

Read the percentile shape — it tells you which branch to take.
Memory: is the index still resident?
Filters: is a selective filter forcing repeated traversals?
Segments: has the count grown?
Tombstones: what fraction is deleted?
Search parameters: did anyone change them? (Usually no.)
Outside the database: embedding, payloads, pooling, neighbours.

Most incidents are step 2 or step 3. If you find yourself at step 5 within the first ten minutes, go back to step 0 and read the shape again.