Sizing a vector index in memory

The pod has 16 GB. Your vectors are 1.2 GB. The index OOMs during build, and nobody can explain where the other 14 GB went.

This is the most common capacity surprise with vector databases, and it happens because the raw vector payload is usually the smallest term in the total. Everything else — the graph structure, the build-time scratch space, the metadata, the second copy that exists during a rebuild — is invisible until you go looking for it.

The terms

A serviceable estimate looks like this:

steady-state RAM ≈ vector payload
                 + index structure
                 + metadata / payload fields
                 + engine overhead and free-list slack

And separately, the number that actually kills you:

peak RAM ≈ steady-state + build scratch + (second copy, if rebuilding in place)

You provision for peak, not steady state. Taking the terms in order.

Vector payload

The easy one:

vector bytes = vectors × dimensions × bytes_per_component

bytes_per_component is 4 for float32, 2 for float16 or bfloat16, 1 for int8, and a fraction of a byte under product quantization. This term is fully predictable and it is the one everyone calculates.

Worked example — all inputs below are hypothetical placeholders. Substitute yours.

Take 5,000,000 vectors at 768 dimensions, float32:

5,000,000 × 768 × 4 bytes = 15.36 GB

Index structure

This is where the surprise lives, and it depends on the index family.

Graph indexes (the HNSW family, and most defaults) store, for every vector, a list of neighbour references — and multiple such lists, one per layer. The dominant term is the base layer:

graph bytes ≈ vectors × avg_degree × bytes_per_reference × layer_factor

avg_degree is governed by the engine’s connectivity parameter, bytes_per_reference is typically 4 or 8, and layer_factor accounts for the upper layers — those are geometrically smaller, so it’s a modest multiplier above 1, not a large one.

The important property: this term is independent of dimensionality. It scales with vector count and with the connectivity setting. A collection of 100-dimension vectors and one of 1536-dimension vectors, at the same count and connectivity, carry near-identical graph overhead. That’s why low-dimensional collections often show the graph dominating the payload, which surprises people who budgeted from dimensionality alone.

Cluster-based indexes (the IVF family) are much lighter structurally — centroids plus an inverted list assignment per vector — but they buy that saving with a recall/latency profile you tune differently, and quantized variants trade payload size for a codebook and a reconstruction step.

Continuing the hypothetical example, at 5,000,000 vectors with an average degree of 32 and 4-byte references, allowing a modest multiplier for upper layers:

5,000,000 × 32 × 4 × 1.15 ≈ 0.74 GB

Small next to the payload here, because 768 dimensions is a lot. Rerun it at 128 dimensions and the payload drops to 2.56 GB while the graph term doesn’t move — now it’s a quarter of the total.

Metadata and payload fields

Anything you store alongside the vector: IDs, source URLs, tenant keys, timestamps, chunk text if you kept it in the same store. Highly variable, frequently underestimated, and it does not compress the way you hope when it’s held for fast filtered access.

If you filter on metadata, some engines maintain secondary structures for those fields. Those are additional, and they scale with cardinality.

The honest approach is measurement: write 10,000 representative records into a scratch collection, measure the delta, multiply.

Engine overhead and slack

Allocator fragmentation, free lists, per-segment headers, in-flight request buffers, and the engine’s own working set. This is not a rounding error — treat it as a real fraction of the total rather than assuming zero.

Build scratch and the second copy

The two terms that turn a comfortable estimate into an OOM.

Build scratch. Constructing a graph index is not a streaming operation. The builder holds candidate sets, priority queues, and often a partially-built structure alongside the data. Some engines build segment-by-segment and keep the peak bounded; others build the whole thing in one pass and peak far above steady state.

The second copy. If you rebuild in place, both the old index (still serving) and the new one (being built) exist simultaneously. That is close to a doubling of the index terms, on top of build scratch. This is the single most common cause of “it worked yesterday and OOMed today” — nothing about the data changed, someone just triggered a rebuild.

Whether you pay this depends on your rebuild strategy, which is a decision to make deliberately rather than discover during an incident — see reindexing without downtime.

Putting it together

For the hypothetical 5M × 768 float32 collection:

Term	Estimate
Vector payload	15.36 GB
Graph structure	0.74 GB
Metadata	measure it
Overhead / slack	a real fraction, not zero
Steady state	~16 GB + metadata + slack
Rebuild peak	roughly double the index terms, plus scratch

The lesson from the table isn’t the numbers, which are made up. It’s the shape: at high dimensionality the payload dominates and the graph is a rounding error; at low dimensionality they converge; and the rebuild peak is the number you actually provision against.

Levers, and what each one costs

When the estimate doesn’t fit the budget, in rough order of value-for-pain:

Reduce bytes_per_component. Moving float32 → int8 or float16 cuts the largest term by 2–4×. It costs some recall, and how much is entirely dependent on your embedding distribution — measure it against a known query set rather than accepting a general claim.

Reduce dimensionality. Some embedding models are explicitly trained so that truncated prefixes remain useful, which makes this cheap where available. Otherwise it means a different model and a full rebuild.

Lower the connectivity parameter. Shrinks the graph term directly and reduces build time. Costs recall, more so on hard queries. Only meaningful when the graph term is actually significant — at 768 dimensions it isn’t.

Move payload off the hot path. Keep IDs in the vector store and the chunk text in whatever you’re already running. Adds a lookup to every query; often worth it.

Shard. Doesn’t reduce total memory, but it splits it across machines and caps the blast radius of a rebuild. This is usually the answer past a certain size regardless.

What to do before provisioning

1. Write 10,000 representative records — real vectors, real metadata — into a scratch collection.
2. Record resident memory before and after. Divide by 10,000 to get bytes-per-record as your engine actually stores it, including metadata and overhead. This one measurement replaces three of the estimates above.
3. Multiply by your projected count at your planning horizon, not today’s count.
4. Add your rebuild strategy’s peak multiplier.
5. Provision for that, then confirm by building a full-scale index once in a scratch environment before it matters.

Rollback: if the full-scale build fails, the scratch environment absorbs it and nothing is serving from it. That is the entire point of doing it there. Delete the collection, adjust one lever, repeat.

Sizing that’s been validated by an actual full-scale build is worth more than any formula on this page. Then watch it — what to monitor covers the signals that tell you the estimate is drifting before it becomes an incident.