I Benchmarked Dense Retrieval on 1M Docs With Python and Go

While working on dense retrieval systems, I wanted a clear answer to a simple question: how far can plain dense search go on a real machine before something becomes the bottleneck?

I ran a capacity experiment on a real box: 8 vCPUs, ~32 GB RAM (31 GB usable), no swap, ~300 GB NVMe, Amazon Linux 2023. The corpus was 1,000,000 MS MARCO passages, embedded with BGE-small (bge-small-en-v1.5) at 384 dimensions, L2-normalized.

The goal was not to make a RAG demo look impressive. The goal was:

How hard can I push plain dense search before the machine — or my stack — starts giving misleading results?

This is the full write-up. Every stage gate, every serving stack (Python single-process, multi-process, Python→Qdrant proxy, Go searchd→Qdrant), every limitation I hit, and every table that mattered. The numbers come from measured runs under a consistent SLO unless I say otherwise.

I kept the tables because that is how I measured. After each one, I explain what it actually means, because the numbers alone are not enough without context.

1. Scope and ground rules

In scope

Dense vector retrieval only: BGE-small 384-d, L2-normalized, Cosine / inner-product style scoring
HNSW (and flat / PQ / partitioned variants where I gated them)
HTTP POST /search with precomputed query vectors (qidx) so embedding time is excluded
Capacity: QPS, latency percentiles, RSS / Docker memory, worker counts
Serving in Python (master) and Go + Qdrant (feat/go-qdrant-server)

Out of scope for the capacity numbers I treat as reliable

Cross-encoder / LLM reranking
Production query embedding at load (GPU)
Multi-node distributed Qdrant clusters (I only estimate those)
Track S (30M+ docs) full capacity — designed for, but not fully load-tested here

SLO I used

Name	Definition
Sustainable QPS	Highest achieved QPS in a concurrency ramp where p99 ≤ 100 ms and error rate ≤ 5%
Peak / exhaust QPS	Highest QPS seen with SLO disabled (latency may be terrible)
Autocannon QPS	Fixed JSON body every request — optimistic, often cache-shaped

What this table is saying: These three “QPS” numbers mean different things, so I keep them separate. Sustainable is the number I would actually ship with — tails still under 100 ms. Peak is useful for finding where the system plateaus, not for marketing. Autocannon with the same body forever often looks 1.5–2× better because it keeps hitting one hot path. If someone quotes 3–4k from autocannon, it is worth asking whether the queries actually rotated.

2. Hardware, corpus, and how I measured

Machine

Resource	Value
vCPU	8
RAM	31 GB
Swap	none
Data disk	~300 GB NVMe (`data/`)
OS	Amazon Linux 2023

What this means in practice: Eight cores is enough to compare GIL vs multi-process and Go vs Python, but not enough to pretend one box is a product for 100k QPS. Thirty-one GB sounds large for 1M vectors (~1.5 GB raw) — and it is, until you copy the HNSW index once per process. No swap means that when I overshoot RAM, I get an honest failure (pressure / OOM risk), not a soft slowdown through paging.

Corpus

Item	Value
Passages	1,000,000 (MS MARCO subset)
Embedding model	`bge-small-en-v1.5`
Dim	384
Normalization	L2
Passage vectors on disk	~1.5 GB `passages.f32.npy`
Dev queries with vectors	6,980
Ground truth	FAISS `IndexFlatIP`, k=100

Reading this: One million is large enough that flat search dies (~12 QPS) and HNSW becomes necessary, but small enough that the full graph still fits in RAM on this box. That matters later: partitions and graph managers are tools for when the index does not fit — on 1M they mostly just cost QPS. L2 normalize means cosine ≈ inner product, so I can use IP-style indexes cleanly. Ground truth from flat FAISS is how I know recall@10 is real.

Load tools

Tool	Role
`bench/capacity_test.py`	Python concurrency ramp + p50/p95/p99 + SLO stop
`go/cmd/loadtest`	Same idea in Go
`autocannon`	HTTP blast; fixed body unless customized
`/stats` + `ps` + `docker stats`	RSS / CPU samples

Why all four: The ramp finds the SLO knee. Autocannon finds an optimistic ceiling. Stats tell me which process is burning CPU/RAM — searchd vs Qdrant vs Python worker — so I do not blame the wrong layer.

Request shape

POST /search
Content-Type: application/json

{"qidx": 1234, "k": 10}

The server maps qidx → a row in queries.f32.npy and runs ANN. That isolates retrieval, not the embedder. If you include embedding in a “search QPS” number on CPU, you are measuring a different system.

3. What “simple search” is (and is not)

Pipeline I capacity-tested as “plain”

qidx → float32[384] → HNSW search (ef=64, k=10) → JSON hits

Pipeline stages that exist in the roadmap but are not in the Go hot path

Stage	Idea	Capacity outcome on 1M
0 Flat	Exact IP	Correct GT; ~12 QPS
1 HNSW	Single in-RAM graph	Best simple path
2 PQ	Product quantization	Recall fail + slow smoke
4 Partitioned	K graphs + nprobe	Quality OK; QPS much worse
5 Graph manager	LRU partitions under RAM budget	Quality OK; QPS much worse
Cache / rescore	Extra layers on HNSW	Lower QPS under load
8 Hybrid BM25+RRF	Lexical + dense	Smoke only; BM25 ~2s (Python)
9 Rerank	Cross-encoder	Not gated under load

How to read this roadmap table: Stage 0 proves the harness. Stage 1 is the only thing that wins on capacity on this box when the index fits. Stages 2/4/5/cache/rescore are the “smarter” ideas that seemed like they should help throughput — and they did not, not at 1M in RAM. Hybrid and rerank are quality/product features; I never treated them as capacity truth. Go + Qdrant today = Stage-1-class ANN only (HNSW inside Qdrant). No rerank. If plain top-k is broken, everything fancy on top does not fix the foundation.

4. Stage gates: quality before capacity

I refused to publish capacity wins that destroyed recall. Gates first.

Stage 0 — Flat exact

Metric	Result
recall@1 / @10 / @100	1.0 / 1.0 / 1.0 (500 queries)
p50 / p99 latency	~164 / ~241 ms
Gate	Passed (harness + GT valid)

Explanation: Perfect recall at every k means my ground-truth pipeline is trustworthy. Latency in the hundreds of ms is expected — you are scanning 1M vectors. Flat is not a serving strategy; it is a correctness anchor. If this gate fails, every later recall number is meaningless.

Stage 1 — Naive HNSW (`hnswlib`)

Config	recall@10	p50 ms	p99 ms	Notes
Flat (500 q)	1.000	164	241	Reference
HNSW M=16, ef_construction=200, ef=64	0.956	0.58	1.55	Floor ≥ 0.90
Index file	—	—	—	~1.6 GB `hnsw.bin`

Explanation: HNSW keeps ~95.6% of flat’s top-10 neighbors while dropping latency from ~164 ms to sub-millisecond. That is the whole point of ANN. The ~1.6 GB on disk is the graph+vectors file you will pay again in RSS when you load it in Python. Gate passed. Serving QPS dies later in HTTP/JSON/process model — not inside knn_query when warm.

Stage 2 — PQ smoke (FAISS IndexPQ m=64)

Metric	Result
recall@10	0.678 (failed floor)
p50 / p99	33.9 / 55.5 ms (slower than HNSW)
Index size	~64 MB

Explanation: This is where the “quantize and everything gets faster” idea failed for me. The index got tiny (~64 MB), but recall fell to 0.68 (below my 0.90 floor) and search got slower than HNSW. So naive PQ bought neither quality nor speed. Quantization can still matter for RAM at bigger scale — but “quantize = free QPS” was wrong here.

Stage 4 — Partitioned multi-graph HNSW

Setup: K=256 partitions, overlap=1, M=16, ef=64.

nprobe	recall@10	p50 ms	p99 ms
8	0.856	~4.8	~11
16	0.914	~9.3	~15
32	0.954	~18.8	~25
64	0.980	~38	~45

Partition size imbalance: min / median / max = 307 / 3651 / 11128.

Explanation: Raising nprobe buys recall by searching more partition graphs — and latency scales roughly with that work. At nprobe=16 I clear the 0.90 floor (~9 ms p50). At nprobe=64 I am almost flat-quality but ~40× slower than single HNSW’s sub-ms search. The partition size skew (307 vs 11k docs) means some queries hit fat partitions and get worse tails. Quality gate passed at nprobe=16. Capacity later showed: when one HNSW already fits, paying nprobe walks is a QPS cost for a problem you do not have yet.

Stage 5 — Graph manager (LRU, 8 GB budget)

Metric	Warm zipf (500 q)
recall@10	0.956
p50 / p99	~9.6 / ~19.5 ms
Gate	Passed functionally

Explanation: Warm zipf looks fine — same recall as HNSW, latency ~10–20 ms because you are routing through partition loads. Cold / load-on-miss under real concurrency later destroyed p99. So: useful when the full graph will not fit; harmful when it does and you still pay the manager.

Stage 8 — Hybrid BM25 + dense + RRF (smoke only)

Metric	Value
nDCG@10 / MRR@10	~0.64 / ~0.59
Latency	~2 s/query (pure-Python BM25 over 1M)
Status	Smoke — not a capacity candidate until Tantivy (or similar)

Explanation: Ranking metrics are not terrible for a smoke hybrid, but 2 seconds per query from pure-Python BM25 over 1M docs means this path is not usable for load tests. I never put hybrid in the capacity scoreboard. Lexical search needs a real inverted index engine first.

5. Python architecture

End-to-end request path

Client
  → ThreadingHTTPServer (1 process)  OR  N× SO_REUSEPORT workers
    → JSON parse (Python, GIL)
    → query_vectors[qidx]
    → pipeline.search(...)
         → HNSWIndex.knn_query  (hnswlib C++, releases GIL during search)
    → JSON serialize (Python, GIL)
  ← hits

Native search can overlap. The glue around it mostly cannot — that is the GIL story in one diagram.

HNSW index (algorithm core)

Roughly what lived on master in engine/index/hnsw.py:

class HNSWIndex:
    def __init__(self, vectors=None, space="ip", M=16,
                 ef_construction=200, ef_search=64, index_path=None):
        import hnswlib
        vectors = np.ascontiguousarray(vectors, dtype=np.float32)
        self.n, self.dim = vectors.shape
        self._index = hnswlib.Index(space=space, dim=self.dim)
        self._index.init_index(
            max_elements=self.n, ef_construction=ef_construction, M=M
        )
        self._index.add_items(vectors, np.arange(self.n))
        self._index.set_ef(ef_search)

    def search(self, query, k: int):
        q = np.ascontiguousarray(query.vector.reshape(1, -1), dtype=np.float32)
        labels, distances = self._index.knn_query(q, k=k)
        return [
            Hit(doc_id=int(lab), score=float(-dist), source="dense")
            for lab, dist in zip(labels[0], distances[0])
        ]

Index-level search is sub-millisecond when warm. Serving is the bottleneck for QPS.

Single-process HTTP server

ThreadingHTTPServer, one process, many threads:

httpd = ThreadingHTTPServer((host, port), Handler)

# each request thread:
body = json.loads(self.rfile.read(length))
vec = query_vectors[int(body["qidx"]) % len(query_vectors)]
hits = pipeline.search(
    Query(vector=np.asarray(vec, dtype=np.float32)),
    k=int(body.get("k", 10)),
)

Multi-process server (GIL escape hatch)

Each worker builds a full private HNSW copy (~1.8 GB) and binds with SO_REUSEPORT:

class ReuseThreadingHTTPServer(ThreadingHTTPServer):
    def server_bind(self):
        self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
        super().server_bind()

# Each worker process:
pipeline = build_pipeline(cfg)   # FULL private HNSW copy (~1.8 GB)
httpd = ReuseThreadingHTTPServer((host, port), make_handler(...))
httpd.serve_forever()

python3 -m engine.serve_mp --config configs/stage01_hnsw.yaml --workers 4 --port 8080

Architecture takeaway: Threads share one interpreter and one index — cheap RAM, GIL cost. Processes share the port via SO_REUSEPORT but duplicate the ~1.8 GB index — expensive RAM, real multi-core. That trade is the entire Exp 2 story.

Python → Qdrant proxy

Same HTTP API; ANN runs in Qdrant (Rust). Python still owns accept / JSON / GIL:

res = client.query_points(
    collection_name="msmarco_1m",
    query=vec,
    limit=k,
    search_params={"hnsw_ef": 64},
)

6. Experiment 1 — Single-process capacity

Server: engine/serve_http.py (threaded, 1 process)

Headline table

System	Sustainable QPS	SLO used	Server RSS	Notes
Our HNSW	~960	p99≤100ms @ c=8	~1.8 GB	Best dense single-proc path early
Our HNSW (exhaust)	~1500 peak	none	~1.8 GB	p99 → seconds at high c
Autocannon HNSW c=64	~1668 req/s	—	—	Overloaded / optimistic
Partitioned nprobe=16	~174	p99≤200ms	~2.4 GB	All partitions in RAM
Graph manager LRU 8GB	~166	p99≤300ms	~2.4 GB	After warm
HNSW + rescoring	~170	p99≤100ms	~3.2 GB	Needs fp32 vectors
HNSW + result/semantic cache	~231	p99≤100ms	~1.8 GB	Zipf helps; locks hurt
Flat (FAISS)	~12	—	~3.0 GB	Unusable for load
Qdrant (Python proxy)	~403	p99≤100ms @ c=8	Qdrant ~175 MB cold	RAM king, QPS weak via proxy

Walk through the rows: Plain in-RAM HNSW is the only normal answer here — ~960 sustainable, ~1.5k if you ignore the SLO. Everything I layered on top to be clever (partitions, graph manager, cache, rescore) lands in the 160–230 QPS band — roughly 4–6× worse than plain HNSW, often with more RAM. Flat at 12 QPS is the lower bound. Qdrant via Python proxy looks RAM-efficient (~175 MB cold) but stuck around 400 QPS — at this point I thought on-disk Qdrant was slow. Later I learned the proxy was the real ceiling.

Also notice different SLO relaxations (200–300 ms) on partitioned/GM — those systems could not hold a fair fight under the same 100 ms p99 without looking worse.

Raw JSON peaks (Exp 1 files)

File	Peak QPS	c	p99 ms
`hnsw.json`	1557	4	4.4
`hnsw_exhaust_v2.json`	1547	8	6.5
`qdrant.json`	408	4	15.9
`hnsw_cache.json`	231	8	90.6
`hnsw_rescored.json`	175	8	110.7
`partitioned_v2.json`	176	4	33.4
`graph_manager_v2.json`	168	4	34.8
`partitioned.json` (early)	79	4	105
`graph_manager.json` (cold-ish)	61	4	270
`flat.json`	12	4	533
`go_qdrant.json` (cold smoke)	342	8	523

How to read peaks vs sustainable: Peak QPS in these files is “best step in a ramp,” not always SLO-clean. HNSW peaks ~1.5k with tiny p99 at low concurrency — then tails explode if you push harder. Early partitioned/GM rows (79 / 61) are cold or mis-tuned; warm v2 rows (~168–176) are the fairer story and still far below HNSW. The cold Go smoke (342 QPS, p99 523) is a warning: never publish first-minute numbers.

What Exp 1 taught me

Observation	Meaning
~1.8 GB RSS, ~29 GB still free	Not RAM-bound for 1M HNSW
QPS stuck ~1–1.5k	CPU + GIL / one process
Flat ~12 QPS	Exact search dead for capacity
Partitioned / GM ≪ HNSW	Extra probes cost more than they save on 1M
Cache / rescore ≪ plain HNSW	Extra Python work + locks

I had almost 30 GB free and still could not break past ~1–1.5k in one process. That is not a “buy more RAM” problem — it is a process-model problem. Fancy layers made it worse on this corpus size.

7. The GIL problem

What people expect

ThreadingHTTPServer → many threads → use all 8 cores → ~8× QPS.

What actually happens

┌─────────────────────────────────────────────┐
│  One Python process                         │
│  ┌─────┐ ┌─────┐ ┌─────┐                    │
│  │ T1  │ │ T2  │ │ T3  │  ...               │
│  └──┬──┘ └──┬──┘ └──┬──┘                    │
│     │       │       │                       │
│     ▼       ▼       ▼                       │
│   JSON / routing / numpy glue  ←── GIL      │
│     │       │       │                       │
│     ▼       ▼       ▼                       │
│   hnswlib C++ knn_query  ←── can run parallel│
│     (GIL released in native code)           │
└─────────────────────────────────────────────┘

Native HNSW work can overlap across threads.
Request parsing, pipeline glue, JSON, stats locks still serialize on the GIL.
Net effect: sublinear scaling; I plateaued near ~1k sustainable / ~1.5k exhaust in one process.

Proof that process count mattered

When I moved to multi-process (separate interpreters + separate indexes), QPS jumped toward ~2.1k — until cores and RAM became the limits. Classic “GIL escape via processes” pattern. Threads were never going to get me there. If your mental model is “hnswlib releases the GIL so threads should scale,” you are half right about the C++ call and fully wrong about the HTTP server around it.

8. Experiment 2 — Multi-process SO_REUSEPORT

Server: python3 -m engine.serve_mp --workers N

HNSW primary scaling curve

Workers	Best under p99≤100ms	Peak QPS (JSON)	Approx RAM
1	~1507 @ c=16	1622	~1.8 GB
2	~2013 @ c=32	2158	~3.6 GB
4	~2139 @ c=64	2275	~7 GB
8	~2138 @ c=64	2299	~14 GB

Autocannon HNSW w=4, c=64, 20s: ~4211 req/s (fixed-body optimistic — do not confuse with sustainable).

Explanation of the scaling curve: Going 1→2 workers is the big win (~1500 → ~2000). 2→4 squeezes another ~100–150 QPS. 4→8 is flat on QPS and doubles RAM again (~7 → ~14 GB). On an 8-vCPU box you have saturated useful parallelism by ~4 workers; the eighth worker is mostly an expensive clone of work you cannot schedule. Sweet spot: w=4 ≈ 2140 QPS for ~7 GB. That is still the absolute QPS champion on this machine.

The autocannon 4211 number is what happens when every request is the same body — useful as an optimistic ceiling, but not the fair number.

Partitioned (nprobe=16) under MP

Workers	Best QPS	SLO note
2	~305	p99≤200ms class
4	~477	still ≪ HNSW

Explanation: Multi-process helps partitioned search scale (~305 → ~477), but you are still at roughly 1/4 of plain HNSW MP. Workers multiply RAM for partition sets too. So MP does not rescue a probe-heavy design on 1M.

Graph manager under MP

Workers	Best QPS
2	~290
4	~448

Explanation: Same shape as partitioned — scales with workers, still far below HNSW, and each process has its own LRU so you duplicate caches instead of sharing one hot set. Worse RAM efficiency than a single fat process for the same idea.

HNSW + cache under MP w=4

Result	Detail
Peak	~386 QPS
p99≤100ms	Failed (p99≈116 ms on first step)
Issue	Private caches per worker → diluted hits + lock overhead

Explanation: Caching under multi-process backfired. Each worker owns a private cache, so zipf hits do not concentrate; locks add latency; the system failed the 100 ms p99 gate immediately. If you want cache + MP, you need shared memory or Redis — not N independent dicts.

Qdrant under Exp 2 probes

Probe	Result
Proxy c=8 p99≤100	~403 QPS, ~175 MB
Autocannon c=64	~398 req/s, timeouts
`qdrant_high.json`	Collapsed (~2.8 QPS, p99 ~6.6 seconds) — overload / timeout cascade

Explanation: Qdrant via Python proxy does not magically get better under Exp 2-style pressure — you are still bottlenecked at the front door. Autocannon stalls around the same ~400. qdrant_high shows the overload cascade: push concurrency past the plateau and the system stops being “slow” and becomes almost unavailable (multi-second p99, single-digit QPS). That is why admission control matters in production.

Exp 2 issues (HNSW MP)

Issue	Detail	Impact
Diminishing returns after w=4	w8 ≈ w4 QPS on 8 cores	Wasted RAM
RAM × workers	Full HNSW copy per process	Memory wall at w=8
Latency cliff	Client concurrency ≫ useful parallelism	p99 → ~1s
No shared memory index	Each worker private	Cannot share 1.8 GB graph
Still one machine	No horizontal story	Caps at ~2.1k here

In plain English: Processes fix the GIL and then introduce a new cost — memory. Without a shared mmap/index, N workers ≈ N× RAM. On 8 cores, more workers after 4 do not buy QPS. Client concurrency past the useful point just queues work and destroys p99. And none of this is a multi-node story yet — you are capped near 2.1k on this host.

Cross-system bottleneck table (Exp 2)

Bottleneck	Who hits it first
Python GIL / single process	Exp 1 HNSW (~1k)
Cores saturated + index copies	Exp 2 HNSW w≥4 (~2.1k plateau)
RAM × workers	HNSW w=8; partitioned/GM w=4
Probe fan-out (nprobe)	Partitioned / GM
Disk / cold cache	Graph manager cold start
On-disk ANN + Python proxy	Qdrant ~400 QPS
Client overload / timeouts	Autocannon on slow backends; `qdrant_high`

Explanation: This is a map of failure modes, not a ranking of products. Different stacks fail differently: GIL first for single-proc Python; cores then RAM for MP HNSW; probe cost for partitioned; cold disk for graph manager; proxy for Python→Qdrant; timeout storms when you ignore the plateau. Knowing which wall you are on decides whether the fix is processes, fewer workers, Go, or simply stopping the flood.

Exp 2 verdict: MP ~2× HNSW QPS (1.5k → 2.1k) with sweet spot w=4. Beyond that: little QPS, lots of RAM. That ~2140 number is still the absolute QPS leader on this box — paid for with ~7 GB.

9. Qdrant path: Python proxy → Go searchd

Collection (shared)

Setting	Value
Name	`msmarco_1m` (later also `msmarco_1m_ram`)
Points	1,000,000
Distance	Cosine
HNSW	M=16, ef_construct≈100–200
Vectors on_disk (original)	true
Docker	`qdrant/qdrant:v1.13.2`, ports 6333/6334

What this config implies: Same ANN family as my Python HNSW (M≈16, Cosine). on_disk=true means Qdrant can keep a smaller resident set and lean on page cache — great for RAM, not automatically great for QPS. I later tried RAM / sharded collections in exhaust runs; simpler often won.

Why Python→Qdrant was slow (~400 QPS)

Factor	Effect
Python ThreadingHTTPServer + GIL	Front-door serialization
Sync client + JSON	Extra overhead per query
On-disk vectors	Lower than in-RAM hnswlib
Misread	I first blamed Qdrant; Go proved Qdrant could do ~4× more

Explanation: Three real costs stacked, and I attributed all of them to “Qdrant.” The clear evidence later: swap only the front door to Go, keep the same collection, and QPS jumps ~4×. So on-disk vs in-RAM mattered, but not enough to explain a 400 vs 1600 gap. This was the most important lesson in the project. I almost concluded “Qdrant is slow at ~400 QPS.” That was the wrong layer.

What Go changed

Before	After
Python proxy ~403 QPS	Go `searchd` ~1640 QPS (p99≤100ms)
Proxy RSS tied to Python	searchd ~52 MB
Same Qdrant	Same Qdrant (now the limiter)

Explanation: Same database, different front door. Throughput ~4×. Front-door RSS collapses to ~52 MB. After that, when I look at Docker stats under load, Qdrant CPU is the thing pegged — which is what you want. The limiter moved to the ANN engine, not the language runtime.

10. Go architecture

Layout

go/cmd/searchd          HTTP + SO_REUSEPORT flag
go/internal/server      /health /stats /search → Qdrant gRPC
go/cmd/loadtest         concurrency ramp
go/cmd/ingest           rebuild collections (RAM / shards)
scripts/bench_go_qdrant.sh
scripts/bench_exhaust.sh
scripts/run_go_searchd_mp.sh

Search handler (idea)

vec := vectors.Row(s.queries, s.nQueries, s.dim, *req.QIdx)

res, err := s.client.Query(ctx, &qdrant.QueryPoints{
    CollectionName: s.cfg.Collection,
    Query:          qdrant.NewQueryDense(vec),
    Limit:          qdrant.PtrOf(uint64(req.K)),
    WithPayload:    qdrant.NewWithPayloadEnable(false),
    Params: &qdrant.SearchParams{
        HnswEf: qdrant.PtrOf(s.cfg.EF),
    },
})

Multi-process listen (optional)

unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEPORT, 1)
ln, err := lc.Listen(ctx, "tcp", *addr)
httpSrv.Serve(ln)

Concurrency model difference

	Python	Go
Unit of concurrency	Thread (+ GIL) or process	Goroutine (no GIL)
Front-door cost	High	Low (~52 MB RSS)
To use 8 cores on in-proc HNSW	N processes × N RAM	N/A on this branch (Qdrant owns ANN)
To use 8 cores with Qdrant	Proxy often limited first	Goroutines feed Qdrant until Qdrant CPU saturates

Explanation: Python needs processes (and RAM copies) to feed cores for in-proc HNSW. Go can accept and fan out cheaply; with Qdrant as the ANN owner, searchd’s job is to keep Qdrant busy without becoming the bottleneck itself. Once Qdrant is at ~7× CPU on 8 vCPUs, spawning more Go processes does not help — which the exhaust matrix confirms.

Go did not invent a new ANN. It removed the front-door cost and spoke gRPC properly. Then Qdrant became the honest wall.

11. Go capacity + exhaust matrix

Fair loadtest (rotating qidx), p99≤100ms

c	QPS	p50 ms	p95 ms	p99 ms
8	1290	5.7	10.6	14.2
16	1410	10.8	18.5	23.2
32	1543	20.2	31.7	38.3
64	1617	39.2	57.1	65.8
96	1640	58.0	81.5	96.6
128	1645	77.2	105.9	126 → stop

How to read this ramp: QPS climbs quickly from c=8→64, then barely moves 64→128, while latency keeps rising. That is the textbook plateau: adding concurrency no longer adds throughput; it only lengthens queues. The last legal step under p99≤100 is c=96 → 1640 QPS (p99 96.6). At c=128, p99 is 126 ms so the ramp stops. Sustainable ≈ peak here (~1620–1650) — you are latency-bound at the ANN, not leaving free QPS on the table under the SLO.

Resources (warm / under load)

Component	Memory
searchd	~52 MB RSS
Qdrant under load	~500–650 MB
Total vs Python HNSW MP w=4	~0.7 GB vs ~7 GB

Explanation: Under load Qdrant grows from the cold ~175 MB class into ~0.5–0.65 GB (cache/working set). Still an order of magnitude leaner than Python MP HNSW’s ~7 GB for only ~500 fewer QPS. searchd itself is basically a thin gRPC/HTTP shim in memory terms.

Autocannon (fixed body — optimistic)

c	Avg req/s	p99 latency
64	~2988	~39 ms
128	~2759	~78 ms
256 (exhaust finale)	~2786	~177 ms

Explanation: Same query forever → ~3k req/s and still-OK latency at c=64. That is why I label autocannon optimistic. Fair rotating loadtest says 1640. If you only ever quote autocannon, you are measuring a cache-shaped path, not fair query mix.

Exhaust matrix

Config	Peak QPS	c	p99 ms	Lesson
w4 ef=32 RAM	1792	512	483	Best exhaust; latency bad
w8 ef=32 RAM	1776	512	521	Same plateau
w1 ef=64 disk	1713	256	226	Strong; simple wins
w4 ef=64 disk	1654	256	237	More Go workers ≠ more QPS
w1 ef=64 RAM (4 shards)	1229	384	494	Sharded RAM coll. slower here
w2/w4/w8 ef=64 RAM	~1194–1210	high	519–670	Worse than disk coll.

Explanation of exhaust: I tried more Go workers, lower ef, RAM collections, shards. Best peak (~1792) needs ef=32 and accepts half-second p99 — not shippable under my SLO. w8 ≈ w4. Single searchd into the disk collection is already strong (~1713 peak). The 4-shard RAM collection was a regression (~1.2k) — “more shards / more RAM flags” is not a free win on one 8-vCPU host. Qdrant stayed the CPU wall (~700% CPU while searching).

Cold-start issue (early Go smoke)

Run	QPS	p99
First short ramp (cold)	~342	523 ms
After warm	~1300+ @ c=8	~14 ms

Explanation: Same binary, same collection, very different numbers once page cache and connections are warm. Cold start lied by ~4× on QPS and ~37× on p99. Always warm before you measure; discard the first ramp if you forgot.

12. Master comparison tables

A. All serving stacks (headline)

Stack	Sustainable QPS	Peak	RAM	Wall
Flat exact	~12	—	~3 GB	CPU (brute force)
Python→Qdrant proxy	~403	~400	~175 MB cold	Proxy + GIL
Python HNSW 1-proc	~960–1507	~1550–1620	~1.8 GB	GIL / 1 proc
Go + Qdrant	~1640	~1650–1792	~0.7 GB	Qdrant CPU
Python HNSW MP w=4	~2140	~2275	~7 GB	8 cores; RAM at w8

The scoreboard in words: Flat is proof, not a product. Python→Qdrant is the false “DB is slow” row. Single-proc HNSW is the GIL ceiling. Go+Qdrant is the efficiency winner (~1640 @ ~0.7 GB). Python MP w=4 is the raw QPS winner (~2140) if you are willing to burn ~7 GB and accept process copies. The walls column tells you what to upgrade next for each row.

B. Quality vs speed (index level)

Index	recall@10	Typical latency	Capacity role
Flat	1.00	~164 ms	GT only
HNSW ef=64	0.956	~0.6 ms	Production simple path
Partitioned nprobe=16	0.914+	~9 ms	Scale/RAM designs
Graph manager warm	0.956	~10 ms	When full graph won’t fit
PQ m=64 smoke	0.678	~34 ms	Failed

Explanation: Only HNSW gives near-flat recall at microsecond–millisecond latency. Partitioned/GM keep recall acceptable but move latency into tens of ms before HTTP overhead — which hurts QPS when a single graph fits. PQ failed the quality bar entirely.

C. “Fancy” layers vs plain HNSW (1-proc load)

System	QPS vs plain HNSW
Plain HNSW	baseline
+ cache	worse (~231)
+ rescore	worse (~170)
Partitioned	worse (~174)
Graph manager	worse (~166)
Hybrid BM25 (smoke)	not capacity-viable (~2s)

Explanation: On 1M that fits in RAM, every “smart” add-on I load-tested lost to plain HNSW. Not slightly worse — order-of-magnitude in some cases. Caches and partitions are tools for other constraints (repeat traffic with shared cache, indexes that do not fit). They are not universal QPS multipliers.

D. Python vs Go for the same Qdrant

Front door	QPS	Δ
Python proxy	~403	1×
Go searchd	~1640	~4.1×

Explanation: Controlled experiment. Same Qdrant, same collection, same box. Change only the HTTP/gRPC front door → 4.1×. That is the cleanest proof the old bottleneck was not “Rust ANN cannot go fast.”

E. Cost of another ~500 QPS (MP HNSW vs Go+Qdrant)

	Go+Qdrant	Python HNSW w=4
QPS	~1640	~2140 (+~500)
RAM	~0.7 GB	~7 GB (~10×)

Explanation: You are buying roughly +30% QPS for roughly +10× RAM. Sometimes that is worth it (dedicated fat node, max throughput). Often it is not (density, multi-tenant, headroom). That is the trade I care about when someone says “just keep the index in process memory.”

13. Issue catalog (everything that hurt)

Operational and technical issues across the project. I am listing them because half of capacity work is fighting the harness, not the ANN. Each block below is “what broke → why it mattered → what I did.”

Infrastructure / data

#	Issue	What happened	Impact	Mitigation
I1	Data disk not mounted	Large downloads/embeddings needed `data/` on xvdf	Blocked Stage 0	Format + mount 300GB disk
I2	Embedding time / CPU	1M×384 BGE on CPU is slow	Interrupted; needed resume	50k checkpoint flushes + progress log
I3	No swap	Large peaks have no safety net	OOM risk at HNSW w=8	Cap workers; watch `free`
I4	`pip` / toolchain gaps	`hnswlib` needed g++	Build failures	`dnf` install compilers

Story behind this block: Before any QPS work, the box fought me on basics — disk mount, hours of CPU embedding without checkpoints, compilers missing for hnswlib, and no swap when multi-process got greedy. Capacity work starts as ops work.

Measurement / process hygiene

#	Issue	What happened	Impact	Mitigation
I5	`pkill` matched script cmdline	Capacity jobs killed mid-run	Lost runs	Prefer `fuser -k port` / kill by PID file
I6	Cold vs warm confusion	Go first ramp 342 QPS / p99 523	False “Qdrant is slow”	Always warm; discard cold
I7	Autocannon fixed body	Same `qidx` forever	Inflated QPS (2.8–4k)	Report zipf/`qidx` ramp as fair
I8	SLO vs peak mixing	Exhaust QPS quoted as sustainable	Overclaim	Separate tables
I9	Graph manager cold p99	First loads from disk	Early GM numbers looked dead (~61 QPS)	Warm + v2 rerun

Story: Bad measurement creates wrong architecture conclusions. Cold ramps, fixed-body autocannon, and mixing exhaust with SLO are how you ship the wrong stack. And pkill matching your own bench script is an easy way to lose a run.

Python / GIL / MP

#	Issue	What happened	Impact	Mitigation
I10	GIL + threaded server	Threads ≠ multi-core for Python glue	~1k QPS ceiling	`serve_mp` processes
I11	RAM × workers	Each worker full HNSW	~14 GB at w=8	Sweet spot w=4
I12	Private caches under MP	Each process own cache	Cache MP failed SLO	Shared cache needs shm/redis
I13	Stats lock contention	Thread-safe latency lists	Minor overhead	Keep sampling bounded
I14	Pure-Python BM25	Hybrid smoke ~2s/query	Unusable under load	Tantivy / C++ lexical later

Story: The Python path is a sequence of escapes — threads do not escape the GIL; processes escape GIL and invent RAM copies; caches under MP dilute themselves; BM25 in pure Python is a non-starter.

Index design

#	Issue	What happened	Impact	Mitigation
I15	Partition imbalance	Sizes 307–11128	Uneven latency	Accept or rebalance centroids
I16	nprobe cost	Higher nprobe → higher recall and latency	QPS tanks vs single HNSW	Use only when RAM forces split
I17	PQ without care	recall 0.68, slower	Failed Stage 2	HNSW+quant+rescore if revisit
I18	Rescore path	Extra fp32 touches	~170 QPS	Skip for plain capacity

Story: Index “improvements” have costs. nprobe and rescore buy quality knobs and sell QPS. PQ without a careful pipeline buys neither. Partitions help fit; they do not help when you already fit.

Qdrant / API / Go

#	Issue	What happened	Impact	Mitigation
I19	`client.search` removed	qdrant-client API rename	Proxy breakage	`query_points` / Go `Query`
I20	Python proxy bottleneck	Blamed Qdrant for 400 QPS	Wrong architecture choice risk	Rewrite front door in Go
I21	PATCH `on_disk` API shape	400 Bad Request on naive PATCH	Couldn’t flip disk→RAM in place	Recreate collection via `ingest`
I22	Overload death spiral	`qdrant_high` → ~3 QPS, multi-second p99	Looks like total failure	Backpressure; don’t past plateau
I23	Go MP no gain	w1≈w4≈w8 into Qdrant	Wasted complexity	Single searchd enough
I24	4-shard RAM collection slower	Peak ~1.2k vs ~1.7k disk	Surprising regression	Prefer simpler shard layout here
I25	Errors under huge autocannon	searchd `errors` counter rose in long suite	Need monitoring	Timeouts / context cancel

Story: Client APIs break, patches fail, overload looks like “Qdrant is broken,” and “just add shards/workers” can regress. The Go rewrite fixed the real proxy issue; more Go processes did not fix Qdrant’s CPU ceiling.

Product / expectation issues

#	Issue	What happened	Impact	Mitigation
I26	“Add cache/partitions for QPS”	Intuition vs measurement	Would reduce QPS on 1M	Measure first
I27	“Quantize for huge QPS”	Stage 2 smoke failed	False hope	Quant for RAM; expect ~1.5–2× QPS max
I28	“100k QPS on one box”	Math: need ~50–80× this host	Unrealistic	Horizontal replicas

Story: These are the ideas I wanted to be true. Measurement vetoed them. Intuition is not a loadtest.

Hardware / network (local tests didn’t hit these; production will)

#	Issue	What happened / risk	Impact	Mitigation
I29	Colocated-only benches	All QPS on localhost	Hides NIC/RTT walls	Add remote-Qdrant soak before prod
I30	Cross-AZ / cross-region DB	+1–80 ms RTT on every search	Destroys p99≤100ms	Same-AZ; cache; relax SLO
I31	Raw vector over HTTP at high QPS	~3 KB/req → ~2.4 Gbit/s @ 100k	NIC / LB saturation	`qidx`/id lookup; binary proto; 10 GbE
I32	Snapshot / reindex transfer	Multi-GB index moves	Deploy windows blow up	Object storage + 10/25 GbE
I33	Short-lived connections	SYN/PPS storm at high QPS	Soft NIC / LB failure	Keep-alive, HTTP2, gRPC pools
I34	No swap + MP copies	w=8 ~14 GB on 31 GB box	OOM / reclaim jitter	Cap workers; dedicated mem

Story: My published QPS never paid a network tax because everything was localhost. Production will. This section is the “next wall” checklist so I do not rediscover it the hard way after splitting tiers.

14. Walls: CPU, RAM, GIL, disk, proxy

                    free RAM
                       │
     Exp1 HNSW ────────┤  still ~29 GB free  →  NOT memory wall
                       │
     Exp2 w=4 ─────────┤  ~7 GB used         →  OK
                       │
     Exp2 w=8 ─────────┤  ~14 GB + pressure  →  MEMORY WALL
                       │
     Go+Qdrant ────────┤  ~0.7 GB            →  RAM comfortable
                       │
                    CPU / GIL
                       │
     Python 1-proc ────┤  GIL + 8 cores underused → GIL WALL
     Python MP w=4 ────┤  cores full              → CPU WALL (~2.1k)
     Go+Qdrant ────────┤  Qdrant ~7× CPU          → CPU WALL (~1.6k)
     Go many workers ──┤  no extra QPS            → already at CPU WALL

Wall name	Symptom	Approx QPS at wall	Fix
Brute-force CPU	Flat search	~12	HNSW
GIL / 1 process	Free RAM, stuck QPS	~1.0–1.5k	Multi-process
Core saturation	w4≈w8	~2.1k	More machines
Memory × copies	w8 ~14 GB	same QPS as w4	Fewer workers
Proxy language	Qdrant underused	~400	Go front door
Qdrant ANN CPU	Go plateau	~1.6k	Replicas / bigger box
Disk cold cache	GM p99 spikes	unstable	Warm / mmap / keep hot

How to use this table: Match your symptom to a wall before changing architecture. Free RAM + stuck ~1k QPS → GIL, not “need more memory.” w4≈w8 QPS → buy machines or accept the plateau, do not clone more indexes. ~400 into Qdrant with idle Qdrant CPU → fix the proxy language. Go plateau with Qdrant at ~7× CPU → you finally found the real ANN wall. On localhost I never hit the NIC.

15. What improved results (and what did not)

Improvements that worked

Change	From → To	Why it worked
Flat → HNSW	12 → ~960+ QPS	ANN
HNSW 1-proc → MP w=4	~1.5k → ~2.1k	Bypass GIL; use cores
Python Qdrant proxy → Go	~400 → ~1640	Remove GIL proxy; gRPC
Warm-up before measure	342 → 1300+ @ c=8	Page cache / JIT / connections
ef 64 → 32 (exhaust)	~1.6k → ~1.8k peak	Less graph work (recall trade)

Explanation: The wins are structural: better algorithm (HNSW), better process model (MP), better front door (Go), honest measurement (warm), cheaper ANN params (ef). No magic feature flags.

Changes that did not improve QPS on 1M

Change	Result
More threads in one Python process	Plateau
Partitioned multi-graph	Much lower QPS
Graph manager	Much lower QPS
Result/semantic cache (Python)	Lower / failed SLO
fp32 rescore layer	Lower QPS
More Go searchd workers	Flat plateau
Naive PQ	Worse recall + latency
Hybrid BM25 (pure Python)	Seconds per query

Explanation: Intuition said “add cache / partitions / quantize / more workers for speed.” Measurement said the opposite (or “no change”) on a 1M index that already fits. This list is the anti-roadmap for this corpus size.

16. Hardware limitations (deeper)

The box looked “big” (31 GB RAM). Capacity was never a pure software story.

The resource stack

Layer	What it does in retrieval	On this box	Did it limit me?
CPU cores / clocks	HNSW distance + graph walk; JSON; gRPC	8 vCPU	Yes — primary wall (~1.6–2.1k QPS)
DRAM capacity	Hold graph + vectors + OS page cache	31 GB	Only when copying indexes (MP w=8)
DRAM bandwidth	Feed SIMD distance loops	Typical cloud ~50–100+ GB/s shared	Secondary; fat in-RAM HNSW
Disk (NVMe)	Cold load, on-disk Qdrant vectors, checkpoints	Fast local NVMe	Cold GM / first Qdrant warm; not steady-state HNSW
NIC / network	Client↔API; API↔remote DB	Localhost in tests	Not hit here; real when DB is off-box
Swap	Emergency paging	None	No soft landing — OOM or reclaim only

Explanation: Primary wall = cores. RAM only mattered when I duplicated indexes. Disk mattered for cold starts, not warm HNSW. NIC was free on localhost — which is why the next section exists. No swap means you size honestly or you crash honestly.

Why 8 vCPUs capped me (back-of-envelope)

At Go+Qdrant sustainable ~1640 QPS, mean service time under load at c=96 was ~58 ms waiting in the system (queueing), while unloaded p50 was ~6 ms.

Little’s law:

L \approx \lambda \cdot W

At λ = 1640 QPS, W ≈ 0.058 s → L ≈ 95 requests in flight — matches the best concurrency step (c=96).

If each useful search needs roughly one “core-slice” of ANN work and Qdrant reported ~7× CPU under load:

\text\{QPS\}_\{\max\} \lesssim \frac\{\text\{busy cores\} \times 1000\}\{\text\{CPU-ms per query\}\}

With ~7 cores busy and ~1640 QPS → ~4.3 CPU-ms/query of Qdrant work (order-of-magnitude). Doubling cores (16 vCPU), if linear, suggests ~3k QPS class — not 10k — unless the ANN itself gets cheaper (ef, quant, smaller k).

Hardware lesson: buying RAM alone would not have moved the Go+Qdrant plateau; more cores (or more Qdrant replicas) would. Little’s law also explains why my best concurrency step and in-flight count lined up — the ramp was not random.

Memory math for this corpus

Object	Formula	Size
fp32 vectors	1e6 × 384 × 4	~1.46 GB
HNSW graph (hnswlib)	measured	~1.6 GB
Python HNSW process RSS	measured	~1.8 GB
×4 MP workers	4 × ~1.8	~7 GB
×8 MP workers	8 × ~1.8	~14 GB → pressure on 31 GB
Qdrant under load	measured	~0.5–0.65 GB
Query matrix (6980 × 384 × 4)	—	~10.7 MB (tiny)

Explanation: Raw vectors ~1.5 GB; loaded Python process ~1.8 GB; multiply by workers and you eat the machine. Qdrant’s ~0.6 GB under load is why the Go path feels so dense. Query matrix is irrelevant noise at ~11 MB. 1M fits once. It stops fitting when software duplicates the working set.

Disk / page cache effects I saw

Situation	Hardware effect	Symptom
First Go/Qdrant ramp	Cold page cache / cold connections	~342 QPS, p99 523 ms
After warm	Vectors/graph in RAM cache	~1300+ QPS @ c=8, p99 ~14 ms
Graph manager cold	Partition fault-in from NVMe	Early ~61 QPS, p99 ~270 ms
Embeddings / ingest	Sequential NVMe write + CPU	Long wall-clock; needed 50k checkpoints
No swap	Cannot hide RAM overcommit	Must size workers to RAM

Explanation: NVMe is fast, but random cold HNSW touches still spike p99. Warm-cache numbers and cold-cache numbers are different products. Steady-state capacity claims should say which one they are.

Cloud / hypervisor reality

Effect	How it shows up
CPU steal / throttling	p99 grows without code changes
Shared NIC	Bandwidth and PPS limits below “spec”
EBS vs local NVMe	Remote disks add ms to cold paths
Credit-based burstable SKUs	Great smoke tests, false capacity

Rule: treat one-box QPS as ±20–30% soft unless you pin dedicated CPU / local SSD. I measured on a cloud VM; noisy neighbors are real even when I did not isolate them in the tables.

Hardware × software interaction

If you only upgrade…	Likely QPS effect on this design
RAM 31 → 64 GB	Helps MP HNSW w=8+; little for Go+Qdrant plateau
8 → 16 vCPU	Best single-knob for Qdrant / HNSW
Faster NVMe	Helps cold start & on-disk vectors; small for warm HNSW
GPU	Helps embedding, not the `qidx` search path
10/25/100 GbE	Matters only when DB or clients are remote

Explanation: Match the upgrade to the wall. More RAM helps the Python MP copy problem. More cores help the Go/Qdrant plateau. GPU does not help qidx search. Fancy NICs do not help localhost benches.

17. When the DB is remote: network math

All published QPS here used localhost (searchd ↔ Qdrant on the same machine). Production often puts the vector DB on another host or AZ. Then the NIC joins the wall list.

Deployment shapes

A) What I measured (colocated)
   Client ──HTTP──► searchd ──gRPC──► Qdrant     (loopback, µs RTT)

B) Split tier (common)
   Client ──HTTP──► searchd ──gRPC/HTTP──► Qdrant (other host)
                      ▲                         ▲
                   NIC #1                    NIC #2

C) Clients far away
   Clients ──WAN/LB──► searchd ──private net──► Qdrant

Every extra hop adds RTT + serialization + bandwidth.

Bytes per search (measured / derived)

From autocannon on this API: ~2.04 MB/s at ~2988 req/s ⇒ response ≈ 683 bytes/req (top‑10 JSON hits).

Path	Payload	Approx size
HTTP request (`qidx` only)	`{"qidx":N,"k":10}`	~23–40 B
HTTP request (raw vector)	384 floats JSON	~2.3 KB
HTTP response (k=10)	hits JSON	~650–700 B
gRPC query vector	384 × fp32	1536 B raw (+ protobuf framing)
gRPC result	10 × (id + score)	~100–300 B + framing
Fair total HTTP (qidx)	req+resp	~700–750 B/query
HTTP with vector in body	req+resp	~3.0 KB/query
searchd↔Qdrant gRPC	vector + hits	~1.8–2.2 KB/query (order)

Explanation: With qidx, the client HTTP path is tiny (~700 B round trip). Send the full 384-d vector as JSON and you ~4× the bytes. searchd→Qdrant still has to ship the dense vector over gRPC (~2 KB class). Tiny JSON is why client bandwidth stays easy even at huge QPS; the gRPC hop is the one to size when the DB leaves the box.

Bandwidth formulas

Let $Q$ = queries/sec, $B$ = bytes per query (both directions on that link).

\text\{MB/s\} = \frac\{Q \times B\}\{10^6\}, \quad \text\{Gbit/s\} = \frac\{Q \times B \times 8\}\{10^9\}

Add ~20–40% overhead for TCP/TLS/HTTP2 framing, retries, and stats — or use a 1.5× safety factor for planning.

Bandwidth tables (planning)

Link 1 — Clients → searchd (HTTP, qidx style, ~720 B/query):

Target QPS	Payload MB/s	≈ Gbit/s raw	Plan with 1.5×
1,640 (Go SLO)	1.2	0.009	trivial (≪1 GbE)
2,140 (HNSW MP)	1.5	0.012	trivial
10,000	7.2	0.058	still fine on 1 GbE
50,000	36	0.29	comfortable on 1 GbE
100,000	72	0.58	still under 1 GbE for this tiny JSON

Explanation: At my measured QPS, client bandwidth is a rounding error. Even 100k QPS with qidx stays under 1 Gbit/s raw. Client NIC is not why 100k is hard — cores and ANN replicas are.

Link 1b — Clients send full vectors (~3.0 KB/query):

Target QPS	MB/s	≈ Gbit/s	Plan 1.5×
1,640	4.9	0.039	fine
10,000	30	0.24	fine
100,000	300	2.4	need ≥5–10 GbE or compress/binary

Explanation: Same QPS targets, but vectors in the HTTP body change the story at the high end. 100k QPS wants multi-Gbit planning. Prefer id/qidx lookup or binary protocols if you are chasing that scale.

Link 2 — searchd → remote Qdrant (gRPC ~2 KB/query):

Target QPS	MB/s	≈ Gbit/s	Plan 1.5×	Notes
1,640	3.3	0.026	~0.04	same-AZ 1 GbE OK
10,000	20	0.16	~0.24	1 GbE OK if not shared heavily
50,000	100	0.80	~1.2	≥10 GbE safer
100,000	200	1.6	~2.4	10 GbE minimum; 25 GbE comfortable

Explanation: This is the link that starts to matter when you split tiers. My 1640 QPS is still fine on 1 GbE same-AZ. At 50–100k you want 10 GbE class and pooled gRPC, not hope.

Latency budget if Qdrant is off-box

Unloaded colocated search p50 ≈ 6 ms. That budget is mostly ANN.

Path	Typical added RTT	Effect on p50 / p99
Loopback	~0.01–0.05 ms	Ignorable
Same AZ, same VPC	~0.2–0.5 ms	Small
Cross-AZ	~1–2 ms	Eats SLO headroom
Cross-region	~20–80+ ms	Breaks p99≤100ms easily under load

Explanation: Under load my p99 already lives near 40–100 ms. Adding cross-region RTT on every search is incompatible with a 100 ms p99 SLO unless you cache aggressively or relax the SLO. Same-AZ is the practical choice.

Rough capacity derate if you add $d$ ms fixed RTT:

QPS_\{\text\{remote\}\} \approx QPS_\{\text\{local\}\} \times \frac\{W_\{\text\{local\}\}\}\{W_\{\text\{local\}\} + d/1000\}

Example: $W=0.006$ s, $d=2$ ms → factor $6/8=0.75$ → ~1640 → ~1230 QPS before even counting bandwidth.

Packets/sec and connection mechanics

Item	At 100k QPS (order)
HTTP requests/sec	100k
If short-lived TCP/request	huge SYN rate — don’t; use keep-alive / HTTP2 / gRPC streams
Concurrent connections	For c≈ concurrency; at scale use LB with pooling
gRPC to Qdrant	Multiplex many RPCs on few connections (required)

Explanation: Bandwidth is not the only NIC limit — PPS and connection churn kill you first if you open a new TCP per request at 100k QPS. Keep-alive / HTTP2 / gRPC multiplexing are not optional at that scale.

Ingest / index replication bandwidth (often forgotten)

Serving QPS is small JSON. Building and syncing indexes is the heavy network path.

Transfer	Size	Time on 1 GbE (~100 MB/s useful)	Time on 10 GbE
fp32 vectors 1M	~1.5 GB	~15–20 s	~2 s
HNSW index file	~1.6 GB	~20 s	~2 s
Qdrant snapshot / full reindex ship	few GB	minutes	tens of seconds
30M vectors fp32	~44 GB	~7–10 min	~1 min

Explanation: Teams provision NICs for query traffic (easy) and then fail on snapshot restore / blue-green index swap. Query path can be fine on 1 GbE while deploys time out on the same link.

If DB is outside: recommended network design

QPS class	Client→API	API→Qdrant	Placement
≤5k	1 GbE	1 GbE	Same AZ
5–20k	1–10 GbE	10 GbE	Same AZ, pooled gRPC
20–100k	10 GbE + LB	10–25 GbE, sharded/replicated Qdrant	Same AZ; avoid cross-region
+ embeddings	often dominates	binary protobuf, batch	GPU tier separate

Explanation: This is planning guidance from the math above, not something I load-tested at 100k. Stay same-AZ for the p99 SLO; scale the API→DB link earlier than the client link when using qidx.

Did network limit my experiments?

Link	In my runs	Limit?
Client → server	localhost / same host	No
searchd → Qdrant	localhost gRPC	No
Disk	local NVMe	Cold path only

Explanation: So every wall I hit was CPU / GIL / RAM copies, not NIC. The tables above are for the next failure mode once you split tiers or chase 100k QPS.

18. Deeper mechanics I needed to build this

Beyond “call HNSW,” the system required a stack of mechanics. These are the non-optional pieces that made experiments trustworthy.

Data & embedding mechanics

Mechanic	Why required	What I did
Corpus subsetting	Full MS MARCO too big for first gates	1M passage slice
Consistent embedding space	Mixed dims break indexes	BGE-small 384 everywhere
L2 normalize	Cosine ≡ IP	Normalize at embed time
Checkpointed embedding	CPU embeds take hours; crashes lose work	50k flush + manifest
Separate query matrix	Load tests must not call the embedder	`queries.f32.npy` + `qidx`
Ground-truth flat index	Recall@k needs exact neighbor lists	FAISS IndexFlatIP k=100

Explanation: Without checkpoints I lose hours of embedding. Without a separate query matrix, every loadtest secretly includes the embedder. Without flat GT, recall gates are not trustworthy. This is the boring foundation that makes every later table legitimate.

Index mechanics

Mechanic	Why	Notes
HNSW params (M, ef_construction, ef_search)	Recall/latency trade	M=16, ef_c=200, ef_s=64 default
Persist index (`hnsw.bin` + meta)	Don’t rebuild every run	~1.6 GB
Qdrant collection lifecycle	Green status before load	Wait for `indexed_vectors_count`
Optional sharding / on_disk flags	RAM vs QPS experiments	4-shard RAM coll. hurt here
Partition centroids + nprobe	Stage 4	Quality OK, QPS bad on 1M
LRU graph manager	Stage 5 RAM budget	Cold-start p99 hazard

Explanation: Indexes are not “build once and forget.” Persist the index files, wait for Qdrant to finish indexing, and treat shard/on_disk flags as experiments — they can regress.

Serving mechanics

Mechanic	Python	Go
HTTP JSON API	`ThreadingHTTPServer`	`net/http`
`qidx` resolution	numpy row	`vectors.Row` subslice
Concurrency	threads / processes	goroutines
Multi-process accept	`SO_REUSEPORT`	same (`unix.SO_REUSEPORT`)
Backend protocol	in-proc hnswlib or HTTP Qdrant client	gRPC Qdrant
Timeouts	ad hoc	`context.WithTimeout` per search
Stats	locked lists + psutil	atomics + `runtime.MemStats`

Explanation: Same external API (qidx, k), different concurrency tax and backend protocol. Go’s per-request contexts and cheap goroutines are part of why the front door stopped being the story.

Load-test mechanics (why numbers are believable)

Mechanic	Purpose
Concurrency ramp	Find SLO knee, not just max flood
Warmup step	Kill cold-cache lies
p50/p95/p99	Tails matter more than mean
Error-rate cap	Don’t count timeout storms as QPS
Rotating `qidx`	Avoid single-key cache fantasy
Autocannon separate	Explicitly labeled optimistic
`fuser`/PID files	Avoid `pkill` collateral damage
Docker stats + RSS	Attribute CPU/RAM to the right process

Explanation: This harness is why I trust 1640 more than 2988. If your bench skips warmup, rotating queries, or error caps, you are measuring a different (usually prettier) system.

Correctness / gate mechanics

Gate	Threshold	Role
Flat recall@10	1.0	Harness trust
HNSW recall@10	≥ 0.90	Ship floor
Stage 4 nprobe sweep	pick min nprobe hitting floor	Cost control
Capacity SLO	p99≤100ms	“Sustainable” definition

Without gates, you can “win” QPS by destroying recall (PQ smoke: 0.68 recall).

OS mechanics I leaned on

OS feature	Use
`SO_REUSEPORT`	MP accept balancing
Page cache	Warm Qdrant / mmap behavior
No swap	Forces honest RAM sizing
`docker` bridge + port publish	Qdrant isolation
`taskset` / CPU affinity (optional, unused)	Would reduce jitter in harder benches

Explanation: A lot of systems performance here is just using the kernel correctly — reuseport for MP accept, page cache for warm ANN, docker for isolation, and refusing swap so memory math stays real.

Mechanics I did not build (but 100k QPS would need)

Missing piece	Why it matters at scale
Client-side / edge cache	Cut QPS to origin
Shared result cache (Redis/shm)	MP-safe hits
Admission control / queue limits	Prevent death spirals (`qdrant_high`)
Hedged requests / retries with budget	Tail latency
Service mesh mTLS offload	Save ANN CPU
Index blue/green + snapshot automation	Safe deploys
Multi-AZ failover with SLO math	Availability vs latency
Embed microservice + batching	Real user traffic
Autoscale on p99 not CPU avg	Match the actual wall

Explanation: This article is about finding walls on one box, not building a full production control plane. The missing list is what you would add after you know you are CPU-bound on ANN rather than GIL-bound on Python.

End-to-end path (mechanics annotated)

[embed offline] → passages.f32.npy ─► build HNSW / upsert Qdrant
[embed offline] → queries.f32.npy  ─► loadtest picks qidx
                                         │
loadtest ─HTTP JSON─► searchd ─gRPC vector─► Qdrant HNSW
                         │                     │
                    goroutines              Rust threads
                    keep-alive               segments/shards
                         │                     │
                    JSON tops-k ◄──────────── ids+scores
                         │
                   p99 / QPS recorded

Every arrow is a place hardware (CPU, RAM, NIC, disk) can become the wall. On this box, arrows on localhost meant the wall sat inside ANN CPU and Python process model.

19. Scaling check: 100k QPS

From measured ~1640 (Go+Qdrant) or ~2140 (HNSW MP):

Assumption	Math
Linear replicas of Go+Qdrant	100000/1640 ≈ 61 boxes like this
With 75% efficiency	~80 boxes / ~500–650 vCPU
HNSW MP style	~47–55 fat-RAM nodes
Client HTTP (qidx) NIC	~0.6 Gbit/s raw (~1 Gbit/s planned) — not the hard part
searchd→remote Qdrant	~1.6–2.4 Gbit/s → plan 10 GbE
Embedding at 100k QPS	Separate, usually larger (GPU) problem

Explanation: Divide target QPS by measured per-box QPS and you get tens of machines even before inefficiency. Client JSON bandwidth stays easy; remote Qdrant gRPC does not. Embeddings are a separate GPU problem I deliberately excluded. Not solvable by cache/partitions/quantize alone on one 8-vCPU host.

20. Final verdict

Algorithm for capacity truth: plain HNSW top‑k. No rerank in the numbers I trust.
Highest QPS on this box: Python in-RAM HNSW, 4 processes → ~2140 sustainable-class, ~7 GB RAM.
Best QPS/RAM serving I shipped: Go → Qdrant → ~1640 QPS at ~0.7 GB.
Python’s main enemy: the GIL for single-process serving; escape = processes, which hit a memory wall at w=8.
Go’s main win vs Qdrant: removing the Python proxy bottleneck (~4×), not inventing a new ANN.
Go’s ceiling here: Qdrant CPU, not goroutines or searchd RAM.
Hardware: cores first; RAM when copying indexes; disk for cold paths; NIC unused on localhost but mandatory in the math once Qdrant is remote.
Remote DB: tiny JSON keeps client bandwidth easy even at 100k QPS; gRPC to remote Qdrant wants 10 GbE class at that scale; cross-region RTT fights a 100 ms p99 SLO.
Partitions / graph manager / Python caches: useful for fitting big indexes, harmful for QPS on 1M.
Ship recommendation: Go searchd + Qdrant for efficient serving; keep Python HNSW MP notes as the “max QPS / max RAM” reference; design NICs/pooling before splitting tiers.

I did not set out to prove Go is magic. I set out to find which wall I was actually hitting. Half the story was the process model. The other half was realizing I had been looking at the wrong layer of the stack — until the numbers made that impossible to keep doing.

Setup in one line: 8 vCPU / 31 GB RAM, MS MARCO 1M × 384-d, HNSW, p99≤100 ms, Python serving vs Go searchd → Qdrant.

Written from measured runs on this box. Prefer fair loadtest numbers (rotating qidx, warm, p99 SLO) over fixed-body autocannon peaks.