Distributed Vector Search

SeaweedFS serves the S3 Vectors API from its S3 gateway. In the open-source build, one weed-vector engine answers every query. SeaweedFS Enterprise adds a pool of search engines and spreads each query over a few of them. Each index lives on a small, fixed set of engines. A query goes to one replica group of that index, one request per engine, and the gateway merges their answers into the top-k. Fan-out follows the size of the index, not the size of the fleet, so adding engines adds capacity without widening every query.

The vectors stay where they are: Lance tables in SeaweedFS. Engines keep their share of an index warm in memory, but any engine can still read any part of it cold from S3. Placement decides where queries go, never where data lives.

Highlights

• One group per query. A query reaches at most W engines of one replica group, never the whole pool. max_fanout caps W at 32 by default.
• Sized from the index. W grows with the index: enough engines to hold it in memory, and enough that each engine scans at most max_scan_mb (64 MiB) of it per query. R, the number of groups, grows with load: one group per 200 queries/s by default, 2 to 16 groups.
• Exact results. Each engine scans its share in full and the gateway merges the partial top-k lists. Recall@10 was 1.000 in every benchmark run below, filtered queries included.
• Stable placement. Weighted rendezvous hashing places each index’s groups, in different zones where it can. When an engine joins or leaves, most slots stay on the engines that already have them warm. A new placement is make-before-break: queries stay on the old groups until the new owners report warm.
• Overload is handled, not queued. Engines admit searches through a bounded queue and refuse at once when it’s full. The gateway hedges slow requests to the other group and retries once on another owner, both under small budgets. Clients get ServiceUnavailableException (503), which AWS SDKs retry, rather than a query that waits forever.
• No new client, no new API. Same S3 Vectors endpoint, same calls. When no engine is serving, the gateway answers through the single engine.

Benchmarks

Measured with SeaweedFS’s S3 Vectors performance suite: 384-dimension vectors, cosine distance, top-10, and four engines with 1 GiB of memory each. Every process (master, volume server, filer, S3 gateway and engines) ran on one 16-core x86_64 host shared with other work. Each cell shows two runs. Recall@10 was 1.000 in every run, and no timed query fell back to the single engine.

The distributed column puts all four engines in one group (min_width = 4, min_replicas = 1), so every query splits four ways.

• Filtered queries are 3–5× faster at 50k vectors and 4–5× faster at 200k. At 200k, 316–380 ms dropped to 73–78 ms.
• Unfiltered queries are about 3× faster at 50k (20–21 ms against 57–66 ms) and about 1.4× faster at 200k.
• One client gets about twice the throughput at 50k.
• At 8 clients, throughput is about the same. Every process shared one host’s 16 cores, so CPU, not the engines, set the ceiling. Engines on separate hosts were not measured.

With default settings, the 50k index sized itself to W=2: two groups of two engines. In that session the query p50 was 24 and 41 ms against 47 ms on the single engine. The filtered p50 was 38 and 69 ms against 91 ms.

Where the speed comes from

Early builds of the distributed path ran at about a quarter of the single engine’s throughput. Two changes fixed that:

• Engines cache reads of table files. Lance files are written once, so engines keep what they read (--read-cache-mb, 512 MiB by default) instead of reading the index again from S3 on every query.
• The unindexed part of an index is spread over the group. S3 Vectors indexes have no ANN segments today, so all of their data is “tail”, scanned exactly. Each tail fragment is now placed like a segment, so each engine in a group of four scans about a quarter instead of one engine scanning it all.

Counters for the same 50k index and the same 40 queries:

With 8 clients, the early build’s p50 was 478–497 ms at 50k vectors and 2,540 ms at 200k. The current build answers in 114–121 ms and 422–426 ms on the same layout.

Before max_scan_mb, W grew only when an index outgrew one engine’s memory, so these indexes always got one engine per query. Latency follows the bytes each engine scans. For filtered queries, an engine scanning 19 MB answered in about 25 ms, 77 MB took 61–95 ms, and 307 MB took 249–290 ms. That is why W now also grows with the scan.

How it works

• Controller (in the master leader). Engines open a session to the leader and report which slots they have warm. From the index size and query rate that gateways report, the controller computes W and R for each index and places its groups. It is never on the query path, and a follower master redirects engines to the leader.
• Placement. Each index gets R groups of up to W engines. Each group is in one zone where possible, and engines are ranked by weighted rendezvous hashing (weight = engine memory). Inside a group, segments and tail fragments go to members by hash, under a load cap of 1.25× the group average. The gateway (Go) and the engines (Rust) compute the same mapping.
• Router (in every S3 gateway). It resolves the index version, lists its segments (cached; versions are immutable) and watches the placement. It picks a group, preferring its own zone and then the less busy one, and sends one SearchSegments call per engine with the query’s deadline. Then it merges the partial top-k lists.
• Engines (weed-vector-worker). They warm their assigned slots, search through a bounded admission queue shared fairly between indexes, and drain on SIGTERM so queries move off before the process stops.
• Writes. PutVectors, DeleteVectors and index management go to the single weed-vector, as before. Each write makes the gateway drop its cached version of that index.

Sizing: W and R

The controller sizes every index on every tick:

W_mem  = ceil(resident / (0.8 × engine budget))       # fit in memory
W_scan = ceil(resident / max_scan_mb)                 # bound each engine's scan
W      = max(W_mem, min(W_scan, eligible engines / min_replicas))   # clamped to [min_width, max_fanout]
R      = ceil(peak queries/s / qps_per_group)                       # clamped to [min_replicas, max_replicas]

The engine budget is the median --capacity-bytes of the engines. The memory width always wins. The scan width only grows while min_replicas full groups still fit, so the warm replica is never given up to make scans smaller. max_scan_mb is an even-split target: a single fragment is never split, so one fragment larger than the limit is still one engine’s scan.

With the defaults and 8 GiB engines:

R rises as soon as load needs it and falls only after the load has stayed lower for replica_cooldown (10 minutes).

Turning it on

Start with S3 Vectors already running: the gateway with its Lance catalog, and one weed-vector registered with the master. That engine keeps handling writes, and answers queries whenever no distributed engine is serving.

1. Start a weed-vector-worker on each search host. It registers with the masters and joins the controller:

weed-vector-worker --masters 10.0.0.1:9333,10.0.0.2:9333,10.0.0.3:9333 \
  --state-dir /var/lib/weed-vector-worker --zone dc1 \
  --storage-option aws_endpoint=http://10.0.0.5:8333 \
  --storage-option aws_region=us-east-1 --storage-option allow_http=true \
  --storage-option aws_access_key_id=<key> --storage-option aws_secret_access_key=<secret>

• --state-dir keeps the engine’s id, so a restarted engine gets its slots back.
• --zone is the failure domain: an index’s replica groups go to different zones.
• --capacity-bytes defaults to 75% of RAM (or of the cgroup limit), and is the engine’s placement weight.

2. Tune the controller (optional). The defaults live under [master.vector] in master.toml. Keep the file identical on every master, because whichever master leads applies its own copy:

[master.vector]
max_fanout = 32      # largest W
max_scan_mb = 64     # per-engine scan target; 0 sizes W by memory only
min_replicas = 2
max_replicas = 16
qps_per_group = 200

# per-index bounds, keyed by the index's table URI
[[master.vector.index]]
key = "s3://bucket/vector/index"
min_width = 4

3. Gateways need no change. A gateway uses its -dataCenter as its zone and prefers groups there.

Current limits

• Only QueryVectors is distributed. Writes and index management go through the single weed-vector, which has to stay running.
• A fragment is the smallest unit of work. One fragment larger than max_scan_mb is scanned by one engine.
• The benchmarks above ran on one shared host. Engines on separate hosts have not been measured yet.

For every setting, flag and default, see the Distributed Vector Search reference. For Lance tables on SeaweedFS, see S3 Table Buckets and Vector search with LanceDB.