S3 over RDMA

SeaweedFS Enterprise S3 over RDMA serves object data straight from the volume servers into your client’s memory over RDMA — the gateway never touches the payload. It speaks the NVIDIA cuObject convention: an ordinary, signed S3 GET carries an x-amz-rdma-token describing where the bytes should land, and the volume servers RDMA-write each chunk directly into that client buffer. Same endpoint, same SigV4 authentication, same bucket policy — the HTTP response carries only headers.

CONTROL PLANE — ordinary signed HTTP, unchanged S3 client + x-amz-rdma-token weed s3 auth · chunk map · tokens object service transfer coordinator filer meta DATA PLANE — RDMA WRITE, the gateway never sees a byte client buffer host or GPU memory one-sided RDMA WRITE · DC volume servers each pushes its own chunks HTTP response carries headers only — the bytes already arrived by RDMA
Control messages stay on HTTP; each volume server pushes its chunks straight into the client buffer, so many servers can fill one buffer in parallel.

Why it matters

Over HTTP, every byte of an object travels in a response body: storage server → gateway → client over TCP, with kernel copies on each hop. On a 100 GbE fabric that path runs out of CPU long before it runs out of network. S3 over RDMA removes the payload from that path entirely — and because each volume server pushes its own chunks, the transfers scale with the number of storage servers, not with one gateway’s CPU.

Measured on a two-host lab (ConnectX-5, 100 GbE RoCE v2, byte-verified):

  • A single 64 MiB GET: 5.47 GB/s vs 1.06 GB/s over HTTP — about 5×.
  • 16 GETs in flight: 11.4 GB/s, ~95% of the link — up to 7.8× with chunks spread across four storage servers.
  • Cold reads from NVMe still beat HTTP 3.5×; even a 1 MiB range is ~2×.

The full benchmark is in the S3 over RDMA post.

How it works

  • The request stays normal. The client sends a signed GET (or PUT, opt-in) with an x-amz-rdma-token header naming its registered buffer — the same convention NVIDIA cuObject uses, so libcuobjclient and the NIXL object plugin work as-is, into host or GPU memory. A stock S3 client sends no token and stays on plain HTTP.
  • The gateway builds the plan. weed s3 authenticates, checks bucket policy, resolves the object’s chunk map — and mints a short-lived per-file read token per chunk.
  • The object service coordinates. A resident sw-rdma-object-service asks each volume to push its chunk directly to the client’s buffer offset.
  • Each volume pushes over DC transport. The volume checks the read token, registers only the range being read, and RDMA-writes the bytes into the client buffer — many volumes in parallel.
  • The HTTP reply is only headers — 200/206, x-amz-rdma-reply, x-amz-rdma-bytes-transferred — plus the error path, which never silently falls back to an HTTP body.

When to use it

  • GPU training and inference loading checkpoints, datasets, and weights straight into host or GPU memory — verified with NVIDIA’s libcuobjclient and NIXL’s object plugin.
  • HPC and analytics on RoCE v2 / InfiniBand fabrics (Mellanox ConnectX-5 or newer) where per-request CPU cost is the ceiling.
  • Any workload reading large objects where the gateway’s HTTP payload path — not the wire — is the bottleneck.

Turning it on (beta)

Three pieces, all on Linux hosts with DC-capable RDMA NICs:

1. RDMA-enabled volume servers — Rust weed-volume built with rdma,rdma-dc:

SWFS_RDMA_DC_DEVICE=rocep1s0 SWFS_RDMA_DC_PORT=1 SWFS_RDMA_DC_GID_INDEX=3 \
SWFS_RDMA_DC_PUSH_CONTROL_PORT=18530 \
  weed-volume --dir /data/vol --max 8 --ip 10.0.0.3 --port 8080 \
    --mserver 10.0.0.3:9333 --rdma.enabled --rdma.ip 10.0.0.3 --rdma.port 18516

2. The resident object service on the gateway host:

sw-rdma-object-service serve --socket /run/seaweed/rdma-object.sock \
  --volume-loader-store --client-sink \
  --volume-dc-push-map "10.0.0.3:8080=10.0.0.3:18530"

3. The S3 gateway pointed at that socket:

SEAWEED_RDMA_OBJECT_SOCKET=/run/seaweed/rdma-object.sock \
  weed s3 -filer=10.0.0.3:8888 -ip.bind=10.0.0.3 -port=8333 -config=/etc/seaweed/s3.json

To also serve RDMA PUT, add SEAWEED_RDMA_OBJECT_PUT=1 on the gateway and the --client-dc-* flags on the service. For security on a shared fabric, configure [jwt.signing.read] so every push read carries a per-file token; otherwise keep the RDMA control ports on an isolated network.

Current limits

S3 over RDMA is in beta — these stay on plain HTTP today:

  • Supported: GET of a whole object or a single range (200/206). Multi-range and suffix ranges are refused.
  • PUT is opt-in and currently slower than plain HTTP at 4 MiB (one chunk); faster at 16 MiB+.
  • SSE-encrypted objects, inline content, compressed chunks, and sparse objects are refused on the RDMA path.
  • PUT refuses versioned and Object Lock buckets, SSE, zip extraction, and non-SHA-256 checksums; multipart is not covered.
  • One transfer is capped at 4 GiB − 1 by the token’s 32-bit size field.
  • Linux + rdma-core with DC transport (mlx5, ConnectX-5+) only.

For the wire format, headers, service flags, and the security model, see the S3 over RDMA reference.