S3 over RDMA — Protocol & Configuration Reference

Technical reference for S3 over RDMA (beta): the request/reply contract, the object-service wire protocol, and every knob on the gateway, service, and volume server.

Request and reply

A client that speaks the NVIDIA cuObject convention sends an ordinary signed S3 request with x-amz-rdma-token. The token describes the client’s DC target:

addr:size:rkey:lid:dctn:1:gid    (fixed-width hex fields)

• addr, size, rkey — base address, byte length, and remote key of the client’s registered memory region. size is 32-bit, so one transfer is at most 4 GiB − 1, and the token must cover the requested size.
• lid, dctn, gid — InfiniBand/RoCE addressing (dctn is the DC target number).
• The literal 1 is the format marker.

The token-only request takes the DC direct-push path: each volume server RDMA-writes its chunks into the client buffer. Requests that instead carry the Seaweed descriptor headers (x-amz-rdma-client-endpoint, x-amz-rdma-client-descriptor, x-rdma-info) take the older RC path.

Gateway (weed s3)

With neither transport set, token requests get 501 NotImplemented.

The wire contract is identical on both transports: one JSON request per line in, one RdmaObjectServiceWireResult per line out; bytes, status, and chunks_fail must be present and non-null. Each call is bounded at 30 s; the gateway opens one socket connection per request.

Object service (sw-rdma-object-service, --features mlx5-dc, Linux)

sw-rdma-object-service serve --socket /run/seaweed/rdma-object.sock \
  --volume-loader-store \
  --volume-dc-push-map 'VOLUME_HTTP_HOST:PORT=VOLUME_HOST:DC_CONTROL_PORT[;...]' \
  --client-sink

• In DC direct push the volume writes straight into the client buffer, so the client store is never used — the parser still requires exactly one; --client-sink is what the lab runs.
• --volume-dc-push-map maps each volume’s HTTP endpoint, as the filer reports it for a chunk, to that volume’s DC push control address. It requires --volume-loader-store. A chunk whose volume has no map entry is refused — never silently relayed.
• serve --socket keeps one resident process; --serve-stdio (one request per stdin line) and bare one-shot invocation remain available.
• For PUT, the service needs its own DC initiator: --client-dc-device DEVICE --client-dc-port PORT --client-dc-gid-index GID.

PUT (opt-in)

Off unless SEAWEED_RDMA_OBJECT_PUT=1 is set on the gateway. A cuObject PUT has an empty body; the length comes from the token’s size.

1. The gateway assigns one fid per 4 MiB chunk — the same assign a plain PUT makes — and sends each chunk with its upload JWT. It sets want_sha256 only when the request asks for a SHA-256 checksum.
2. The service reads the chunks in order from the client’s DC target with RDMA READs, computes the object MD5 (and SHA-256 when asked), and writes chunks to their volumes over RDMA (prepare, RDMA WRITE, commit) — up to 8 in flight.
3. The gateway verifies Content-MD5 / x-amz-checksum-sha256 and publishes with one CreateEntry — the same publication path as a plain PUT. A short or malformed transfer deletes the assigned fids and publishes nothing.

Measured (one client, 100 GbE, median of 5): 4 MiB 0.07 vs 0.10 GB/s plain (slower — one chunk, nothing overlaps); 16 MiB 0.23 vs 0.19; 64 MiB 0.52 vs 0.25 (~2×). The in-order MD5 is the limit.

On failure the service reports each chunk committed, not_committed, or unknown; the gateway deletes every assigned fid and logs cleanup UNKNOWN for chunks that might land a late commit as an orphan.

Volume server (weed-volume, built with rdma,rdma-dc, Linux)

The DC access key is the cuObject default 0xffeeddcc. Pinned memory stays bounded — each read registers only its own range under a budget derived from the host’s locked-memory limit (~70 MB per storage process in lab runs).

Access log

For any request that reached the object service, the S3 access log (s3.access) adds rdma_bytes and rdma_transfer_path (e.g. dc-direct, or unreported). Refused requests that never reached the service have neither field.

Security

• The gateway authorizes as for plain S3 — same identities, bucket policy, and ACLs. A refused request moves no bytes.
• Without [jwt.signing.read] the volume’s RDMA control ports do not authenticate — any host that reaches them can read a needle by file id. Put them on an isolated RDMA network.
• With [jwt.signing.read], every push read is checked like an HTTP read. The gateway sends each chunk’s per-fid read JWT; a missing, forged, expired, or wrong-fid token is refused before a byte is written. A SIGHUP key change applies to open connections.
• Mixed versions fail closed — a client sends a token only to a volume that advertises READ_AUTH; when a token is required and the volume doesn’t advertise it, the request is refused rather than sent unsigned. --rdma.allow-unauthenticated-reads skips all checks, for a trusted RDMA network only.
• EC shard RDMA reads carry no read JWT (mirroring the gRPC path) and rely on cluster network trust.
• RDMA writes have no extra protection without [jwt.signing] — same as unauthenticated HTTP writes.

Limits