Querying SeaweedFS Table Buckets with DuckDB

A SeaweedFS table bucket is an S3 bucket that doubles as an Apache Iceberg warehouse. SeaweedFS fronts it with a built-in Iceberg REST catalog, so any Iceberg engine — Spark, Trino, Flink, PyIceberg, and DuckDB — can create, load, and query tables. There’s no separate catalog service to stand up: no Hive Metastore, no Nessie, no AWS Glue.

This post walks through it end to end on a single machine, with nothing hand-waved: every command below was run against a local SeaweedFS Enterprise build, and every output block is a real capture from that session. By the end you’ll have started SeaweedFS, created and loaded an Iceberg table, and queried it from DuckDB — reading the Parquet straight out of the bucket.

What you’ll need

• The SeaweedFS Enterprise weed binary. Table buckets and the built-in Iceberg REST catalog are Enterprise features.
• DuckDB — a current release (this was run with DuckDB 1.5). Grab the CLI from duckdb.org/docs/installation.
• Python 3.9+ — only to create and load the sample table with PyIceberg. Any Iceberg writer would do; PyIceberg keeps the walkthrough self-contained.

Step 1 — Start SeaweedFS with a table bucket

First, define an S3 identity so the writer and DuckDB can authenticate. Save this as s3config.json:

{
  "identities": [
    {
      "name": "analyst",
      "credentials": [
        { "accessKey": "tutorialkey", "secretKey": "tutorialsecret" }
      ],
      "actions": ["Admin", "Read", "Write", "List", "Tagging"]
    }
  ]
}

Now launch everything with weed mini — the Enterprise all-in-one command that runs a master, volume server, filer, S3 gateway, and the Iceberg catalog in one process. The -tableBucket flag pre-creates the table bucket on startup:

weed mini -dir=./data -s3.config=s3config.json -tableBucket=analytics

Every component comes up, including the catalog:

  All enabled components are running and ready to use:

    Master UI:       http://localhost:9333
    Filer UI:        http://localhost:8888
    S3 Endpoint:     http://localhost:8333
    Iceberg Catalog: http://localhost:8181
    Admin UI:        http://localhost:23646

Two endpoints matter for the rest of this walkthrough: the S3 endpoint (:8333, where the Parquet lives) and the Iceberg Catalog (:8181, where engines find tables). Leave this running in its own terminal.

In production you’d run the components separately and point weed s3 at your identities. weed mini bundles them so you can try the whole thing on a laptop.

Step 2 — Create a table and load data

Install PyIceberg with the PyArrow extra:

python3 -m venv venv && source venv/bin/activate
pip install "pyiceberg[pyarrow]"

Save this as seed.py. It points PyIceberg at the catalog, authenticates with the same access key / secret (the catalog speaks OAuth2 — see How it works), creates a namespace and table, and appends six rows:

import datetime
import pyarrow as pa
from pyiceberg.catalog.rest import RestCatalog

catalog = RestCatalog(
    name="analytics",
    uri="http://localhost:8181",
    warehouse="s3://analytics",              # s3://<table-bucket>
    credential="tutorialkey:tutorialsecret",  # accessKey:secretKey
    **{
        "s3.endpoint": "http://localhost:8333",
        "s3.access-key-id": "tutorialkey",
        "s3.secret-access-key": "tutorialsecret",
        "s3.path-style-access": "true",
        "s3.region": "us-east-1",
    },
)

catalog.create_namespace_if_not_exists("sales")

data = pa.table({
    "id":       pa.array([1, 2, 3, 4, 5, 6], type=pa.int32()),
    "region":   ["us-east", "us-west", "eu-west", "us-east", "eu-west", "us-west"],
    "amount":   [120.50, 87.20, 210.00, 42.75, 133.10, 64.00],
    "order_ts": [
        datetime.datetime(2026, 7, 27,  9, 15),
        datetime.datetime(2026, 7, 27,  9, 20),
        datetime.datetime(2026, 7, 27,  9, 31),
        datetime.datetime(2026, 7, 27, 10,  5),
        datetime.datetime(2026, 7, 27, 10, 22),
        datetime.datetime(2026, 7, 27, 11,  2),
    ],
})

table = catalog.create_table_if_not_exists("sales.orders", schema=data.schema)
table.append(data)
print("tables:", catalog.list_tables("sales"))
print("rows appended:", data.num_rows)

Run it:

python seed.py

tables: [('sales', 'orders')]
rows appended: 6

This step uses PyIceberg, but sales.orders is a standard Iceberg table. Spark, Flink, Trino, or the SeaweedFS Admin UI would create and populate it exactly the same way — DuckDB in the next step doesn’t care who wrote it.

Step 3 — Query with DuckDB

Start the DuckDB CLI:

duckdb

Install and load the iceberg extension, then create two secrets — one tells DuckDB how to authenticate to the catalog (OAuth2), the other how to read Parquet from S3:

INSTALL iceberg;
LOAD iceberg;

CREATE SECRET catalog_auth (
    TYPE ICEBERG,
    CLIENT_ID 'tutorialkey',
    CLIENT_SECRET 'tutorialsecret',
    OAUTH2_SERVER_URI 'http://localhost:8181/v1/oauth/tokens'
);

CREATE SECRET s3_read (
    TYPE S3,
    KEY_ID 'tutorialkey',
    SECRET 'tutorialsecret',
    ENDPOINT 'localhost:8333',
    URL_STYLE 'path',
    USE_SSL false
);

Now attach the table bucket as a catalog. The first argument is the warehouse — s3://<table-bucket>:

ATTACH 's3://analytics' AS lake (
    TYPE ICEBERG,
    ENDPOINT 'http://localhost:8181',
    SECRET catalog_auth
);

The table is now visible as lake.sales.orders. Query it like any other table:

SELECT * FROM lake.sales.orders ORDER BY id;

┌───────┬─────────┬────────┬─────────────────────┐
│  id   │ region  │ amount │      order_ts       │
│ int32 │ varchar │ double │      timestamp      │
├───────┼─────────┼────────┼─────────────────────┤
│     1 │ us-east │  120.5 │ 2026-07-27 09:15:00 │
│     2 │ us-west │   87.2 │ 2026-07-27 09:20:00 │
│     3 │ eu-west │  210.0 │ 2026-07-27 09:31:00 │
│     4 │ us-east │  42.75 │ 2026-07-27 10:05:00 │
│     5 │ eu-west │  133.1 │ 2026-07-27 10:22:00 │
│     6 │ us-west │   64.0 │ 2026-07-27 11:02:00 │
└───────┴─────────┴────────┴─────────────────────┘

Aggregations, filters, and predicate pushdown all work — DuckDB uses the Iceberg metadata to read only the Parquet it needs:

SELECT region, count(*) AS orders, round(sum(amount), 2) AS revenue
FROM lake.sales.orders
GROUP BY region
ORDER BY revenue DESC;

┌─────────┬────────┬─────────┐
│ region  │ orders │ revenue │
│ varchar │ int64  │ double  │
├─────────┼────────┼─────────┤
│ eu-west │      2 │   343.1 │
│ us-east │      2 │  163.25 │
│ us-west │      2 │   151.2 │
└─────────┴────────┴─────────┘

That’s the whole loop: a table written by one engine, queried by another, with SeaweedFS storing both the metadata and the data.

Tip: put the LOAD, CREATE SECRET, and ATTACH statements in a file and start DuckDB with duckdb -init setup.sql so the catalog is ready every session.

Look under the hood

A table bucket is not a black box — it’s ordinary Parquet and Iceberg metadata sitting in your S3 bucket. With the s3_read secret still loaded, list it straight from DuckDB:

SELECT replace(file, 's3://analytics/', '') AS key
FROM glob('s3://analytics/**')
ORDER BY key;

┌────────────────────────────────────────────────────────────────────────────┐
│                                    key                                       │
├────────────────────────────────────────────────────────────────────────────┤
│ sales/orders/data/00000-0-....parquet                                        │
│ sales/orders/metadata/....-m0.avro                                           │
│ sales/orders/metadata/snap-....avro                                          │
│ sales/orders/metadata/v1.metadata.json                                       │
│ sales/orders/metadata/v2.metadata.json                                       │
└────────────────────────────────────────────────────────────────────────────┘

Because the data files are plain SeaweedFS objects, they inherit everything SeaweedFS does for the rest of your storage — replication, erasure coding, and cloud tiering.

How it works

• Warehouse addressing. Each table bucket is its own Iceberg warehouse, addressed as s3://<table-bucket>. DuckDB and PyIceberg send that to the catalog’s /v1/config endpoint, which returns the bucket as the routing prefix; subsequent calls are scoped to that bucket.
• Authentication. The catalog uses OAuth2 client-credentials. Your S3 access key is the client_id and the secret key is the client_secret; clients exchange them at /v1/oauth/tokens for a bearer token. That’s why the same credentials work for both the catalog and the S3 data reads.
• Two data paths. The catalog (:8181) hands out table metadata — schemas, snapshots, and the list of data files. The engine then reads those Parquet files directly from the S3 endpoint (:8333), so scans never bottleneck on the catalog.
• Who writes. DuckDB attaches the catalog read-only and is a superb query engine. Table creation and writes go through any Iceberg writer that commits to the catalog — PyIceberg here, but Spark, Flink, and Trino work identically.

Keeping tables fast

Streaming or frequent small writes leave Iceberg tables with many tiny Parquet files and stale snapshots that slow every scan. SeaweedFS compacts and cleans them automatically, on dedicated workers, with no separate compaction service to run — see Iceberg Table Maintenance. Point Spark, Trino, or DuckDB at the same catalog and they all read the compacted tables with no changes.