Configuring Lakekeeper
ColdFront needs three services, each configured once per deployment:
- PostgreSQL with the pg_duckdb and
coldfrontextensions. - Lakekeeper, the Iceberg REST catalog.
- An object store, such as SeaweedFS, MinIO, AWS S3, GCS, or Azure ADLS Gen2.
This guide starts and bootstraps Lakekeeper, then installs and configures the extensions in PostgreSQL. Complete this step before you create any table.
Lakekeeper is a separate upstream project, not part of the ColdFront build. Perform the steps in the section that matches how you install Lakekeeper: with packages, on bare metal, or on Docker.
Installing and Configuring Lakekeeper with Packages
The pgedge-lakekeeper package installs the lakekeeper binary, a
lakekeeper system user, and a lakekeeper systemd service, but it does not
configure, enable, or start the service. Lakekeeper stores its catalog in a
PostgreSQL 15 or later database and reads its settings from
/etc/lakekeeper/lakekeeper.env.
Use the following steps to install, configure, and start Lakekeeper:
-
If you did not install the
pgedge-lakekeeperpackage with the ColdFront packages, install it now from the pgEdge repository.On RHEL, Rocky Linux, or AlmaLinux:
sudo dnf install -y pgedge-lakekeeperOn Ubuntu or Debian:
sudo apt install -y pgedge-lakekeeper -
Connect as a PostgreSQL superuser, such as
postgres(for example, withsudo -u postgres psql), and create a role and a database for the catalog. Replace<password>with a generated password, such as the output ofopenssl rand -hex 32; a hex value needs no escaping in the connection string in step 3:CREATE ROLE lakekeeper LOGIN PASSWORD '<password>'; CREATE DATABASE lakekeeper OWNER lakekeeper;Hint
The migration in step 4 creates the
uuid-ossp,pgcrypto,pg_trgm,btree_gin, andbtree_gistextensions. Either allow thelakekeeperrole to runCREATE EXTENSION, or create the extensions as a superuser before you run the migration. -
In
/etc/lakekeeper/lakekeeper.env, set the catalog database's connection string and an encryption key:LAKEKEEPER__PG_DATABASE_URL_WRITE=postgresql://lakekeeper:<password>@127.0.0.1:5432/lakekeeper LAKEKEEPER__PG_ENCRYPTION_KEY=<encryption-key>- Replace
<password>with the password from step 2, and adjust the host and port if the catalog database is on another server. - Replace
<encryption-key>with a random secret, such as the output ofopenssl rand -base64 32. Lakekeeper encrypts stored credentials with the key, so keep the key stable and backed up. Every node that shares the catalog needs the same key.
- Replace
-
Run the one-time database migration as the
lakekeeperuser. The command reads the environment file as that user, because the file is not readable by other accounts:sudo -u lakekeeper sh -c \ 'set -a; . /etc/lakekeeper/lakekeeper.env; set +a; exec /usr/bin/lakekeeper migrate' -
Secure the listener before you start the service. By default, Lakekeeper listens on port 8181 on every address and, with no authorization backend, accepts every request. Do not start it that way:
-
If every client runs on the same host, set the following in
/etc/lakekeeper/lakekeeper.env:LAKEKEEPER__BIND_IP=127.0.0.1 -
If clients connect from other hosts, configure authentication and an enforcing authorization backend, and limit access to port 8181 with a firewall.
-
-
Enable and start the service:
sudo systemctl enable --now lakekeeper
Then, bootstrap Lakekeeper as described in Bootstrapping Lakekeeper.
Installing and Configuring Lakekeeper on Bare Metal
Lakekeeper ships a single prebuilt binary that requires glibc 2.34 or later. Use the following steps to install the binary and its service files:
-
Download the release for your architecture and install the binary:
ARCH=$(uname -m) # x86_64 or aarch64 LAKEKEEPER_VERSION=0.13.1 curl -LO "https://github.com/lakekeeper/lakekeeper/releases/download/v${LAKEKEEPER_VERSION}/lakekeeper-${ARCH}-unknown-linux-gnu.tar.gz" tar xzf "lakekeeper-${ARCH}-unknown-linux-gnu.tar.gz" sudo install -m 0755 lakekeeper /usr/bin/lakekeeper -
Create the system user that runs Lakekeeper:
sudo groupadd -r lakekeeper sudo useradd -r -g lakekeeper -d /var/lib/lakekeeper -s /sbin/nologin \ -c "Lakekeeper Iceberg REST Catalog" lakekeeper -
From your ColdFront repository clone, copy the environment file and the systemd unit that the package uses. The environment file is readable only by root and the
lakekeepergroup, as the package installs it:sudo install -D -m 0640 -g lakekeeper packaging/lakekeeper/common/lakekeeper.env \ /etc/lakekeeper/lakekeeper.env sudo install -m 0644 packaging/lakekeeper/common/lakekeeper.service \ /etc/systemd/system/lakekeeper.service sudo systemctl daemon-reload -
Connect as a PostgreSQL superuser, such as
postgres(for example, withsudo -u postgres psql), and create a role and a database for the catalog. Replace<password>with a generated password, such as the output ofopenssl rand -hex 32; a hex value needs no escaping in the connection string in step 5:CREATE ROLE lakekeeper LOGIN PASSWORD '<password>'; CREATE DATABASE lakekeeper OWNER lakekeeper;Hint
The migration in step 6 creates the
uuid-ossp,pgcrypto,pg_trgm,btree_gin, andbtree_gistextensions. Either allow thelakekeeperrole to runCREATE EXTENSION, or create the extensions as a superuser before you run the migration. -
In
/etc/lakekeeper/lakekeeper.env, set the catalog database's connection string and an encryption key:LAKEKEEPER__PG_DATABASE_URL_WRITE=postgresql://lakekeeper:<password>@127.0.0.1:5432/lakekeeper LAKEKEEPER__PG_ENCRYPTION_KEY=<encryption-key>- Replace
<password>with the password from step 4, and adjust the host and port if the catalog database is on another server. - Replace
<encryption-key>with a random secret, such as the output ofopenssl rand -base64 32. Lakekeeper encrypts stored credentials with the key, so keep the key stable and backed up. Every node that shares the catalog needs the same key.
- Replace
-
Run the one-time database migration as the
lakekeeperuser. The command reads the environment file as that user, because the file is not readable by other accounts:sudo -u lakekeeper sh -c \ 'set -a; . /etc/lakekeeper/lakekeeper.env; set +a; exec /usr/bin/lakekeeper migrate' -
Secure the listener before you start the service. By default, Lakekeeper listens on port 8181 on every address and, with no authorization backend, accepts every request. Do not start it that way:
-
If every client runs on the same host, set the following in
/etc/lakekeeper/lakekeeper.env:LAKEKEEPER__BIND_IP=127.0.0.1 -
If clients connect from other hosts, configure authentication and an enforcing authorization backend, and limit access to port 8181 with a firewall.
-
-
Enable and start the service:
sudo systemctl enable --now lakekeeper
Then, bootstrap Lakekeeper as described in Bootstrapping Lakekeeper.
Installing and Configuring Lakekeeper on Docker
Warning
The Docker stack is for development and testing on one host: PostgreSQL
trusts every connection, and Lakekeeper runs without authentication.
Before you let other hosts connect, configure authentication and an
enforcing authorization backend for Lakekeeper, require passwords in
PostgreSQL's pg_hba.conf, and limit access to the ports with a firewall.
The repository's Docker Compose stack runs Lakekeeper alongside PostgreSQL and the SeaweedFS object store. The stack creates Lakekeeper's catalog database, runs the migration, and starts the service, so Lakekeeper needs no further installation or configuration.
If you started the stack while building ColdFront on Docker,
Lakekeeper is already running at localhost:8181, and you can continue with
Bootstrapping Lakekeeper. Otherwise, start the stack
from your ColdFront repository clone. The command publishes each port on the
loopback address only, so other hosts cannot connect:
COLDFRONT_PG_PORT=127.0.0.1:5432 COLDFRONT_LK_PORT=127.0.0.1:8181 \
COLDFRONT_S3_PORT=127.0.0.1:8333 \
docker compose --profile local-store up -d --build
Then, continue with Bootstrapping Lakekeeper.
When the stack runs the SeaweedFS store, which it does only with the
local-store profile, use http://seaweedfs:8333 as the store endpoint when
you bootstrap. For any other store, see
Configuring your Object Store.
Bootstrapping Lakekeeper
Use the following steps to bootstrap Lakekeeper and, for the SeaweedFS store, create the warehouse and the Iceberg namespace. For any other store, perform step 1 only, then create the warehouse, the namespace, and the credential as described in your backend's section of Configuring your Object Store:
-
Bootstrap Lakekeeper:
curl -X POST http://localhost:8181/management/v1/bootstrap \ -H "Content-Type: application/json" -d '{"accept-terms-of-use":true}' -
For the SeaweedFS store, create the warehouse and save its ID in
WID. Replace<store-endpoint>with the address Lakekeeper uses to reach your store:http://seaweedfs:8333for the SeaweedFS store in the Docker stack.http://localhost:8333for SeaweedFS running on the same host.
WID=$(curl -s -X POST http://localhost:8181/management/v1/warehouse \ -H "Content-Type: application/json" -d '{ "warehouse-name": "wh", "storage-profile": { "type": "s3", "bucket": "iceberg", "region": "us-east-1", "endpoint": "<store-endpoint>", "path-style-access": true, "flavor": "s3-compat", "sts-enabled": false, "remote-signing-enabled": false }, "storage-credential": { "type": "s3", "credential-type": "access-key", "aws-access-key-id": "admin", "aws-secret-access-key": "adminsecret" } }' \ | grep -oE '"warehouse-id":"[^"]+"' | cut -d'"' -f4) -
For the SeaweedFS store, create the
publicnamespace in the new warehouse. Decoupled mode needs the namespace before you create a table; in tiered mode, the archiver creates it itself.if [ -z "$WID" ]; then echo "No warehouse ID - check that the warehouse create succeeded." else curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ -H "Content-Type: application/json" -d '{"namespace":["public"]}' fi
Hint
If you run these commands again after the warehouse exists, the create
request fails and WID is empty. Set WID to the warehouse-id of the
wh warehouse in the output of
curl -s http://localhost:8181/management/v1/warehouse, then rerun the
namespace request.
Configuring PostgreSQL for Lakekeeper
After bootstrapping Lakekeeper, install the extensions and set the cold-tier
credential in each database that uses ColdFront. If you use
coldfront.local_pg_dsn, described below, set it before you set the
credential. set_storage_secret() installs the DuckDB postgres extension
that the setting needs, but only when the setting is already present.
Install the extensions:
CREATE EXTENSION IF NOT EXISTS pg_duckdb;
CREATE EXTENSION IF NOT EXISTS coldfront;
For the SeaweedFS store, set the credential. Replace <store-endpoint> with
the store's address as PostgreSQL reaches it, without the http:// prefix:
seaweedfs:8333for the SeaweedFS store in the Docker stack.localhost:8333for SeaweedFS running on the same host.
SELECT coldfront.set_storage_secret('admin', 'adminsecret', '<store-endpoint>');
For any other store, set the credential as described in your backend's section of Configuring your Object Store.
set_storage_secret() stores the secret in the coldfront.storage_secret
table and creates a DuckDB persistent secret from it, which loads when DuckDB
starts. No per-session setup is needed: ColdFront attaches the Iceberg catalog
ice on the first query that reads or writes a tiered or decoupled table.
Hint
pg_dump does not include the coldfront.storage_secret table. In a
Spock mesh, the table replicates by value after
coldfront.ensure_replicated()
adds it to the default replication set during the one-time mesh setup.
The arguments after the secret are optional:
| Argument | Description | Default |
|---|---|---|
p_endpoint |
The store's address, without the http:// prefix. Omit it for AWS S3. |
NULL |
p_region |
The store's region. For AWS S3, set it to the bucket's Region. | 'us-east-1' |
p_url_style |
'path' or 'vhost'. Applies only when p_endpoint is set. |
'path' |
p_use_ssl |
Whether to connect over TLS. Set it to true for a TLS endpoint such as GCS. Applies only when p_endpoint is set. |
false |
The image writes the server settings ColdFront needs into postgresql.conf
when it initializes a new data directory. A server installed from packages or
built another way sets them itself, as the
package and
bare-metal sections of
installation.md show:
shared_preload_libraries = 'pg_duckdb,coldfront'loads both extensions at server start. coldfront refuses to load any other way:CREATE EXTENSION coldfrontfails on a server that does not preload it, with an error that names the setting.coldfront.warehouseandcoldfront.lakekeeper_endpointname the Lakekeeper warehouse and its catalog endpoint, which the Iceberg catalogiceattaches to. Only a superuser can set them, and while either is empty the catalog does not attach.coldfront.local_pg_dsnis the connection string DuckDB uses to read PostgreSQL tables from this server, which a decoupled table'sINSERT … SELECTfrom a PostgreSQL table needs, as does a cold write to a table with a clustered vector column. A tieredINSERTstreams its cold rows through this connection, and renders them row by row when the setting is empty. Only a superuser can set or readcoldfront.local_pg_dsn.
If your application connects as a role that is not a superuser, grant that
role access to ColdFront. Replace <role_name> with the name of the existing
role that your application connects as:
SELECT coldfront.grant_app_access('<role_name>');
coldfront.grant_app_access() grants access to the tables that ColdFront
manages when you run it, so run it again after you create or adopt a table,
or tier a table for the first time. The
Connecting to ColdFront
guide lists the grants.
The call fails when
duckdb.postgres_role is not set,
because only superusers can run DuckDB without that setting.