Standalone or Spinifex · Quick Start · S3 API Support · Architecture · Configuration · Spinifex Integration · Development · Roadmap

---

Predastore: Distributed, S3-compatible object storage

Predastore is a distributed object-storage system implementing commonly used Amazon S3 APIs. It combines Raft metadata, Reed–Solomon erasure coding, QUIC transport and append-only storage segments.

Predastore can run independently and provides the default object-storage backend for Spinifex.

Standalone or Spinifex?

Predastore runs on its own or as the object store of a Spinifex cluster, and one thing decides which: whether the config carries an [iam] table. Without it the gate resolves every access key from the [[auth]] entries in its own TOML, dials nothing, and depends on no other service. With it the gate also reads users, groups, roles, policies and session credentials out of Spinifex's NATS JetStream KV buckets, with the config-defined accounts still matched first.

Standalone is enough when a fixed set of service accounts and buckets, written into the config and distributed with it, is the whole access model. Reach for Spinifex when credentials have to change without a restart, or when more than one tenant shares the cluster.

	Standalone	Under Spinifex
Credentials	[[auth]] entries in the TOML	IAM access keys in NATS KV, plus the config accounts
Authorisation	none — every config account is fully trusted	IAM policies, evaluated per request
Buckets	[[bucket]] entries, plus whatever clients create	created through the API, owned per account
Multiple tenants	no — every key reaches every bucket	yes, enforced by account ownership
Users, groups, roles	no	IAM users, groups and roles
Temporary credentials	no	STS assumed-role sessions
Rotating a key	edit the TOML, restart the host	CreateAccessKey, no restart
Keys and certificates	predastore-keygen, distributed by you	issued on spx admin init, distributed on join
Cluster topology	you write the TOML	generated from the joined node set
External dependencies	none	NATS, and the Spinifex daemon that populates it

The second row is the one that decides most deployments. A config-defined account in [[auth]] bypasses both policy evaluation and the bucket-ownership check, so every key in the file reaches every bucket in the cluster no matter which account owns it. Standalone predastore cannot hand out a credential that is less than total, and that — rather than any missing S3 operation — is what a multi-tenant deployment needs Spinifex for.

Quick Start

Build

make build

Build through the Makefile rather than calling go build ./cmd/s3d yourself. The binary links a FIPS boot check that panics on startup unless it was compiled with GOFIPS140=v1.0.0, which the Makefile exports and a bare go build does not.

Run a Development Cluster

scripts/start.sh launches a cluster from one of the profiles in config/, generating the TLS keypair and master key under $PREDA_DIR (default /tmp/predastore) on first run:

./scripts/start.sh -w 1host     # one process, S3 on https://127.0.0.1:8443

Four profiles ship with the repo. They differ only in how many hosts the nodes are spread over, which is what decides the transport between them:

Profile	Hosts	RS	Inter-node transport	Needs sudo
1host	1	(1, 0)	in-process pipe only	no
3host	3 on 10.11.12.{1,2,3}	(2, 1)	QUIC between hosts, pipe within	yes
4host	4 on 10.11.12.{1..4}	(2, 1)	QUIC between hosts, pipe within	yes
5host	5 on 10.11.12.{1..5}	(3, 2)	QUIC between hosts, pipe within	yes

Every host in every profile sets admin_port = 9099, so /healthz and /readyz answer on the host's own address. That is plain HTTP with no authentication, which is safe on a loopback dev cluster and is why the flag default is 0 rather than a port.

The multi-host profiles put every host on one machine behind loopback aliases, so they need sudo to add those aliases and to install the generated certificate as a trust anchor. Each host answers S3 on its own address.

Stop, reset or benchmark the cluster with:

./scripts/stop.sh              # every cluster under $PREDA_DIR
./scripts/stop.sh 3host        # only that one, leaving any others running
./scripts/clean.sh
./scripts/bench.sh 3host
./scripts/bench.sh disk

Run a Host

./bin/s3d runs one host of a cluster: the nodes the config pins to it, the S3 gate among them. -host names the [[host]] to run, and everything else about the process — which nodes, which addresses, which ports — follows from that entry:

./bin/s3d \
  -config cluster.toml \
  -host 1 \
  -data-dir /var/lib/predastore \
  -tls-cert /etc/predastore/server.pem \
  -tls-key /etc/predastore/server.key \
  -encryption-key /etc/predastore/master.key

Flag	Description
-config	Path to the configuration file (required)
-host	ID of the [[host]] this process runs (required)
-data-dir	On-disk root for this host's nodes; overrides data_dir
-encryption-key	Path to the 32-byte AES-256 key protecting data at rest; overrides encryption_key
-tls-cert, -tls-key	This host's TLS identity; overrides tls_cert / tls_key
-bind-addr	Listen address for this host's cluster traffic, without a port; overrides bind_addr
-gate-bind-addr	Listen address for the S3 API, without a port; overrides the gate node's bind_addr
-log-level	debug, info, warn or error

Each of these host-local settings may come from either the file or a flag, so the same configuration file can be deployed unchanged to every machine and the paths supplied per process. The S3 port is not a flag: it is the gate node's port.

A cluster whose [[host.node]] entries all sit under one [[host]] runs entirely in one process over the in-process pipe, with no inter-node socket and no certificate beyond the one the gate serves. That is a property of the config, not a launch mode.

The encryption key file must be exactly 32 raw bytes (no base64, no header) with mode 0600. Generate one with ( umask 0177 && openssl rand -out master.key 32 ). The same key must be supplied to every host in a cluster; rotating it is not currently supported (see Roadmap → envelope encryption).

Run as a systemd Service

make install lays down the binary, a unit, and the drop-ins that create the service's user and its directories. It installs files and performs nothing else, so it is usable from a package build and leaves the sequencing to you:

make build
sudo make install                                   # binary, unit, sysusers, tmpfiles, sysctl
sudo systemd-sysusers && sudo systemd-tmpfiles --create
sudo sysctl --system                                # required for multi-host

sudo cp /etc/predastore/predastore.toml.example /etc/predastore/predastore.toml
sudo $EDITOR /etc/predastore/predastore.toml        # replace every CHANGEME
sudo predastore-keygen /etc/predastore              # the at-rest key and a TLS identity

sudo systemctl daemon-reload && sudo systemctl enable --now predastore

DESTDIR and PREFIX are honoured, so a package build can stage the same target into a buildroot. An existing ./bin/s3d is installed as it stands rather than rebuilt, which is what lets the install run under sudo without go on root's PATH, so run make build first to pick up any source change.

The service runs as an unprivileged predastore user under a strict sandbox: /etc read-only, /var/lib/predastore the only writable path, no capabilities at all, and a syscall filter. Its host-local settings — which [[host]] it runs, where its data and keys live — come from /etc/predastore/predastore.env, which installs as predastore.env.example so an upgrade cannot overwrite an edited one. The shipped example config names those paths itself, which is what a single machine wants; a fleet distributing one identical TOML overrides them per host in the env file instead. The service logs at warn, rather than the info a hand-run s3d defaults to, on the grounds that a long-running service wants a quiet journal.

predastore-keygen is idempotent, and it refuses to mint a TLS identity for a config naming more than one host. Peers verify each other against the system trust store, so hosts that each generated their own keypair never elect a leader rather than failing outright. Generate one keypair, distribute it to every host, and install its CA — see Standalone TLS Trust.

Run in Docker

The image needs no arguments and no configuration file. On first start it generates a TLS identity and an at-rest key into its volume and serves a single-host RS(1, 0) cluster:

make docker-build
docker run -d -p 8443:8443 -p 9099:9099 -v preda-data:/var/lib/predastore predastore:dev

Two compose topologies ship under deploy/. The cluster profile is four hosts at RS(2, 1), each with a gate, a meta replica and a blob node; every gate answers for the whole cluster, so 8443–8446 are four ways into the same thing:

make compose-up                              # four hosts
make compose-up COMPOSE_FILE=deploy/compose.single.yml
make compose-down

make docker-smoke then checks the image itself — non-root, no private key in any layer — and drives aws-cli, rclone, restic and a registry:3 on top of the running cluster. It asserts response content rather than exit codes, and gates on scripts/smoke-baseline.txt so it fails on a regression rather than on the checks predastore does not pass yet. Re-record that file with make docker-smoke-baseline in the same change as the fix.

Environment

s3d itself reads no environment variables, so the entrypoint renders a configuration file from these and passes it as -config. Every one is optional.

Variable	Default	Description
PREDA_PEERS	127.0.0.1	Comma-separated host addresses in host-id order. More than one entry makes it a cluster
PREDA_HOST_ID	1	Which [[host]] this process runs
PREDA_HOST_ID_FROM_ORDINAL	0	Set to 1 to derive the host id from a StatefulSet pod ordinal
PREDA_BIND_ADDR	0.0.0.0	Cluster-plane listen address
PREDA_RS_DATA, PREDA_RS_PARITY	1/0 alone, 2/1 otherwise	Erasure-coding width
PREDA_BLOB_NODES	1	Blob nodes per host
PREDA_REGION	ap-southeast-2	Region
PREDA_S3_PORT, PREDA_META_PORT, PREDA_BLOB_PORT	8443, 6660, 9991	Node ports
PREDA_ADMIN_PORT	9099	/healthz and /readyz; 0 runs neither
PREDA_ACCESS_KEY_ID, PREDA_SECRET_ACCESS_KEY, PREDA_ACCOUNT_ID	the AWS example pair	Config-defined service account
PREDA_DATA	/var/lib/predastore	Data root
PREDA_TLS_CERT, PREDA_TLS_KEY	under $PREDA_DATA/tls/	TLS identity, generated if absent
PREDA_KEY	$PREDA_DATA/master.key	AES-256 key at rest, generated if absent
PREDA_CONFIG	—	Use this profile verbatim and render nothing
LOG_LEVEL	info	debug, info, warn or error

Two of these have constraints the container cannot paper over:

• A cluster is refused without explicit credentials. The AWS example pair is a single-host evaluation default, not something to run four hosts on.
• A cluster needs one TLS keypair shared by every host, because peers verify each other against the system trust store. Hosts that each generated their own identity do not fail loudly, they simply never elect a leader, so the entrypoint refuses to generate one when PREDA_PEERS names more than one address. deploy/docker/gen-certs.sh mints a shared local-development keypair; make compose-up runs it for you.

S3 API Support

Area	Operations
Buckets	CreateBucket, DeleteBucket, ListBuckets, HeadBucket
Objects	PutObject, GetObject, DeleteObject, HeadObject, ListObjects, ListObjectsV2
Multipart	CreateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload
Authentication	AWS Signature Version 4, presigned URLs

Multi-object delete (DeleteObjects) takes up to 1000 keys per request and reports the outcome of each one.

How far that support goes is measured rather than claimed. docs/S3-COMPATIBILITY.md holds the result of running ceph/s3-tests — the suite Ceph RGW, MinIO and Garage are all validated against — with the per-case manifest in scripts/s3-tests-baseline.txt. Read it before assuming an operation behaves the way S3 documents it.

AWS CLI Examples

Against the 1host dev profile. Its certificate is self-signed, so either install it as a trust anchor or pass --no-verify-ssl:

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=ap-southeast-2

# Create a bucket
aws --no-verify-ssl --endpoint-url https://127.0.0.1:8443/ s3 mb s3://my-bucket

# Upload a file
aws --no-verify-ssl --endpoint-url https://127.0.0.1:8443/ s3 cp ./file.txt s3://my-bucket/

# List bucket contents
aws --no-verify-ssl --endpoint-url https://127.0.0.1:8443/ s3 ls s3://my-bucket/

# Download a file
aws --no-verify-ssl --endpoint-url https://127.0.0.1:8443/ s3 cp s3://my-bucket/file.txt ./downloaded.txt

Architecture

A cluster is built from three node roles. s3d is one process running the roles a host is assigned, and any mix of them can share a host:

• gate — serves the S3 HTTP interface and Signature Version 4 authentication.
• meta — maintains strongly consistent metadata with HashiCorp Raft over a local embedded database.
• blob — holds erasure-coded shards in append-only segment files, sealed with AES-256-GCM.

Nodes sharing a host talk over an in-process pipe; nodes on different hosts talk over QUIC. Nothing in a deployment names a transport — it follows from where the config pins each node.

See docs/DESIGN.md for the design in full.

Capabilities

• Reed–Solomon erasure coding
• Raft-based metadata
• QUIC storage transport
• Append-only segment storage
• AES-256-GCM encryption at rest
• Consistent-hash placement
• Multipart uploads
• Single-binary deployment

Configuration

A cluster configuration is a TOML file describing two levels: [[host]] entries, which are the s3d processes owning an address and a data directory, and the [[host.node]] entries nested under them, which are the roles that host runs. The same file also carries the Reed-Solomon parameters, the config-defined buckets and the S3 credentials.

The file is meant to be identical on every machine, so the settings only the local machine cares about — its data directory, TLS identity and encryption key — may be left out and supplied as flags instead.

version = 1
region  = "ap-southeast-2"

[rs]
data   = 2    # data shards
parity = 1    # parity shards; data + parity must not exceed the blob node count
# degraded_writes = false   # accept a write once `data` shards are durable, so one
                            # node down does not refuse writes; repair closes the gap
# hinted_handoff  = false   # put a shard its owner refuses on the next node along the
                            # ring, so the stripe is complete when the write is acked

# [repair]                  # restore shards a node holds at a stale generation
# enabled          = false
# interval_seconds = 0      # zero defaults to 300

# One host = one s3d process, launched with `-host <id>`.
[[host]]
id   = 1
addr = "10.11.12.1"        # what other hosts dial; no port — nodes carry those
# bind_addr = "10.11.12.1" # where raft and blob listen; defaults to addr
# admin_port = 9099        # /healthz and /readyz on bind_addr; absent runs neither
# data_dir  = "/var/lib/predastore"   # absolute; -data-dir supplies it otherwise

  # A role this host runs. One of "gate", "blob" or "meta".
  [[host.node]]
  id   = 1
  role = "gate"            # the S3 endpoint; port is the S3 port
  port = 8443
  # bind_addr = "0.0.0.0"  # where S3 listens; gate only, defaults to the host's

  [[host.node]]
  id   = 2
  role = "meta"
  port = 6660

  [[host.node]]
  id   = 3
  role = "blob"
  port = 9991

# A config-defined service account. Every request one signs is authorised, on
# every bucket, so treat an entry here as a root credential.
[[auth]]
access_key_id     = "AKIAIOSFODNN7EXAMPLE"
secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
account_id        = "123456789012"   # required; stamped as owner on buckets this key creates

# A bucket that exists from startup rather than being created through the API.
[[bucket]]
name       = "artifacts"
region     = "ap-southeast-2"
account_id = "123456789012"          # required
# public   = true                    # serve GET and HEAD without a signature

Node ids are unique across the whole file; ports are unique within a host. A blob or meta node without its own data_dir derives one from the host's root and its node id, so separate disks are a per-node setting rather than a deployment layout.

A freshly mounted drive is owned by root, and s3d refuses to start while any local node's directory is not writable by the user it runs as, naming each node and path it cannot write. Hand each drive's directory to the service user before starting it, e.g. sudo chown predastore:predastore /var/lib/predastore/disk1 && sudo chmod 0700 /var/lib/predastore/disk1. Under the systemd unit /var/lib/predastore is the only writable path, so mount the drives below it rather than elsewhere, or add their paths to ReadWritePaths.

S3 is a public service and replication is not, so the two bind separately on a multi-homed machine: the host's bind_addr is where raft and blob traffic listens, and the gate's own bind_addr is where S3 does. Set the first to a private address and the second to 0.0.0.0 and the S3 endpoint answers on every interface while nothing reaches consensus or shard traffic from outside. Neither is required — a host that names only addr binds that one address for both, which is what a single-homed machine wants.

admin_port puts /healthz and /readyz on the host's bind_addr, alongside the cluster plane and never on the S3 port. /healthz answers 200 once the process is listening. /readyz answers 200 when this process can serve: a meta replica observes a leader, the meta plane answers a read, and at least data blob nodes answer one — whichever of those the roles on the host can be asked. It answers 503 otherwise, with a body naming which check failed and nothing else. A host that names no admin_port runs no admin listener.

A background sampler runs those checks on a fixed interval and /readyz only ever serves its last result, so answering a request costs nothing beyond reading it — hammering the endpoint does not add load to the meta plane or the blob nodes. The answer can lag reality by up to one sampling interval, and a request arriving before the first cycle completes gets 503 rather than an unfounded 200.

Write every node under one [[host]] and the cluster runs in one process over the in-process pipe. Spread them across hosts and each process is launched separately with its own -host id. Nothing else changes.

config/ holds the ready-made profiles — see Run a Development Cluster.

Accounts and Buckets

[[auth]] is the whole of standalone access control. An entry's account_id is mandatory and becomes the owner of any bucket that key creates, but it does not confine the key: a config-defined account skips both policy evaluation and the bucket-ownership check, so it reads and writes every bucket in the cluster whoever owns it. Declare as few as the deployment needs, and use Spinifex if credentials have to be scoped — see Standalone or Spinifex?.

[[bucket]] declares a bucket that exists from startup without a CreateBucket call, which is what a deployment with a fixed set of them wants. account_id is required and an entry missing it refuses the whole config, while an entry with an invalid name is dropped with a warning rather than being fatal. public = true serves GET and HEAD without a signature, and is worth setting only on a bucket genuinely meant to be world-readable.

ListBuckets answers from the metadata plane alone, so a bucket declared in the config does not appear in aws s3 ls until something creates it through the API. HeadBucket, GET and PUT all work against it in the meantime.

Standalone TLS Trust

When predastore is deployed by Spinifex, the cluster CA is installed into the host trust store automatically as part of node bootstrap — no manual action is required. Standalone operators must install the cluster CA into the host trust store before launching s3d, otherwise nodes cannot dial each other:

# Debian / Ubuntu
sudo cp cluster-ca.pem /usr/local/share/ca-certificates/predastore-cluster-ca.crt
sudo update-ca-certificates

# RHEL / Fedora / Amazon Linux
sudo cp cluster-ca.pem /etc/pki/ca-trust/source/anchors/predastore-cluster-ca.pem
sudo update-ca-trust

Socket Buffers for QUIC

A multi-host cluster needs the kernel's socket buffer limit raised, and this is a correctness requirement rather than tuning. Blob, meta and raft traffic all ride QUIC, quic-go asks the kernel for a 7 MiB receive buffer, and at the default of 208 KiB that request is silently clamped and the socket drops datagrams under load. The symptom is failed shard writes, with nothing anywhere reporting a misconfigured limit.

make install ships the setting as a drop-in, so a systemd deployment only has to apply it:

sudo sysctl --system        # applies deploy/systemd/99-predastore-net.conf

Set it directly otherwise, and persist it under /etc/sysctl.d so it survives a reboot:

sudo sysctl -w net.core.rmem_max=16777216
sudo sysctl -w net.core.wmem_max=16777216

A single-host cluster runs every node over the in-process pipe and opens no QUIC socket at all, so none of this applies to it.

Storage Backend

Distributed storage with erasure coding, Raft-consensus metadata, and QUIC transport. The data model:

Unit	Size	Description
Object	arbitrary	RS-encoded end-to-end into K data + M parity shards
Shard	⌈object_size / K⌉	Per-node RS slice; occupies a contiguous extent
Fragment	32 B header + 8 KiB body + 16 B GCM tag = 8240 B	On-disk unit; AES-256-GCM seals body with AAD bound to (objectHash, shardIndex, shardNum, fragNum)
Segment file	up to 4 GiB	Append-only container holding extents from one or more shards

See DESIGN.md §6 for the on-disk format: the segment layout, the fragment header field by field, and the AAD the GCM seal is bound to.

Spinifex Integration

Predastore is the default S3 storage provider for Spinifex, which turns the integration on by rendering one table into predastore's config:

[iam]
nats_url           = "nats://127.0.0.1:4222"
nats_token         = "..."
master_key_path    = "/etc/spinifex/master.key"    # absolute; decrypts the stored secrets
access_keys_bucket = "spinifex-iam-access-keys"

That table is the whole of it. With it present the gate opens Spinifex's JetStream KV buckets directly — there is no RPC and no callback into the control plane — and resolves an access key to the user, groups, roles and policies attached to it, decrypting each stored secret with the shared master key. A key prefixed ASIA is resolved as an STS session instead. Config-defined [[auth]] accounts are still matched first, so the service credentials in the file keep working whatever IAM holds.

The connection retries indefinitely and the KV buckets are opened lazily, so predastore starts cleanly before the Spinifex daemon has created them and recovers on its own when NATS restarts. What the table buys is per-request authorisation: IAM users and roles, policy evaluation, temporary session credentials, and account ownership that genuinely separates tenants. Without it every credential is a root credential — see Standalone or Spinifex?.

Spinifex also leans on predastore as the store behind several of its own services, all of it ordinary S3 traffic through the same gate:

• EC2 AMI images — machine images for VM launches
• EBS volume snapshots — via Viperblock, which uses Predastore as its S3-compatible backend
• User data — cloud-init configurations and system artifacts

Development

make build            # Build the s3d binary
make certs            # Generate the dev TLS certs the integration tests serve
make test             # Run tests
make preflight        # Full CI checks (lint, govulncheck, coverage, integration)
make test-race        # Run tests under the race detector
make clean            # Clean build artifacts

make preflight must pass before committing; make fix auto-fixes what the linter can.

Conformance

./scripts/start.sh -w s3tests   # the only profile with the accounts and region the suite needs
make s3-tests                   # fails on a regression against scripts/s3-tests-baseline.txt
make s3-tests-strict            # fails on any failing case; red by design
make s3-tests-baseline          # re-record, in the same change as the fix

The suite is ceph/s3-tests, pinned to a commit and cloned on first run. It takes about fifteen minutes. See docs/S3-COMPATIBILITY.md for what the current numbers mean.

Roadmap

• S3 API core (buckets, objects, multipart)
• AWS Signature V4 authentication
• Distributed storage with Reed-Solomon erasure coding
• Raft-consensus metadata
• QUIC transport with connection pooling
• Consistent hash ring placement
• AES-256-GCM encryption at rest (single cluster-wide master key)
• Background segment compaction
• Envelope encryption (master key rotation, per-bucket / per-tenant keys)
• Gossip-based node discovery
• Automatic shard rebalancing
• Background read-repair
• Bucket versioning
• Lifecycle policies

Roadmap items describe direction and are not commitments to a release date.

Trademarks

Amazon Web Services, AWS and Amazon S3 are trademarks of Amazon.com, Inc. or its affiliates. Predastore is not affiliated with or endorsed by Amazon Web Services.

License

Predastore is licensed under the GNU Affero General Public License v3.0 (AGPLv3) license.