Deploy a Cluster

Build iggy-server and the iggy CLI, run a single development node, then bring up a three-node cluster on one machine. See Configuration for the full [cluster] reference.

Build from source

Build the server and CLI from the repository root:

cargo build --bin iggy-server --bin iggy

The binaries are:

target/debug/iggy-server
target/debug/iggy

The server uses Linux io_uring. Use a recent Linux kernel and a sufficient locked-memory limit.

Run one development node

The default core/server/config.toml has clustering disabled, so a plain run starts a single node:

cargo run --bin iggy-server -- --with-default-root-credentials

--with-default-root-credentials sets the root user to iggy/iggy unless IGGY_ROOT_USERNAME and IGGY_ROOT_PASSWORD are already exported. Only the first boot on an empty data directory reads these values.

When clustering is disabled, --replica-id 0 is still accepted. Any other id is rejected because it would have to match a cluster.nodes entry.

Run a three-node cluster

Every node loads the same configuration. Nodes are told apart only by --replica-id, which selects that node's own entry in the cluster.nodes roster.

Save this as /tmp/iggy-vsr.toml:

[cluster]
enabled = true
name = "iggy-vsr-dev"

[cluster.auth]
enabled = true
shared_secret = ""

[[cluster.nodes]]
name = "iggy-node-1"
ip = "127.0.0.1"
replica_id = 0
ports = { tcp = 8090, quic = 8080, http = 3000, websocket = 8093, tcp_replica = 9090 }

[[cluster.nodes]]
name = "iggy-node-2"
ip = "127.0.0.1"
replica_id = 1
ports = { tcp = 8091, quic = 8081, http = 3001, websocket = 8094, tcp_replica = 9091 }

[[cluster.nodes]]
name = "iggy-node-3"
ip = "127.0.0.1"
replica_id = 2
ports = { tcp = 8092, quic = 8082, http = 3002, websocket = 8095, tcp_replica = 9092 }

cluster.name is a permanent on-disk identity. It is hashed into the cluster id stamped into every metadata and partition superblock on first boot. Changing the name later makes the server refuse to start against the existing data, so renaming a cluster means starting from an empty data directory.

Export the settings shared by all three processes:

export IGGY_CONFIG_PATH=/tmp/iggy-vsr.toml
export IGGY_ROOT_USERNAME=iggy
export IGGY_ROOT_PASSWORD=iggy
export IGGY_CLUSTER_AUTH_SHARED_SECRET="replace-with-at-least-32-random-bytes"

Root credentials are mandatory once the cluster is enabled. The shared secret must be at least 32 bytes. See Security for how it's used and rotated.

Start each replica in a separate terminal. Use a different data path for every process:

# Replica 0
IGGY_SYSTEM_PATH=local_data/node-0 ./target/debug/iggy-server --replica-id 0

# Replica 1
IGGY_SYSTEM_PATH=local_data/node-1 ./target/debug/iggy-server --replica-id 1

# Replica 2
IGGY_SYSTEM_PATH=local_data/node-2 ./target/debug/iggy-server --replica-id 2

The exported settings must be present in all three terminals.

In cluster mode:

--replica-id is required and must match exactly one cluster.nodes entry
replica_id values must be unique and contiguous from 0
ports is the single source of listener ports: every enabled transport needs an explicit per-node port, otherwise the server refuses to start
tcp_replica carries replica-to-replica consensus traffic and is always required
ip must be a literal IP address. Use advertised_address when clients can't reach it (see Configuration)
use a different system.path for each process on the same host

Spanning multiple hosts

The roster ip is the address other replicas dial. It's not the bind interface for the client transports: tcp, quic, http, and websocket bind whatever their own address settings say, and the roster supplies only the port. The defaults listen on loopback, so a cluster spread across hosts needs each transport's address set to 0.0.0.0 or the routable interface. The server warns at startup when a bind cannot serve the advertised ip.

Containers

Every roster field can be provided through typed environment variables instead of TOML, which suits container deployments:

IGGY_CLUSTER_ENABLED=true
IGGY_CLUSTER_NAME=my-cluster
IGGY_CLUSTER_NODES_0_NAME=node-1
IGGY_CLUSTER_NODES_0_IP=172.28.0.101
IGGY_CLUSTER_NODES_0_REPLICA_ID=0
IGGY_CLUSTER_NODES_0_PORTS_TCP=8090
IGGY_CLUSTER_NODES_0_PORTS_QUIC=8080
IGGY_CLUSTER_NODES_0_PORTS_HTTP=3000
IGGY_CLUSTER_NODES_0_PORTS_WEBSOCKET=8093
IGGY_CLUSTER_NODES_0_PORTS_TCP_REPLICA=9090
IGGY_CLUSTER_NODES_1_NAME=node-2
IGGY_CLUSTER_NODES_1_IP=172.28.0.102
# ... and so on for every node and every enabled transport
IGGY_CLUSTER_AUTH_ENABLED=true
IGGY_CLUSTER_AUTH_SHARED_SECRET=replace-with-at-least-32-random-bytes
IGGY_ROOT_USERNAME=iggy
IGGY_ROOT_PASSWORD=iggy

Give each container a static IP and put that IP in the roster: cluster.nodes[*].ip does not accept hostnames or Docker service names. The repository ships a working reference in bdd/docker-compose.cluster.yml, a real two-node cluster used by the BDD suites.

Check cluster metadata

Ask any node for the cluster state with the CLI:

./target/debug/iggy \
  --tcp-server-address 127.0.0.1:8090 \
  -u iggy \
  -p iggy \
  cluster metadata

The response lists the cluster name, nodes, client endpoints, and current roles. cluster m is a shorthand for cluster metadata.

Reset a replica

iggy-server --fresh deletes the configured data directory before boot. In cluster mode this wipes that replica only: it rejoins the cluster and refills its state by transfer from the other replicas. Wiping a quorum of replicas at the same time destroys committed data, and a service unit that always passes --fresh re-transfers the whole dataset on every restart.

Run the cluster test suites

The cross-SDK BDD suites run against a real cluster started from bdd/docker-compose.cluster.yml:

./scripts/run-bdd-tests.sh rust
./scripts/run-bdd-tests.sh rust basic_messaging
./scripts/run-bdd-tests.sh rust leader_redirection

The first argument selects the SDK (rust, python, php, go, go-race, node, csharp, java, cpp, or all) and the optional second argument selects one feature (basic_messaging, leader_redirection, raw_command, or all). leader_redirection is the clustered feature. The Rust, Go, C#, and Java suites support it.

On this page