Viewstamped Replication

Apache Iggy is adding replicated clusters based on Viewstamped Replication Revisited (VSR). VSR keeps an ordered state machine consistent across replicas and elects a new primary when the current primary fails.

Read the VSR paper used by the project for the protocol specification.

VSR clustering is experimental and targets Iggy v1. It is close to feature completion, but it is not yet a production release.

Current implementation

The implementation lives in the next-generation core/server-ng server. It includes:

• normal operation with quorum commits
• view changes and deterministic primary selection
• metadata and partition replication
• persistent WAL recovery and snapshots
• replica rejoin with journal repair
• client sessions, request fencing, and duplicate detection
• replica authentication and optional TLS
• follower-aware routing in the Rust SDK
• deterministic simulation and Rust BDD coverage

The main components are:

Replication model

Iggy splits replication by namespace:

Each group can have a different primary. Metadata writes route through the metadata primary. Partition writes route through the primary for that partition.

Reads use the local replicated state where possible. Writes return after the required VSR commit or return a retryable error when the replica is changing view or catching up.

Failure handling

VSR uses three main flows:

1. Normal operation: the primary assigns an operation number, sends Prepare, and commits after a quorum sends PrepareOk.
2. View change: replicas exchange StartViewChange and DoViewChange, then the new primary sends StartView.
3. Recovery: a restarted or lagging replica requests the current view and repairs missing WAL ranges before serving current state.

A cluster of 2f + 1 replicas tolerates f unavailable replicas. Use at least three replicas for one-node fault tolerance. A two-node cluster is useful for development, but it cannot make progress after either node fails.

Build from source

The VSR wire path is behind the vsr Cargo feature. Build the server and CLI from the repository root:

cargo build --bin iggy-server-ng --bin iggy --features vsr

The current binary names are:

target/debug/iggy-server-ng
target/debug/iggy

The server uses Linux io_uring. Use a recent Linux kernel and a sufficient locked-memory limit.

Run one development node

The default core/server-ng/config.toml has clustering disabled. Run it without --replica-id:

IGGY_ROOT_USERNAME=iggy \
IGGY_ROOT_PASSWORD=iggy \
IGGY_CONFIG_PATH=core/server-ng/config.toml \
cargo run --bin iggy-server-ng --features vsr

Do not pass --replica-id when cluster.enabled = false.

Run a three-node cluster

Start with a shared config. Every node must use the same cluster name and roster.

[cluster]
enabled = true
name = "iggy-vsr-dev"

[cluster.auth]
enabled = true
shared_secret = ""

[cluster.tls]
enabled = false
self_signed = false
cert_file = ""
key_file = ""
ca_file = ""

[[cluster.nodes]]
name = "iggy-node-1"
ip = "127.0.0.1"
replica_id = 0
ports = { tcp = 8090, quic = 8080, http = 3000, websocket = 8093, tcp_replica = 9090 }

[[cluster.nodes]]
name = "iggy-node-2"
ip = "127.0.0.1"
replica_id = 1
ports = { tcp = 8091, quic = 8081, http = 3001, websocket = 8094, tcp_replica = 9091 }

[[cluster.nodes]]
name = "iggy-node-3"
ip = "127.0.0.1"
replica_id = 2
ports = { tcp = 8092, quic = 8082, http = 3002, websocket = 8095, tcp_replica = 9092 }

Save it as /tmp/iggy-vsr.toml. Export the settings shared by all three processes:

export IGGY_CONFIG_PATH=/tmp/iggy-vsr.toml
export IGGY_ROOT_USERNAME=iggy
export IGGY_ROOT_PASSWORD=iggy
export IGGY_CLUSTER_AUTH_SHARED_SECRET="replace-with-at-least-32-random-bytes"

Start each replica in a separate terminal. Use a different data path for every process:

# Replica 0
IGGY_SYSTEM_PATH=local_data/node-0 ./target/debug/iggy-server-ng --replica-id 0

# Replica 1
IGGY_SYSTEM_PATH=local_data/node-1 ./target/debug/iggy-server-ng --replica-id 1

# Replica 2
IGGY_SYSTEM_PATH=local_data/node-2 ./target/debug/iggy-server-ng --replica-id 2

The exported settings must be present in all three terminals.

In cluster mode:

• --replica-id is required and must match one cluster.nodes entry
• replica_id values must be unique and contiguous from 0
• every enabled transport needs a port for every node
• tcp_replica is reserved for replica traffic
• use advertised_address when clients cannot reach the roster ip
• use a different system.path for each process on the same host

Check cluster metadata

The CLI must also be built with vsr:

./target/debug/iggy \
  --tcp-server-address 127.0.0.1:8090 \
  -u iggy \
  -p iggy \
  cluster list

The response lists the cluster name, nodes, client endpoints, and current roles.

Use the Rust SDK

Until VSR is released on the normal SDK path, depend on the current Git repository and enable vsr:

[dependencies]
iggy = { git = "https://github.com/apache/iggy", features = ["vsr"] }
tokio = { version = "1", features = ["full"] }

The public client API stays the same:

use iggy::prelude::*;

#[tokio::main]
async fn main() -> Result<(), IggyError> {
    let client = IggyClientBuilder::from_connection_string(
        "iggy://iggy:iggy@127.0.0.1:8090",
    )?
    .build()?;

    client.connect().await?;

    let cluster = client.get_cluster_metadata().await?;
    println!("cluster: {}", cluster.name);

    Ok(())
}

The feature enables VSR framing and consensus session handling for TCP, QUIC, and WebSocket clients. A VSR-enabled client is not wire-compatible with the legacy server.

The Rust SDK and CLI currently have the complete VSR feature path. Other language SDKs still use the legacy wire protocol.

Test the VSR path

The repository has a dedicated Rust BDD lane:

cargo build --bin iggy-server-ng --bin iggy --features vsr
./scripts/run-bdd-tests.sh --vsr rust

Run one feature group while developing:

./scripts/run-bdd-tests.sh --vsr rust basic_messaging
./scripts/run-bdd-tests.sh --vsr rust leader_redirection

The VSR build and legacy build use the same output path for iggy. Rebuild the CLI without vsr before returning to legacy-server tests.

Cluster security

Replica authentication uses a shared secret and a BLAKE3 keyed handshake. Set the secret through IGGY_CLUSTER_AUTH_SHARED_SECRET instead of storing it in TOML.

Replica TLS protects the tcp_replica connection:

[cluster.auth]
enabled = true

[cluster.tls]
enabled = true
self_signed = false
cert_file = "/etc/iggy/tls/node.crt"
key_file = "/etc/iggy/tls/node.key"
ca_file = "/etc/iggy/tls/ca.crt"

Enable authentication and TLS on every node in one coordinated restart.

Default consensus timing

The defaults are configurable in [cluster]:

Keep the defaults unless tests show a specific scheduling or network problem.

Path to v1

server-ng is a temporary development name. The plan for Iggy v1 is:

1. finish and stabilize the VSR server
2. make it the only server implementation
3. remove the current core/server
4. rename core/server-ng to core/server
5. rename iggy-server-ng to iggy-server

Expect the paths and binary names on this page to change when that transition lands.

Follow development

• Current server-ng source
• All Apache Iggy pull requests
• Configuration hardening
• Session resume across restarts
• Follower HTTP forwarding
• Rust BDD coverage for VSR
• VSR-enabled CLI