Server-to-server

Replicas talk to each other with the same 256-byte framing the client protocol uses, on a dedicated TCP port, with their own set of command discriminants. None of these frames ever appears on a client connection, and a client cannot reach this plane: it is a separate listener, gated by a handshake.

This page documents the replica plane as of binary protocol 0.11.0. It is internal protocol: only Iggy servers speak it, and it can change between server releases without a client-facing version bump.

Transport

• Every node in [[cluster.nodes]] exposes a tcp_replica port next to the client ports: ports = { tcp = 8090, quic = 8080, http = 3000, websocket = 8092, tcp_replica = 9090 }. The node's ip is the roster address for the replica plane (clients fall back to it when advertised_address is unset).
• The replica plane is TCP only, by design: the prepare hash chain, cross-shard fd delegation, and view-change timing all assume an ordered byte stream.
• Directional dialing: a replica dials only peers with a strictly greater replica_id and accepts inbound connections only from strictly lower ids. Exactly one connection exists per pair, with no tiebreaker races.
• Frames are the standard [256-byte header][optional body] with size at offset 48. The maximum frame is 64 MiB by default (message_bus.max_message_size). Headers are #[repr(C)], decoded zero-copy, and require 16-byte alignment.
• Nothing inside a frame marks it as replica traffic. The separation is structural: client listeners parse every inbound frame as a RequestHeader (command 5 only), the replica listener requires the first frame to be ReplicaHello, and the client-bound commands Reply (8) and Eviction (13) are rejected if they arrive on the replica plane.

Command discriminants

The full command byte registry, client values included. Values above 29 are rejected.

There is no dedicated replica heartbeat: liveness is the Commit broadcast, sent on cluster.commit_broadcast_interval.

Frame integrity

The checksum and checksum_body fields that client frames leave zero are live on the replica plane:

• Frame seal (checksum, bytes 0..16): XxHash3-64 over header bytes 16..256, widened to u128. Sealed on every replica message except Prepare, RepairPrepare, and the three handshake frames (whose integrity comes from the keyed MAC). Verified once, at typed decode, before any field validation; a mismatch drops the frame.
• Prepare identity (Prepare / RepairPrepare only): those two spend checksum on an identity hash instead - XxHash3-64 over the whole 256-byte header computed with checksum = 0 and view = 0, so a retransmit that re-stamps view keeps the same identity. PrepareOk echoes it in prepare_checksum, and each prepare's parent field carries the previous prepare's identity, forming a hash chain.
• Body seal (checksum_body, bytes 16..32): three regimes. Metadata-plane prepares seal their body with XxHash3-64. Partition-plane prepares leave it zero: the message batch inside already carries batch_checksum, verified at network ingress. DoViewChange and StartView seal their suffix bodies. Every other message leaves it zero (state-transfer bodies are verified per artifact, not per frame).
• The seals are unkeyed integrity checks, not authentication. Peer authentication comes from the handshake below; without cluster.auth and TLS enabled, the replica port trusts any peer that can reach it. Do not expose tcp_replica beyond the cluster network.

Prepare, RepairPrepare, and the session-forwarding headers validate their reserved regions as zero; the other replica headers leave reserved bytes covered by the frame seal only.

Handshake

Commands 14-16, exchanged before any consensus traffic. These are raw GenericHeader frames (size = 256) using the per-command area at bytes 128..256:

Statuses: 0 Ok, 1 UnknownCommand, 2 ClusterMismatch, 3 DirectionalRule; values 4 and 5 are reserved (never sent), and a dialer treats any unknown status byte as a rejection. The MAC key derives from cluster.auth.shared_secret (32+ bytes; previous_shared_secret gives a rotation window), and with cluster TLS enabled (TLS 1.3 only, ALPN iggy-replica) the TLS exporter is folded into every MAC as channel binding. The handshake authenticates cluster membership, not per-replica identity: one shared secret for the whole cluster.

The cluster header field (bytes 32..48) is the first 16 bytes of blake3(cluster_name) as a little-endian u128 on every replica frame, checked during the handshake, so nodes from a differently named cluster cannot connect even with auth disabled.

Replication

Prepare (6)

Per-command fields (bytes 128..256):

For metadata operations the body is the admitted command's payload verbatim. For SendMessages it is the message batch as stamped by the primary at journal append (base_offset, base_timestamp, and the recomputed batch_checksum are final); each backup re-derives the expected stamp from its own position and refuses a mismatch. Replication is chain-form: the primary sends to (replica + 1) % replica_count, each backup forwards to its own successor, and the chain stops when the next hop is the primary. RepairPrepare (19) is byte-identical to a journaled prepare with only the command byte rewritten.

PrepareOk (7)

Header-only. parent (echo) at 128, prepare_checksum (echo of the prepare's identity) at 144, then op 160, commit 168, timestamp 176, request 184, operation 192, group 200. Two fields are the acker's own, not echoes: header view is the acking replica's view, and commit is its own commit point. Sent to the view's primary.

Commit (9)

Header-only broadcast; validation rejects size != 256. timestamp_monotonic at 144 is the liveness token (a receiver resets its heartbeat timer only on a strictly newer value), commit at 152 is the commit point, group at 168. Bytes 128..144 and 160..168 are declared (commit_checksum, checkpoint_op) but currently unused.

View changes

• StartViewChange (10): header-only, just group at 128. Broadcast when a replica suspects the primary.
• DoViewChange (11): op (highest op) 128, commit 136, group 144, log_view 152 (view when the sender was last in Normal status), nack_bitset 224 (bit i = "I never prepared suffix entry i"; truncation authority), present_bitset 240 (bit i = "I can serve entry i's body"). Body = the log suffix as up to 128 packed 256-byte PrepareHeaders, headers only. Broadcast.
• StartView (12): op 128, commit 136 (max over the DVC quorum), group 144, incarnation 240 (echo of a RequestStartView incarnation, zero when unsolicited). Body = the view's suffix as packed PrepareHeaders, highest op first; empty is legal. Unicast when answering a probe, broadcast otherwise.
• RequestStartView (17): header-only recovery probe with group 128 and incarnation 240. A restarted replica broadcasts it instead of trusting stale local state; only the current view's primary answers, with a targeted StartView.

Journal repair

A replica missing committed prepares (rejoin window or interior hole) pulls them from a Normal-status peer (the metadata plane also serves repair during a view change, so a forming primary cannot deadlock):

• RequestPrepares (18): nonce 128, from_op 144, to_op 152, group 160. from_op = 0 or from_op > to_op is rejected.
• RepairPrepare (19): the journaled prepares, streamed in op order.
• RepairDone (20) / RangeEvicted (21): one shared layout (nonce 128, op 144, group 152) under two command bytes. RepairDone.op = last op served; RangeEvicted.op = oldest op the peer still retains, the honest answer when the front of the range is gone.

Pacing is bounded by cluster.repair_chunk_max, which must stay below message_bus.peer_queue_capacity: per-peer send queues drop overruns silently.

State transfer

Bulk catch-up for a replica too far behind for journal repair. Pull-based and lockstep (at most one chunk in flight per artifact), because per-peer queues drop overruns:

1. RequestStateTransfer (22): nonce 128, group 144.
2. StateTransferTarget (23): nonce 128, commit_op 144, group 152, available 160, unavailable_transient 161, commit_max 168. available = 0 is a header-only refusal. Body = the state manifest.
3. RequestStateChunk (24): nonce 128, offset 144, group 152, len 160, artifact 164. len describes the reply and is clamped by the server.
4. StateChunk (25): nonce 128, offset 144, group 152, artifact 160. Body = raw artifact bytes. Chunks carry no per-chunk integrity; each artifact is verified whole, by length and XxHash3-64, against its manifest entry.

The manifest (little-endian):

[magic: "ISM1"][count: u32][entry_len: u8]
count x { kind: u8, frontier: u64, len: u64, checksum: u64 }   # entry_len = 25
[trailer: XxHash3-64 over everything above]

At most 65536 entries. Entries longer than 25 bytes decode with the tail skipped (forward compatibility); shorter entries are rejected.

Artifact kinds:

Session forwarding

A client may dial a backup; authentication happens there, and only the verified identity travels to the primary (credentials never cross the replica plane):

• ForwardRegister (26): client 128, nonce 144, user_id 160.
• ForwardRegisterResult (27): nonce 128, client 144, epoch 160, watermark 168, outcome byte at 255: 0 Ok, 1 NotPrimary, 2 NotCaughtUp, 3 PipelineFull, 4 InProgress, 5 Canceled, 6 ClientIdOwnedByAnotherUser. epoch and watermark must be zero on any non-Ok outcome.
• ForwardLogout (28): client 128, nonce 144, session 160, request 168.
• ForwardLogoutResult (29): nonce 128, client 144, commit 160, outcome at 255: 0 Ok, 1 NotPrimary, 2 PipelineFull, 3 InProgress, 4 Canceled.

Consensus planes

Two consensus planes share this one message set; there are no plane-specific commands. Routing is by the group: u64 field carried on every consensus header except the handshake and session-forwarding frames (those are implicitly metadata-plane):

• group = 1 << 63 is the metadata plane (streams, topics, users, consumer groups; durable on-disk journal).
• Any other value is a packed stream/topic/partition key: a partition plane group (message batches, consumer offsets; in-memory journal).

Same wire, a few divergent semantics: partition-plane prepares leave checksum_body zero and rely on the batch's own batch_checksum, their identity checksum covers the header alone, and StateTransferTarget.unavailable_transient / commit_max are read only by the partition arm.