If Apache Iggy is useful to you, give it a star on GitHubStar apache/iggy
Skip to Content
Apache Iggy
Binary Protocol
These are the docs for Apache Iggy 0.10, kept for reference and no longer updated. Read the current docs.

Server-to-server

Replica links, consensus commands, history fences and the header layouts used by Apache Iggy 0.10.0.

Replicas use dedicated TCP listeners and 256-byte headers. Client listeners reject replica control commands. The replica protocol has no mixed-version upgrade path. Run the same server version and replica_connections_per_peer on every node.

Transport

The roster supplies each node's tcp_replica port and literal IP address. Lower replica IDs dial higher IDs. Each pair can use several links, which distribute traffic across shards. The default frame limit is 64 MiB through message_bus.max_message_size.

Each connection starts with an eight-byte placement preamble before TLS and the replica handshake:

[magic: IGRL, 4 bytes][version: u8 = 1][dialer_replica: u8][link_index: u16]

The preamble selects a shard and is not authentication. ReplicaHello repeats the peer and link identity and supplies the link count. The acceptor rejects a mismatch. With authentication enabled, placement fields participate in the handshake MAC. Shard counts can differ between nodes.

Handshake

ReplicaHello, ReplicaChallenge and ReplicaFinish establish the connection. [cluster.auth] enables the shared-key BLAKE3 MAC. [cluster.tls] encrypts traffic with TLS 1.3 and ALPN iggy-replica. TLS requires replica authentication. See Cluster security for configuration and key rotation.

Command discriminants

The command byte is at offset 60. This is the complete registry, including client commands:

ValueCommand
0Reserved
1Ping
2Pong
3PingClient
4PongClient
5Request
6Prepare
7PrepareOk
8Reply
9Commit
10StartViewChange
11DoViewChange
12StartView
13Eviction
14ReplicaHello
15ReplicaChallenge
16ReplicaFinish
17RequestStartView
18RequestPrepares
19RepairPrepare
20RepairDone
21RangeEvicted
22RequestStateTransfer
23StateTransferTarget
24RequestStateChunk
25StateChunk
26ForwardRegister
27ForwardRegisterResult
28ForwardLogout
29ForwardLogoutResult
30ConsumerSessionHeartbeat
31SessionRetirementProgress
32ForwardCompletion
33PrimaryRebalance
34SegmentDelete
35CommitReply
36PrepareProgress

Reserved and unused ping/pong forms do not establish a session. Reply and Eviction are client-bound and rejected on replica ingress. A replica Request uses the internal routed-request header, including its group and metadata watermark.

Consensus and history

The metadata group orders resources, users and sessions. Every partition has its own group for messages and offsets. The group field identifies the target. The metadata group uses 1 << 63, while partition groups use packed stream, topic and partition IDs.

Prepare carries an operation and its body. PrepareOk acknowledges the required replication and persistence work. Commit broadcasts the commit point and a liveness token. CommitReply proves that peers still respond. A primary that cannot reach a quorum relinquishes leadership.

StartViewChange, DoViewChange and StartView establish a new view. DoViewChange includes an untrusted-log flag so a replica without proven history cannot vote as an authoritative empty log. RequestStartView probes for the current view after recovery.

Partition incarnation fields fence an old partition history. Consumer ownership generations separately fence old group members. PrimaryRebalance moves a partition primary under metadata coordination. PrepareProgress lets a slow replica report received, held and committed progress while it catches up.

Frame integrity

Replica control frames use an XxHash3-64 frame seal widened to u128. Prepare and RepairPrepare use a prepare identity hash, and a parent identity links the history. Metadata prepare bodies have their own seal. Message batches carry their batch checksums. These unkeyed hashes detect corruption, not a malicious peer. Authentication and TLS provide separate protection.

Header layouts

Every header begins with the common 128-byte frame prefix shown in Framing. The tables below list the remaining fields in wire order. Multi-byte integers are little-endian. Reserved fields are not extension space for clients. RepairPrepare uses the PrepareHeader layout with its repair command value.

PrepareHeader

OffsetBytesField
12816client
14416parent
16016request_checksum
1768op
1848commit
1928timestamp
2008request
2081operation
2097operation_padding
2168group
2244user_id
2284retry_capacity
2328partition_incarnation
2408reserved
2488session

PrepareOkHeader

OffsetBytesField
12816parent
14416prepare_checksum
1608op
1688commit
1768timestamp
1848request
1921operation
1937operation_padding
2008group
20848reserved

CommitHeader

OffsetBytesField
12816commit_checksum
1448timestamp_monotonic
1528commit
1608checkpoint_op
1688group
1768partition_incarnation
18472reserved

CommitReplyHeader

OffsetBytesField
1288timestamp_monotonic
1368group
1448partition_incarnation
152104reserved

PrepareProgressHeader

OffsetBytesField
1288nonce
1368group
1448partition_incarnation
1528observed_op
1608received_op
1688held_op
1768commit_op
1841kind
1851ready
18670reserved

StartViewChangeHeader

OffsetBytesField
1288group
1368partition_incarnation
144112reserved

DoViewChangeHeader

OffsetBytesField
1288op
1368commit
1448group
1524log_view
1561flags
15759reserved
2168partition_incarnation
22416nack_bitset
24016present_bitset

StartViewHeader

OffsetBytesField
1288op
1368commit
1448group
1528partition_incarnation
16080reserved
24016incarnation

RequestStartViewHeader

OffsetBytesField
1288group
1368partition_incarnation
14496reserved
24016incarnation

RequestPreparesHeader

OffsetBytesField
12816nonce
1448from_op
1528to_op
1608group
1688partition_incarnation
17680reserved

RepairRangeReplyHeader

OffsetBytesField
12816nonce
1448op
1528group
1608partition_incarnation
16888reserved

RequestStateTransferHeader

OffsetBytesField
12816nonce
1448group
1528partition_incarnation
16096reserved

StateTransferTargetHeader

OffsetBytesField
12816nonce
1448commit_op
1528group
1601available
1611unavailable_transient
1626reserved_alignment
1688commit_max
1768partition_incarnation
18472reserved

RequestStateChunkHeader

OffsetBytesField
12816nonce
1448offset
1528group
1604len
1644artifact
1688partition_incarnation
17680reserved

StateChunkHeader

OffsetBytesField
12816nonce
1448offset
1528group
1604artifact
16484reserved
2488partition_incarnation

ForwardRegisterHeader

OffsetBytesField
12816client
14416nonce
1604user_id
16492reserved

ForwardRegisterResultHeader

OffsetBytesField
12816nonce
14416client
1608epoch
1688watermark
17679reserved
2551outcome

ForwardLogoutHeader

OffsetBytesField
12816client
14416nonce
1608session
1688request
17680reserved

ForwardLogoutResultHeader

OffsetBytesField
12816nonce
14416client
1608commit
16887reserved
2551outcome

ForwardCompletionHeader

OffsetBytesField
1282reporter
1301operation
131125reserved

Repair and state transfer

RequestPrepares asks for a retained operation range. RepairPrepare sends the original prepares. RepairDone terminates the stream, and RangeEvicted reports history that is no longer retained. Requests are bounded by repair_chunk_max and the peer queue capacity.

State transfer handles larger gaps. RequestStateTransfer obtains a manifest, then RequestStateChunk and StateChunk transfer artifacts. The receiver checks each complete artifact against its manifest length and checksum before using it. The manifest has magic ISM1, a u32 entry count, a u8 entry length and a trailing XxHash3-64 checksum. Entries contain kind, frontier, length and checksum.

KindArtifactCurrent format
0Metadata snapshotMessagePack version 15, readers accept 14 and 15
1Client tableICT5
2Segment logCanonical batch records
3Consumer offsets and partition stateICO1 version 9, readers accept 8 and 9

The offset artifact also retains partition history, retry state and producer state. These internal readability windows do not provide migration from a 0.9 data directory. Use the upgrade procedure.

The header definitions and command registry define the layouts. The replica transport implements preamble admission and authentication.

On this page