If Apache Iggy is useful to you, give it a star on GitHubStar apache/iggy
Skip to Content
Apache Iggy
Binary Protocol
These are the docs for Apache Iggy 0.10, kept for reference and no longer updated. Read the current docs.

Shared encodings

The primitives command payloads are built from: names, identifiers, consumers and the other shared encodings.

Primitives the command payloads are built from.

Name

Length-prefixed UTF-8 string, used for stream, topic, group, user, and token names:

[length: u8][value: UTF-8, 1-255 bytes]

Identifier

Polymorphic identifier of a stream, topic, user, or consumer group:

[kind: u8][length: u8][value: N bytes]
  • Numeric: kind = 1, length = 4, value is a u32.
  • String: kind = 2, length = 1..255, value is UTF-8 bytes.

Consumer

Identifies a consumer or a consumer group in message and offset commands:

[kind: u8][id: Identifier]
  • kind = 1: consumer.
  • kind = 2: consumer group.
  • kind = 3: externally managed group offsets. This kind cannot poll or join native consumer groups.

Optional partition id

Several commands take an optional partition id encoded as 5 fixed bytes:

[flag: u8][partition_id: u32]

flag = 1 means the id is set. flag = 0 means none (the value bytes are then zero). Group polls and group offset writes require an explicit assigned partition. Plain-consumer polls and consumer-offset reads use partition 0 when the id is omitted.

Partitioning

How SendMessages picks the target partition:

[kind: u8][length: u8][value: 0-255 bytes]
  • Balanced: kind = 1, length = 0, no value. The binary client resolves it to an explicit partition before sending.
  • PartitionId: kind = 2, length = 4, value is a u32 partition id.
  • MessagesKey: kind = 3, length = 1..255, value is a routing key hashed to a partition.

The Rust binary SDK resolves balanced and key routing client-side and sends kind = 2. The binary server requires explicit partition routing. The HTTP API can resolve the other strategies.

Polling strategy

Fixed 9 bytes:

[kind: u8][value: u64]
KindNameMeaning of value
1OffsetStart at this offset
2TimestampStart at this timestamp (microseconds)
3FirstStart from the beginning (value ignored)
4LastStart from the end (value ignored)
5NextContinue after the stored consumer offset (value ignored)

Header kinds

Typed values used by user headers and option values. The kind byte:

CodeKindCodeKind
1Raw9Uint8
2String10Uint16
3Bool11Uint32
4Int812Uint64
5Int1613Uint128
6Int3214Float32
7Int6415Float64
8Int128

Integers are little-endian. Bool is one byte.

User headers block

A user headers block (inside a message frame) is a flat run of TLV fields:

[kind: u8][length: u32][data: length bytes]

Fields pair up: first the key, then the value. Keys and values can both use the header kinds above. Every length must be 1..=255, kind 0 is rejected, and the block must consume its byte range exactly. The wire validator preserves unknown non-zero kind codes. A typed SDK accessor may still reject a kind it does not recognize.

Options block

CreateStream, CreateTopic, CreateUser and UpdateResource end with a key-value options block that reuses the user-headers TLV encoding. The block runs from its start to the end of the payload (no length prefix, and the validator requires exact consumption) and may be empty.

On top of the TLV walk, options enforce: string keys only (kind 2, valid UTF-8), no duplicate keys, at most 1024 entries, and at most 100000 bytes total.

Semantics:

  • Create requests resolve absent keys to server defaults and persist the effective values. Updates use explicit reset keys to restore defaults. Zero retention sentinels do not reset a value. See update semantics.
  • Update requests are patches: keys absent from the block are left alone, never reset. A client built before a key existed cannot erase it.
  • Unknown keys are rejected at the wire edge, never silently skipped.

The catalog is discoverable with DescribeOptions, code 13, using scope 1 for topics, 2 for streams and 3 for users. All three catalogs have entries. Stream keys are create-only and secret. Stream and topic names and user identity/status can be updated. Topic retention fields can be reset, while compression and storage policies are fixed at creation. See Topic options and Stream options.

Compression

The compression_algorithm value used by topic options:

Code / nameAlgorithm
1, noneNo compression (default)
2, gzipGzip

Any other value is rejected. The option is a placeholder today: the value is validated, persisted, and echoed back, but it does not enable automatic payload compression or decompression. To compress today, do it client-side and tag messages via user headers - see the message headers examples in the Iggy repo.

Partition context

A served partition context is 24 bytes:

[incarnation: u64][owner_generation: u64][metadata_op: u64]

The incarnation identifies the partition history. The owner generation identifies group ownership and is zero for ordinary consumers. The metadata operation establishes the minimum metadata needed to interpret the context. Poll replies, continuations and checkpoints preserve these fields.

Option catalog flags

DescribeOptions entries report mutability and visibility. Bit 0 means updatable, bit 1 resettable, bit 2 secret, bit 3 read-only and bit 4 critical. A critical key must be understood before a node serves the resource. Unknown client option keys are rejected.

On this page