Shared encodings

Primitives the command payloads are built from.

Name

Length-prefixed UTF-8 string, used for stream, topic, group, user, and token names:

[length: u8][value: UTF-8, 1-255 bytes]

Identifier

Polymorphic identifier of a stream, topic, user, or consumer group:

[kind: u8][length: u8][value: N bytes]

• Numeric: kind = 1, length = 4, value is a u32.
• String: kind = 2, length = 1..255, value is UTF-8 bytes.

Consumer

Identifies a consumer or a consumer group in message and offset commands:

[kind: u8][id: Identifier]

• kind = 1: consumer.
• kind = 2: consumer group.

Optional partition id

Several commands take an optional partition id encoded as 5 fixed bytes:

[flag: u8][partition_id: u32]

flag = 1 means the id is set. flag = 0 means none (the value bytes are then zero). For consumer-group consumers the partition id is left unset and the server resolves it.

Partitioning

How SendMessages picks the target partition:

[kind: u8][length: u8][value: 0-255 bytes]

• Balanced: kind = 1, length = 0, no value. The server resolves the partition at admission with a per-node round-robin counter.
• PartitionId: kind = 2, length = 4, value is a u32 partition id.
• MessagesKey: kind = 3, length = 1..255, value is a routing key hashed to a partition.

First-party binary SDKs pre-resolve balanced and key routing client-side and normally send kind = 2.

Polling strategy

Fixed 9 bytes:

[kind: u8][value: u64]

Header kinds

Typed values used by user headers and option values. The kind byte:

Integers are little-endian. Bool is one byte.

User headers block

A user headers block (inside a message frame) is a flat run of TLV fields:

[kind: u8][length: u32][data: length bytes]

Fields pair up: first the key, then the value. Keys are String kind. Every length must be 1..=255, kind 0 is rejected, and the block must consume its byte range exactly. Unknown value kind codes are preserved and forwarded, so headers survive mixed-version clusters.

Options block

CreateStream, UpdateStream, CreateTopic, UpdateTopic, CreateUser, and UpdateUser end with a key-value options block that reuses the user-headers TLV encoding. The block runs from its start to the end of the payload (no length prefix, and the validator requires exact consumption) and may be empty.

On top of the TLV walk, options enforce: string keys only (kind 2, valid UTF-8), no duplicate keys, at most 1024 entries, and at most 100000 bytes total.

Semantics:

• Create requests resolve absent keys to server defaults and persist the effective values, so GetTopic always shows what is in force.
• Update requests are patches: keys absent from the block are left alone, never reset. A client built before a key existed cannot erase it.
• Unknown keys are rejected at the wire edge, never silently skipped.

The catalog is discoverable at runtime with DESCRIBE_OPTIONS (code 13), payload [scope: u8] with scope 1 = topic, 2 = stream, 3 = user (HTTP: GET /options/topic). Today only topics have keys. The stream and user catalogs are empty, and any key sent for them is rejected. The topic catalog (segment_size, enforce_fsync, message_expiry, max_topic_size, and the rest) with defaults and constraints is documented on the Topic options page. UpdateTopic accepts only compression_algorithm, message_expiry, and max_topic_size. The storage-layout knobs are create-only.

Compression

The compression_algorithm value used by topic options:

Any other value is rejected. The option is a placeholder today: the value is validated, persisted, and echoed back, but neither the server nor the SDKs compress or decompress payloads yet. To compress today, do it client-side and tag messages via user headers - see the message headers examples in the Iggy repo.