# Shared encodings

> The primitives command payloads are built from: names, identifiers, consumers and the other shared encodings.

Rendered page: https://iggy.apache.org/docs/binary-protocol/encodings/

Source: https://github.com/apache/iggy-website/blob/main/content/docs/binary-protocol/encodings.mdx

Primitives the [command payloads](/docs/binary-protocol/commands) are built from.

## Name

Length-prefixed UTF-8 string, used for stream, topic, group, user, and token names:

```text
[length: u8][value: UTF-8, 1-255 bytes]
```

## Identifier

Polymorphic identifier of a stream, topic, user, or consumer group:

```text
[kind: u8][length: u8][value: N bytes]
```

- Numeric: `kind = 1`, `length = 4`, value is a u32.
- String: `kind = 2`, `length = 1..255`, value is UTF-8 bytes.

## Consumer

Identifies a consumer or a consumer group in message and offset commands:

```text
[kind: u8][id: Identifier]
```

- `kind = 1`: consumer.
- `kind = 2`: consumer group.

## Optional partition id

Several commands take an optional partition id encoded as 5 fixed bytes:

```text
[flag: u8][partition_id: u32]
```

`flag = 1` means the id is set. `flag = 0` means none (the value bytes are then zero). For consumer-group consumers the partition id is left unset and the server resolves it.

## Partitioning

How `SendMessages` picks the target partition:

```text
[kind: u8][length: u8][value: 0-255 bytes]
```

- `Balanced`: `kind = 1`, `length = 0`, no value. The server resolves the partition at admission with a per-node round-robin counter.
- `PartitionId`: `kind = 2`, `length = 4`, value is a u32 partition id.
- `MessagesKey`: `kind = 3`, `length = 1..255`, value is a routing key hashed to a partition.

First-party binary SDKs pre-resolve balanced and key routing client-side and normally send `kind = 2`.

## Polling strategy

Fixed 9 bytes:

```text
[kind: u8][value: u64]
```

| Kind | Name | Meaning of `value` |
|------|------|--------------------|
| 1 | Offset | Start at this offset |
| 2 | Timestamp | Start at this timestamp (microseconds) |
| 3 | First | Start from the beginning (`value` ignored) |
| 4 | Last | Start from the end (`value` ignored) |
| 5 | Next | Continue after the stored consumer offset (`value` ignored) |

## Header kinds

Typed values used by user headers and option values. The kind byte:

| Code | Kind | Code | Kind |
|------|------|------|------|
| 1 | Raw | 9 | Uint8 |
| 2 | String | 10 | Uint16 |
| 3 | Bool | 11 | Uint32 |
| 4 | Int8 | 12 | Uint64 |
| 5 | Int16 | 13 | Uint128 |
| 6 | Int32 | 14 | Float32 |
| 7 | Int64 | 15 | Float64 |
| 8 | Int128 | | |

Integers are little-endian. `Bool` is one byte.

## User headers block

A user headers block (inside a [message frame](/docs/binary-protocol/messages)) is a flat run of TLV fields:

```text
[kind: u8][length: u32][data: length bytes]
```

Fields pair up: first the key, then the value. Keys are `String` kind. Every `length` must be 1..=255, kind 0 is rejected, and the block must consume its byte range exactly. Unknown value kind codes are preserved and forwarded, so headers survive mixed-version clusters.

## Options block

`CreateStream`, `UpdateStream`, `CreateTopic`, `UpdateTopic`, `CreateUser`, and `UpdateUser` end with a key-value **options block** that reuses the user-headers TLV encoding. The block runs from its start to the end of the payload (no length prefix, and the validator requires exact consumption) and may be empty.

On top of the TLV walk, options enforce: string keys only (kind 2, valid UTF-8), no duplicate keys, at most 1024 entries, and at most 100000 bytes total.

Semantics:

- **Create** requests resolve absent keys to server defaults and persist the effective values, so `GetTopic` always shows what is in force.
- **Update** requests are patches: keys absent from the block are left alone, never reset. A client built before a key existed cannot erase it.
- Unknown keys are rejected at the wire edge, never silently skipped.

The catalog is discoverable at runtime with `DESCRIBE_OPTIONS` (code 13), payload `[scope: u8]` with scope `1` = topic, `2` = stream, `3` = user (HTTP: `GET /options/topic`). Today only topics have keys. The stream and user catalogs are empty, and any key sent for them is rejected. The topic catalog (`segment_size`, `enforce_fsync`, `message_expiry`, `max_topic_size`, and the rest) with defaults and constraints is documented on the [Topic options](/docs/server/topic-options) page. `UpdateTopic` accepts only `compression_algorithm`, `message_expiry`, and `max_topic_size`. The storage-layout knobs are create-only.

## Compression

The `compression_algorithm` value used by topic options:

| Code / name | Algorithm |
|-------------|-----------|
| 1, `none` | No compression (default) |
| 2, `gzip` | Gzip |

Any other value is rejected. The option is a **placeholder today**: the value is validated, persisted, and echoed back, but neither the server nor the SDKs compress or decompress payloads yet. To compress today, do it client-side and tag messages via user headers - see the [message headers examples](https://github.com/apache/iggy/tree/master/examples/rust/src/message-headers) in the Iggy repo.