Storage Engine

Iggy's storage engine is built around the concept of a segmented append-only log. Message data follows a hierarchy: System -> Streams -> Topics -> Partitions -> Segments. This page covers how data is stored, indexed, flushed, recovered, and cleaned up on disk.

Directory layout

All data lives under the root path directory (default local_data, overridden with IGGY_PATH):

local_data/
├── metadata/                        # Metadata plane (see below)
│   ├── journal.wal                  # Prepare journal: replicated metadata operations
│   ├── snapshot.bin                 # MessagePack snapshot of the metadata state machine
│   ├── superblock.a                 # Ping-pong superblock pair: checkpoint record,
│   └── superblock.b                 #   newest slot wins by sequence
├── state/
│   └── log                          # Created at boot; legacy state plumbing
├── runtime/                         # Runtime data
├── logs/                            # Server log files
└── streams/
    └── {stream_id}/
        └── topics/
            └── {topic_id}/
                └── partitions/
                    └── {partition_id}/
                        ├── superblock.a                  # Partition consensus superblock pair
                        ├── superblock.b
                        ├── prepares-{created_revision}/  # Prepare WAL, when a cluster topic policy is persisted
                        ├── offsets/
                        │   ├── consumers/                # Stored offsets of individual consumers
                        │   └── groups/                   # Stored offsets of consumer groups
                        ├── 00000000000000000000.log      # Segment: batch records
                        ├── 00000000000000000000.index    # Segment: sparse index
                        ├── 00000000000016000000.log
                        └── 00000000000016000000.index

Stream, topic, and partition ids are numeric and 0-based. Each partition directory holds pairs of .log and .index files. The filename is the segment's start offset, zero-padded to 20 digits. Recovery can create an empty active segment at a reserved offset beyond the last stored message. Next to the segments, every partition keeps its own superblock pair (replica identity and consensus state for that partition) and an offsets/ tree for consumer offset storage. Multi-replica partitions also keep a prepare WAL when either topic durability policy is persisted.

Segmented log

Each partition is a segmented log: an ordered list of sealed (read-only) segments plus one active (writable) segment. When the active segment reaches the topic's segment_size (default 1 GiB), it is sealed and a new active segment starts at the next offset.

segment_size is a per-topic creation option, bounded to a 512-byte multiple between 1 MiB and 1 GiB. It's a soft limit: rotation fires after the append that crosses it, so the crossing batch lands whole and a sealed segment may run one batch past the configured size. preallocate_segments (also per topic) reserves each segment's bytes up front on filesystems that support it. See Topic options.

On-disk batch format

Segments store batch records, not individual messages. The record is byte-identical to the wire encoding: the one layout a SendMessages body, the replicated prepare, the persisted segment record, and the poll reply all share. There is no other message encoding.

[batch header: 256 bytes][blob: message frames]
frame = [frame header: 48 bytes][payload][user_headers]

The 256-byte batch header carries partition_id, base_offset, base_timestamp, origin_timestamp, batch_length, batch_checksum (XxHash3-64), and message_count. The rest is reserved and must be zero. Each 48-byte frame header carries checksum (XxHash3-64), id (u128), offset_delta (u32), timestamp_delta (u32), and the two lengths. A message's absolute offset is base_offset + offset_delta; its server timestamp is the flat base_timestamp, while timestamp_delta resolves producer time against origin_timestamp. All fields are little-endian, and records are stored contiguously with no padding.

The full byte-level tables live on the Message batches page. Absent at-rest encryption, what lands on disk is exactly what the producer sent, with the server stamping partition_id at admission and base_offset / base_timestamp (plus a batch_checksum recompute) at journal append.

Indexes

Each segment has an accompanying .index file holding a sparse index: one 24-byte entry per flushed write, not per message.

Little-endian, no header, no padding. To serve a poll, the server binary-searches the index for the closest entry at or below the target offset (or timestamp), seeks to its position, and walks batch records from there. Because the index is a sparse hint, a lookup below the indexed range falls back to the segment start rather than failing.

There is no index caching configuration: the in-memory index cache is an internal per-segment structure the server manages itself.

Write pipeline

Messages are admitted, stamped, and buffered in the partition journal. Their completion path depends on the topic's durability policy:

The flush thresholds and completion policies are per-topic creation options:

• messages_required_to_save (default 1024): count threshold.
• size_of_messages_required_to_save (default 1 MiB): byte threshold.
• durability (default replicated): message completion policy; persisted requires recoverable stable storage at the replication quorum.
• consumer_offset_durability (default replicated): the independent policy for explicit offset stores and deletes.

A full active segment also triggers an ordinary flush. These are soft limits: a flush writes whole committed batches, so the actual count or size can overshoot. Required persistence, capacity pressure, and lifecycle operations can flush below the thresholds. The settings do not define a periodic flush interval. The MessagesWriter uses vectored I/O with up to 1024 buffers per syscall, so many buffered batches can land in one write.

With replicated, completion does not wait for an additional stable-storage barrier. With persisted, a single-replica partition flushes and synchronizes committed segment state before success. A multi-replica partition instead gates the required prepare acknowledgements on recoverable WAL history, so persisted success can precede an ordinary segment flush. messages_required_to_save=1 is not required.

For multi-replica partitions, the disk prepare WAL is enabled when either policy is persisted and includes message payloads even if only the offset policy is persisted. Checkpoints synchronize materialized files before reclaiming WAL history. See Durability and Cluster Durability for completion guarantees and failure behavior.

Boot-time segment recovery

At startup, recovery validates batch checksums, partition identity, and the segment chain before serving data. With message durability replicated, it walks the log from byte zero because buffered writeback can preserve later pages before earlier ones. With persisted, completed durable segment flushes allow an index-anchored recovery path.

A missing or torn sparse index can be rebuilt from valid log data. An incomplete tail can be truncated, but an interior gap, intact records after damage, or a contradiction with durable history can require recovery refusal. Recovery does not blindly truncate at the first invalid batch.

When a prepare WAL exists, recovery reconciles it with materialized segment and offset state. Unrecoverable partition data is fenced and repaired from peers when available. It must not be replaced by an empty healthy partition. A singleton has no peer copy to fetch.

Read integrity

Disk reads are verified before they reach a consumer:

[partition]
validate_checksum = true

With validate_checksum = true (the default), every batch a disk poll reads is re-hashed and compared against its stored checksum. A mismatch stops the disk walk and logs an error on the server. The consumer receives any valid prefix already read, or an ordinary empty poll if none was read; the protocol does not report the checksum failure to the consumer. Repeated polls can therefore wait indefinitely at damaged data. Setting it to false skips the re-hash and serves whatever decodes, which can hand a consumer bytes provably not the ones written. Only disable it with a corruption guard elsewhere in the stack.

Binary polls reuse the stored batch layout. A reply may slice a larger stored batch and rewrite its header, and at-rest encryption requires decryption before the reply. HTTP additionally serializes the result as JSON.

Memory pool

Iggy includes a custom memory pool to reuse buffers on the hot path. The pool has 28 buckets with buffer sizes from 4 KiB up to 512 MiB (non-uniform spacing, denser around common message sizes, with sizes above 2 MiB rounded to hugepage-friendly steps). Components request a buffer from the appropriate bucket and return it when done.

[memory_pool]
enabled = true
size = "4 GiB"            # Total pool size (minimum 512 MiB, multiple of the 4096-byte page size)
bucket_capacity = 8192    # Buffers per bucket (power of 2, minimum 128)

Buffers are allocated lazily. A pool miss can allocate outside the pool, so message processing is not allocation-free. Internal components can pass ownership of existing buffers without copying their contents.

Retention and cleanup

Two independent retention policies exist, both per-topic options configurable at creation or update (see Topic options). The segment cleaner enforces them:

[data_maintenance.messages]
cleaner_enabled = true    # default
interval = "1 m"          # default

Size-based retention (max_topic_size): the cluster has no single owner of a topic-wide total, so each partition enforces an equal share: max_topic_size / partition_count, counted over sealed bytes. The share is floored at one sealed-segment ceiling (segment_size plus the maximum message-bus frame, since a sealed segment can overshoot by one batch). A cap that divides below that floor is raised to it, so the policy always means at least "keep the newest sealed segment". Once sealed bytes exceed the budget, the oldest sealed segments are trimmed. There is no percentage-based early trigger.

Time-based retention (message_expiry): sealed segments whose newest message is older than the expiry are deleted.

Both policies can be active at once. They only ever touch sealed segments. Deletion also stops at the minimum committed consumer or consumer-group offset, and pending persistence checkpoints can defer it. The active segment is never deleted by retention, even if its messages have expired.

Metadata plane

Metadata operations (create stream, delete topic, create user, and so on) do not use the partition storage above. They replicate through the metadata consensus group and persist in local_data/metadata/:

• journal.wal: the prepare journal. Each entry is a 256-byte PrepareHeader (the consensus header carrying checksums, view, op and commit numbers, operation discriminant, and acting user) followed by the operation's wire-format body, exactly as replicated. This is the write-ahead log: an operation is journaled before it is applied.
• snapshot.bin: a MessagePack (rmp-serde) snapshot of the whole metadata state machine, written at checkpoints so the journal can be truncated. Recovery loads the snapshot, then replays journal entries newer than it, rebuilding the client session table alongside.
• superblock.a / superblock.b: a ping-pong pair of checkpoint records, where the newest valid slot wins by sequence number. The superblock records which checkpoint is current and the snapshot's checksum. Boot cross-checks the pairing: a superblock pointing at a checkpoint newer than the on-disk snapshot, or a snapshot whose checksum doesn't match the record, refuses to boot rather than silently rewinding committed state.

Partition directories carry the same superblock mechanism for their own consensus state (replica identity, view), which is how a restarted replica proves it is itself rather than a blank impostor.

Encryption

Iggy supports optional AES-256-GCM encryption for message payloads and user headers. They are encrypted before storage and decrypted for polling. Metadata journals, metadata snapshots, and structural record headers remain unencrypted. The key must decode from base64 to 32 bytes. See Security.

[encryption]
enabled = false
key = ""  # 32-byte base64-encoded key

Compression

The compression_algorithm topic option accepts none (default) and gzip, but it is a placeholder today: the value is persisted and reported back, but no message compression is applied. Payload encryption still applies when enabled. To compress today, do it client-side and tag messages via user headers - see the message headers examples in the Iggy repo.

Removed and relocated settings

Earlier releases documented several storage features and [system.*] keys that no longer exist:

• cache_indexes: removed. Index caching is internal now (see Indexes).
• [system.message_deduplication]: removed configuration; it does not configure the partition request deduplication table.
• Segment archiving / S3 backup placeholders: removed. Delete archive_expired and the old [system.segment] table rather than retaining a false value.
• enforce_fsync: replaced by the topic durability policy. Consumer-offset completion has its own consumer_offset_durability policy. Both default independently to replicated.
• [system.*] tables: removed or moved to root tables. Retention, segment size, preallocation, and flush thresholds are per-topic options.

The config loader rejects the old [system] table and IGGY_SYSTEM_* environment mappings. The full mapping is in Relocated configuration keys.