Architecture

This page covers the internals of the Iggy server: how work is scheduled across CPU cores, how requests are routed, and how data ends up on disk. If you're looking for the domain model (streams, topics, partitions, consumer groups, message format), see concepts first.

How a message flows through Iggy

Before diving into the architecture details, here's the complete journey of a message from client to disk:

Thread per core (shared nothing) + io_uring

Iggy uses a thread-per-core shared nothing architecture combined with io_uring for maximum performance. This design has been proven by systems like ScyllaDB and Redpanda, and is inspired by the Seastar framework.

How it works

Each CPU core runs its own shard (an instance of IggyShard), pinned to a specific core via sched_setaffinity on Linux. Each shard has its own single-threaded compio async runtime, which means there is no cross-thread synchronization needed within a shard. Memory is bound to the NUMA node of the core via hwlocality for optimal memory access latency.

Shard roles and connection distribution

Shard 0 has a special role: it binds every listener - the replica plane and all client transports (TCP, QUIC, WebSocket, HTTP). Connections are then spread across shards at accept time:

• Plaintext TCP and WebSocket connections are handed off round-robin to peer shards. Shard 0's coordinator duplicates the socket's file descriptor, ships a connection-setup frame to the target shard, and drops its own handle, so the owning shard serves the connection from then on - before a single byte is read.
• QUIC and TLS-wrapped TCP connections terminate on shard 0, because their per-connection state cannot be moved between shards.

All shards, including shard 0, own partitions and serve partition requests.

Request routing

Requests are routed between shards using message passing (via crossfire bounded mpsc channels), which avoids locking entirely. The routing logic splits operations into two planes:

• Metadata operations (create/delete stream/topic/user etc.) always execute on shard 0 - it is the only shard that commits metadata
• Partition operations (send_messages, poll_messages, store_consumer_offset) are routed to the shard owning that partition via a lock-free concurrent map lookup. A request that lands on a non-owning shard rides the inter-shard message bus to the owner

Partition-to-shard assignment is deterministic: the packed IggyNamespace is hashed with Murmur3, and the upper 16 bits of the hash are taken modulo the shard count (the upper bits are used because Murmur3 has weak lower bits for small integer inputs).

IggyNamespace

The IggyNamespace packs stream_id (20 bits), topic_id (12 bits), and partition_id (20 bits) into a single u64 for efficient hashing and routing. This gives maximums of 1,048,576 streams, 4096 topics per stream, and 1,000,000 partitions per topic.

CPU allocation modes

The sharding system supports multiple allocation modes via the cpu_allocation config:

• "all" - one shard per available CPU core
• A numeric value (e.g. 4) - exactly N shards pinned to cores 0..N
• A range (e.g. "5..8") - shards on specified core range
• "numa:auto" - automatically detect NUMA topology and bind accordingly
• "numa:nodes=0,1;cores=4;no_ht=true" - fine-grained NUMA control per node with hyperthread avoidance

io_uring and compio

Traditional async runtimes like tokio use epoll which is readiness-based - you ask the kernel "is this file descriptor ready?" and then perform the I/O yourself. The Linux kernel considers regular files "always ready" for epoll, which means tokio has to outsource file I/O to a blocking thread pool (up to 512 threads). This does not scale well.

io_uring is completion-based - you submit I/O requests to a submission queue (SQ), and the kernel completes them and places results in a completion queue (CQ). Both queues are lock-free ring buffers shared between user space and kernel. This is fundamentally better for disk I/O.

Iggy uses compio as its async runtime, which provides a driver-disaggregated architecture on top of io_uring (Linux) and IOCP (Windows). Each shard gets its own compio executor configured with:

• Capacity: 4096 concurrent I/O operations (by default)
• Event interval: 128 events per loop iteration
• Cooperative task running enabled

Performance: Tokio vs Thread-per-Core

The migration from Tokio to thread-per-core with compio delivered significant latency improvements across the board:

Left-right metadata

Shared metadata (streams, topics, consumer groups) uses a left-right concurrent data structure. Shard 0 holds the single write handle and is the only shard that commits metadata mutations, while every shard holds a read handle. This provides:

• Lock-free reads on all shards (no contention on the hot path)
• Strongly consistent writes serialized through shard 0

Clustering

The server is built on the Viewstamped Replication (VSR) consensus protocol. Metadata operations are committed through the VSR write-ahead log even on a single node, which is the default deployment. Multi-node clustering is enabled via the [cluster] configuration (a node roster plus --replica-id to pick the local entry). See clustering for the details and maturity caveats.

Storage structure

Having in mind that stream consists of topics, which might have one or more partitions assigned, and each partition consists of segments, we can visualize the structure of the Iggy data directory as follows:

local_data/
├── metadata/
│   └── journal.wal
├── runtime/
│   └── current_config.toml
├── logs/
│   └── iggy-server.log
├── state/
└── streams/
    └── 0/
        └── topics/
            └── 0/
                └── partitions/
                    └── 0/
                        ├── 00000000000000000000.index
                        ├── 00000000000000000000.log
                        └── offsets/

The stream, topic and partition directories are named after their numeric IDs, assigned from 0. The metadata/journal.wal file is the VSR write-ahead log that persists all metadata operations (stream/topic/user creation, etc.). The runtime/current_config.toml file captures the configuration the server actually booted with. Segment files are named by the 20-digit start offset of their first record. The .index file is created automatically and speeds up searches by keeping track of the offsets and timestamps of the records. The offsets/ directory holds the server-side consumer and consumer group offsets.

Memory pool

Iggy uses a custom memory pool with 28 buckets holding buffer sizes from 4 KiB to 512 MiB. The default pool size is 4 GiB with up to 8192 buffers per bucket. This eliminates allocation overhead on the hot path and enables zero-copy message passing between components. The pool is page-aligned (4096-byte multiples) and requires a minimum of 512 MiB.

Write pipeline

Messages flow through a multi-stage write pipeline:

1. Messages arrive on the owning shard and are buffered in the partition journal
2. A flush is triggered when either the message count threshold or the size threshold is reached - both are per-topic options set at topic creation (defaults: 1024 messages, 1 MiB)
3. The MessagesWriter uses vectored I/O with up to 1024 buffers per syscall
4. fsync per write is a per-topic option (enforce_fsync) for durability guarantees
5. When a segment reaches the topic's segment size (default 1 GiB), it is sealed and a new segment is created