Networking

Iggy supports four transport protocols simultaneously, each optimized for different use cases. All stateful protocols (TCP, QUIC, WebSocket) use the same custom binary wire protocol, while HTTP provides a standard REST API.

Transport protocols

Performance ranking (lowest to highest overhead): TCP > QUIC > WebSocket > HTTP

Binary protocol

Every request and reply on the stateful transports starts with a fixed 256-byte header (the same Viewstamped Replication header layout the cluster uses internally), followed by the command payload. The iggy_binary_protocol crate provides zero-copy serialization for these headers, and foreign SDKs implement the same contract natively. See the Binary Protocol section for the full specification.

Connection handling across shards

The server runs one thread and runtime per selected shard (thread-per-core, shared-nothing), with CPU selection and optional pinning controlled by [sharding]. All listeners (every client transport) bind on shard 0 only. Shard 0's coordinator hands accepted plaintext TCP and WebSocket connections to peer shards round-robin by transferring the file descriptor at accept time. From then on the connection lives entirely on its shard. QUIC, TCP with TLS, WebSocket with TLS, and HTTP terminate on shard 0, because their per-connection state cannot move between threads. Sockets never migrate after accept: an operation that touches a partition owned by another shard rides the internal message bus instead.

TCP

TCP provides the highest performance as it works directly with binary data without the overhead of HTTP or TLS handshakes (unless TLS is enabled). The full set of keys:

[tcp]
enabled = true
address = "127.0.0.1:8090"

[tcp.tls]
enabled = false
self_signed = true
cert_file = "core/certs/iggy_cert.pem"
key_file = "core/certs/iggy_key.pem"

QUIC

QUIC runs the same binary protocol over UDP, on top of the quinn library, with TLS built in. It isn't quite as fast as TCP, but security comes out of the box (TLS is mandatory). Defaults:

[quic]
enabled = true
address = "127.0.0.1:8080"
max_concurrent_bidi_streams = 1
initial_mtu = "1200 B"
send_window = "64 MiB"
receive_window = "64 MiB"
stream_receive_window = "64 MiB"
keep_alive_interval = "10 s"
max_idle_timeout = "30 s"

[quic.certificate]
self_signed = true
cert_file = ""
key_file = ""

A few of these are deliberate rather than arbitrary:

• max_concurrent_bidi_streams = 1: the Rust SDK opens a fresh stream per command and the server accepts one at a time, so a connection carries one in-flight command. Raising the cap does not make handlers concurrent.
• initial_mtu = "1200 B" is the QUIC minimum. MTU discovery probes upward automatically, and boot rejects values below 1200.
• stream_receive_window is the per-stream slice of receive_window, equal to it because there is a single stream. Lower it if you ever raise the stream cap.
• keep_alive_interval is one third of max_idle_timeout, so two consecutive lost keep-alives fit before the idle timer closes the connection.

HTTP

The HTTP API is built on axum and provides a standard REST interface. It includes JWT authentication, CORS configuration, and optional TLS.

HTTP is the most accessible protocol but has the highest overhead due to JSON serialization and the stateless nature of HTTP (no persistent connections for consumer groups). You can find request examples in the server.http file.

The HTTP server also hosts:

• Prometheus metrics at /metrics (endpoint configurable via [http.metrics]). The route authenticates like every other read: scrapers must present a bearer credential (JWT or personal access token). The /ping API route, CORS preflight responses, and enabled /ui static assets are public. Login and refresh routes prove credentials in the request body.
• Embedded Web UI at /ui, which requires web_ui = true in [http] (off by default) on a server built with the iggy-web feature (part of the default build) and the Web UI static assets. Build the assets before compiling the server; see Configuration.

WebSocket

WebSocket provides bidirectional streaming over HTTP upgrade. Iggy uses the compio-ws crate to connect tungstenite framing to compio's completion-based I/O. The default WebSocket read buffer is 128 KiB and the default maximum message size is 64 MiB; these are separate settings.

The [websocket] section exposes the frame-tuning knobs (read_buffer_size, write_buffer_size, max_write_buffer_size, max_message_size, max_frame_size, accept_unmasked_frames). See Configuration. TLS for WebSocket has its own [websocket.tls] section.

Historical benchmark comparison from the WebSocket implementation article (AWS i3en.3xlarge, 4 producers or consumers, 40M messages, 1,000 messages per batch, fsync enabled). These measurements predate 0.9.0:

• Producer avg latency: TCP 2.61ms vs WebSocket 3.43ms (+31%)
• Consumer avg latency: TCP 0.70ms vs WebSocket 1.44ms (+106%)

TLS

Each transport configures TLS in its own section, and the self_signed semantics differ per transport:

With self_signed = false, cert_file and key_file must both exist. For production, provide proper certificate files everywhere. Ephemeral certificates change on every start, so a trust configuration pinned to one generated certificate does not survive a restart.

Heartbeat

[heartbeat]
enabled = true
interval = "30 s"

The verifier checks once per interval and releases a session whose last heartbeat is more than 1.2x the interval old, so its consumer groups rebalance off it. Requests and pings refresh the heartbeat. Only connections holding a consumer-group membership are evicted. Others are left alone until their socket closes.

The Rust, Go, Python, Node, async Java and C# TCP clients have automatic heartbeats, normally every 5 seconds once connected. This is below the default 36-second inactivity threshold; the periodic verifier means eviction does not occur at an exact 36-second deadline. The blocking Java client has no automatic heartbeat, so an idle consumer-group member must ping manually or reconnect after eviction.

Cluster networking

In cluster mode, listener ports come exclusively from each node's cluster.nodes ports entry, including the tcp_replica port for the consensus plane, and nodes can advertise different client-facing addresses per client network. Replica-to-replica traffic can be authenticated with a pre-shared key and wrapped in TLS 1.3. See Clustering configuration and Clustering security.