Client Failover

A clustered deployment changes what a client can expect from any single node: reads are served locally where possible, writes must reach the right primary, and the SDKs follow the primary as it moves.

Where requests are served

Reads use the local replicated state by default on whichever node owns the namespace locally, followers included. A client connected to a follower can poll messages and read metadata without touching the primary.

Writes go through consensus and are admitted on the plane's primary only: metadata operations on the metadata primary, message and offset writes on the primary of that partition's consensus group. An acknowledged write returns after the required VSR commit, or returns a retryable error when the replica is changing view or catching up. HTTP ack=none returns after dispatch instead.

One caveat on follower reads: server-managed offset auto-commit replicates the polled offset through the partition consensus, which only the partition primary may do. A poll served by a follower does not advance the durable consumer offset. Auto-commit is best-effort and the poll response does not await offset persistence. On a primary, the offset submission precedes delivery of the poll reply, so it does not guarantee that the application received or processed the batch. Store offsets explicitly after processing when that ordering is required.

Leader redirection

Leader-aware SDKs discover the metadata primary. Rust binary clients follow this sequence:

1. The client connects and authenticates to its configured node, then fetches cluster metadata, which lists every node with its name, client endpoints, role, and status.
2. When the connected node isn't the leader, the client reconnects to the leader's advertised endpoint for the transport in use.
3. When the cluster is transiently leaderless, the client polls metadata every 250ms for up to 5s. If no healthy leader appears within that window, it continues on the contacted node and lets subsequent requests report their outcome.
4. Redirects are capped (three in the Rust SDK) so a flapping roster can't bounce the client forever.

The address a client is redirected to is the node's advertised_address (or the matching advertised_addresses selector), falling back to the roster ip. Getting those right matters: the client redials exactly what the metadata advertises. See Configuration for the selector rules.

Metadata and partition primaries can differ. After a TransientNotAccepted response, Rust binary clients check the metadata leader and can then try the remaining advertised endpoints in a bounded roster walk. This fallback is separate from the three-redirect limit.

The Rust implementation lives in core/sdk/src/leader_aware.rs.

Retryable errors

During a view change or while a replica catches up, writes can return transient errors. Rust binary clients retry within their request budgets. TransientNotAccepted proves that the request was never admitted and permits retry on another node. TransientNotCommitted has an uncertain outcome and is replayed with the same session and request identity, then returned if the budget expires. Disconnects and exhausted routing attempts can still reach the caller. Definitive validation and permission errors are not replayed.

SDK support

Every SDK speaks the VSR wire protocol and can connect to any node of a cluster. Client-side leader redirection is exercised by the shared leader_redirection BDD suite, which runs against a real two-node cluster:

Run the suite for one SDK with:

./scripts/run-bdd-tests.sh rust leader_redirection

For SDKs without redirection coverage, connect clients to the current primary for write-heavy workloads, or front the cluster's HTTP transport with a load balancer and rely on follower HTTP forwarding for control-plane operations and acknowledged partition writes.