If Apache Iggy is useful to you, give it a star on GitHubStar apache/iggy
Skip to Content
Apache Iggy
SDKC#
These are the docs for Apache Iggy 0.10, kept for reference and no longer updated. Read the current docs.

High-level SDK

IggyPublisher and IggyConsumer for C#, with background batching, retries and automatic offset commits.

The per-call IIggyClient API is explicit but verbose - you manage partitioning, batching, retries, and offsets yourself. The high-level IggyPublisher and IggyConsumer wrap that surface with fluent builders that handle:

  • Background sending with buffering, batching, and retries
  • Automatic stream/topic/consumer-group creation
  • Automatic offset commits (on each poll, after each received message, or manual)
  • Consuming a topic as an async stream (IAsyncEnumerable)
  • Typed message (de)serialization
  • Sending and receiving over rented, pooled buffers to avoid separate payload copies on TCP

The fragments below reuse an authenticated client and the imports in the first example. The named streams/topics must already exist unless the snippet creates them. Both abstractions can be built from an existing connected client via IggyPublisherBuilder.Create(...) / IggyConsumerBuilder.Create(...), or equivalently via the client.CreatePublisherBuilder(...) / client.CreateConsumerBuilder(...) extension methods from Apache.Iggy.Extensions.

IggyPublisher

Configure a publisher, initialize it, then send. InitAsync validates (and optionally creates) the stream and topic:

using Apache.Iggy.Contracts;
using System.Buffers;
using System.Text;
using System.Text.Json;
using Apache.Iggy;
using Apache.Iggy.Consumers;
using Apache.Iggy.Kinds;
using Apache.Iggy.Messages;
using Apache.Iggy.Publishers;

await using var publisher = IggyPublisherBuilder.Create(
    client,
    Identifier.String("my-stream"),
    Identifier.String("my-topic")
)
.WithPartitioning(Partitioning.None())
.WithBackgroundSending(enabled: true, batchSize: 100)
.WithRetry(maxAttempts: 3)
.Build();

await publisher.InitAsync();

var messages = new List<Message>
{
    new(Guid.NewGuid(), "Message 1"u8.ToArray()),
    new(0, "Message 2"u8.ToArray())
};

await publisher.SendMessagesAsync(messages);

// Drain the background queue, then dispose
await publisher.WaitUntilAllSendsAsync();

Publisher builder options

MethodDescription
WithConnection(protocol, address, login, password, receiveBufferSize = null, sendBufferSize = null, reconnectionSettings = null)Connection settings - only used when the builder creates its own client (i.e. the Create(streamId, topicId) overload, without an existing client). A builder-created TCP client auto-logs in and reuses the credentials across reconnects. For HTTP, supply an explicitly authenticated client
WithConnection(protocol, address, personalAccessToken, receiveBufferSize = null, sendBufferSize = null, reconnectionSettings = null)Same, authenticating with a personal access token instead of a username and password
WithPartitioning(partitioning)Routing strategy for produced messages (default: balanced)
CreateStreamIfNotExists(name)Auto-create the stream on InitAsync if missing
CreateTopicIfNotExists(name, topicPartitionsCount = 1, compressionAlgorithm = None, messageExpiry = TimeSpan.Zero, maxTopicSize = 0)Auto-create the topic on InitAsync if missing
WithRetry(enabled = true, maxAttempts = 3, initialDelay = 100ms, maxDelay = 10s, backoffMultiplier = 2.0)Retry background sends with exponential backoff. Direct sends surface errors
WithBackgroundSending(enabled = true, queueCapacity = 10000, batchSize = 100, flushInterval = 100ms, disposalTimeout = 5s)Queue and flush sends in the background for higher throughput
WithEncryptor(encryptor)Client-side payload encryption. Only valid on a builder-created client - for an external client, set IggyClientConfigurator.MessageEncryptor instead
SubscribeOnBackgroundError(handler)Observe background-processing errors (only fires when background sending is enabled)
SubscribeOnMessageBatchFailed(handler)Fires when a batch fails after all retries are exhausted (only with background sending - direct sends throw to the caller instead)
WithLogger(loggerFactory)Logger factory for diagnostics

WithRetry parameters

ParameterTypeDefaultDescription
enabledbooltrueWhether retry is enabled
maxAttemptsint3Total send attempts, including the first send
initialDelayTimeSpan?100msDelay before the first retry
maxDelayTimeSpan?10sMaximum delay between retries
backoffMultiplierdouble2.0Exponential backoff multiplier

WithBackgroundSending parameters

ParameterTypeDefaultDescription
enabledbooltrueWhether background sending is enabled
queueCapacityint10000Max queued send calls (one slot per SendMessagesAsync call, regardless of batch size)
batchSizeint100Flush threshold in messages. An individual send call is not split
flushIntervalTimeSpan?100msInterval at which pending messages are flushed
disposalTimeoutTimeSpan?5sHow long DisposeAsync waits for the processor to stop. Pending sends may be discarded

A batch also flushes once its accumulated payload reaches IggyPublisherConfig.BackgroundMaxBatchBytes (default 256 KiB, 0 disables the byte gate) - whichever of batchSize or the byte limit is hit first. This knob has no builder method. Set it on the config directly if needed. Await WaitUntilAllSendsAsync before disposal to finish queued sends. Disposal alone can discard them. Background send calls return an empty confirmation list before delivery, and failures are reported through the subscribed events.

Publishing with rented batches

SendMessagesAsync accepts messages with ReadOnlyMemory<byte> payloads, which can share backing storage. On hot paths, build a RentedMessageBatch instead - every payload is written into a single buffer rented from the shared array pool - and hand it to SendAsync:

using Apache.Iggy.Contracts;
using System.Text.Json;
using Apache.Iggy.Messages;

using var builder = new RentedMessageBatchBuilder(sizeHint: 4096);

foreach (var evt in events)
{
    // Serialize straight into the pooled buffer (static lambda + state = no closure allocation)
    builder.Add(evt, static (e, writer) =>
    {
        using var json = new Utf8JsonWriter(writer);
        JsonSerializer.Serialize(json, e);
    }, Guid.NewGuid());
}

var batch = builder.Build();
await publisher.SendAsync(batch);
await publisher.WaitUntilAllSendsAsync();

RentedMessageBatchBuilder:

MemberDescription
new RentedMessageBatchBuilder(sizeHint = 1024)Initial capacity hint (bytes) for the rented buffer
Add(payload, id = null, userHeaders = null)Copies a ReadOnlySpan<byte> payload into the shared buffer
Add(state, writePayload, id = null, userHeaders = null)Writes the payload via an Action<TState, IBufferWriter<byte>> callback directly into the buffer
CountNumber of payloads added so far
Build()Materializes the RentedMessageBatch. The builder must not be used afterwards

A null message id sends the message with id 0, letting the server assign one.

Ownership rules:

  • publisher.SendAsync(batch) takes ownership - do not dispose the batch yourself. Its buffer returns to the pool once the batch is sent (immediately, or after the background flush).
  • For a direct low-level send, pass batch.Messages to client.SendMessagesAsync(...) yourself, await it, then dispose the batch (using var batch = builder.Build()).
  • After disposal, batch.Messages and every payload's Span throw ObjectDisposedException instead of reading recycled pool memory.
  • Dispose the builder only when abandoning it before Build(). A using on the builder is safe either way (it becomes a no-op after Build()).

Typed publisher

For automatic object serialization, use IggyPublisherBuilder<T> with an ISerializer<T>:

await using var publisher = IggyPublisherBuilder<Order>.Create(
    client,
    Identifier.String("orders-stream"),
    Identifier.String("orders-topic"),
    new OrderSerializer()
).Build();

await publisher.InitAsync();
await publisher.SendAsync(new List<Order> { new(Guid.NewGuid(), 99.90m) });

record Order(Guid OrderId, decimal Amount);

class OrderSerializer : ISerializer<Order>
{
    public void Serialize(Order data, IBufferWriter<byte> writer) =>
        writer.Write(JsonSerializer.SerializeToUtf8Bytes(data));
}

Besides the collection overload, SendAsync also accepts a single item (SendAsync(order, messageId: null, userHeaders: null)) or a collection of (data, messageId, userHeaders) tuples when you need per-message ids or headers.

For JSON you don't need a custom serializer - the built-in SystemTextJsonSerializer<T> (in Apache.Iggy.Publishers) writes System.Text.Json output directly into the send buffer, with optional JsonSerializerOptions:

await using var publisher = IggyPublisherBuilder<Order>.Create(
    client,
    Identifier.String("orders-stream"),
    Identifier.String("orders-topic"),
    new SystemTextJsonSerializer<Order>()
).Build();

With background sending enabled, typed values are serialized at flush time. Keep them immutable or pass a snapshot until WaitUntilAllSendsAsync completes.

There is no built-in deserializer counterpart - consumers implement IDeserializer<T> themselves (see Typed consumer).

IggyConsumer

The consumer reads through a PartitionReader and yields one message at a time. A standalone consumer requires WithPartitions. A group consumer follows the server assignment.

using Apache.Iggy.Contracts;
using System.Text;
using Apache.Iggy;
using Apache.Iggy.Consumers;
using Apache.Iggy.Kinds;

await using var consumer = IggyConsumerBuilder.Create(
    client, Identifier.String("my-stream"), Identifier.String("my-topic"), Consumer.New(1))
    .WithPartitions(0)
    .WithPollingStrategy(PollingStrategy.Next())
    .WithPollOptions(new PollOptions { Count = 100, MaxWait = TimeSpan.FromSeconds(5) })
    .Build();
await consumer.InitAsync();
await foreach (var received in consumer.ReceiveAsync())
{
    Console.WriteLine(Encoding.UTF8.GetString(received.Message.Payload));
    await consumer.CommitAsync(received.Checkpoint!);
}

The fragment assumes a connected client. Manual commits are the default. Commit the checkpoint after processing. DisposeAsync calls ShutdownAsync to close the reader and leave the group.

Consumer groups

Use Consumer.Group("my-group") and WithConsumerGroup("my-group", createIfNotExists: true). Do not supply standalone partitions. Set WithPollOptions(new PollOptions { Commit = CommitPolicy.Auto() }) for automatic commits after processing.

AutoCommit.Default stores every 10,000 messages or five seconds. CommitAfter.Processing counts a message when enumeration advances or a ConsumeAsync handler succeeds. A handler exception ends consumption and leaves that message uncounted. CommitAfter.Marked supports explicit progress marking after concurrent work.

CommitPolicy.OnRead submits the offset before processing. It cannot be combined with a consumer encryptor. See Deferred polling.

Consumer builder options

Use WithPollOptions for count, wait, minimum count, byte budget and commit policy. Use WithPartitions for standalone partition selection, and WithPollingStrategy for the initial position. The previous WithBatchSize, WithPollingInterval, WithPartitionId and WithAutoCommitMode consumer methods are replaced.

A read error that stops a partition ends enumeration with ReadException. Inspect StopsPartition, IsResumable and Skip before selecting a new start with SeekAsync. Automatic retries do not discard an undecodable record without an application decision.

Consuming with rented buffers

ReceiveRentedAsync exposes payloads from a pooled buffer. Dispose each yielded message before moving on, and copy bytes that must survive disposal. Read limits and commit policies are the same as for ReceiveAsync.

Typed consumer

IggyConsumerBuilder<T> accepts an IDeserializer<T> and exposes ReceiveDeserializedAsync. Keep the typed builder in a variable before configuring it because inherited fluent methods return the base builder type. Use WithPollOptions to select the commit policy, and inspect each message status before accepting its progress.

See the C# examples for complete producer and consumer programs.

On this page