Write messages from Iggy streams to Amazon S3 and S3-compatible object stores such as MinIO, R2, Spaces and B2.

The S3 sink connector writes messages from Iggy streams to Amazon S3 and S3-compatible object stores (MinIO, Cloudflare R2, DigitalOcean Spaces, Backblaze B2). Messages are buffered in memory and uploaded as files, rotated by size or message count, under configurable path templates with offset ranges in the object names.

This page is a curated subset of the documentation. The canonical reference is the upstream s3_sink README in the apache/iggy repository.

Configuration

type = "sink"
key = "s3"
enabled = true
version = 0
name = "S3 sink"
path = "target/release/libiggy_connector_s3_sink"
plugin_config_format = "toml"

[[streams]]
stream = "application_logs"
topics = ["api_requests", "errors"]
schema = "json"
batch_length = 1000
poll_interval = "100ms"
consumer_group = "s3_sink"

[plugin_config]
bucket = "my-data-lake"
prefix = "iggy/raw"
region = "us-east-1"
path_template = "{stream}/{topic}/{date}/{hour}"
file_rotation = "size"
max_file_size = "8MiB"
output_format = "json_lines"

Common Options

Option	Type	Default	Description
`bucket`	string	required	S3 bucket name
`region`	string	required	AWS region, e.g. `us-east-1` (`auto` for R2)
`prefix`	string	none	Key prefix prepended to all objects
`endpoint`	string	none	Custom S3 endpoint for MinIO, R2, and other compatible stores
`path_template`	string	`{stream}/{topic}/{date}/{hour}`	Directory structure for S3 keys
`file_rotation`	string	`size`	`size` or `messages`
`max_file_size`	string	`8MiB`	Rotation threshold when `file_rotation = "size"`
`max_messages_per_file`	u64	none	Rotation threshold, required when `file_rotation = "messages"`
`output_format`	string	`json_lines`	`json_lines`, `json_array`, or `raw`
`include_metadata`	bool	`true`	Include stream/topic/partition/offset in the output
`include_headers`	bool	`false`	Include message headers in the output

Further options cover credentials (access_key_id / secret_access_key), retries (max_attempts 3, retry_delay 1s with exponential backoff and jitter), and path_style addressing (auto-enabled when endpoint is set). See the upstream s3_sink README for the full reference.

Path templates support {stream}, {topic}, {partition}, {date}, {hour}, and {timestamp}. The time-based variables derive from the first message timestamp in each buffer. File names embed the partition (zero-padded to 5 digits, to prevent cross-partition collisions) and the offset range (zero-padded to 20 digits), e.g. application_logs/api_requests/2026-03-16/14/00000-00000000000000000000-00000000000000000999.jsonl.

Credentials

Credentials resolve in order of precedence: explicit access_key_id + secret_access_key in the config (both together or neither), the standard AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN environment variables, the AWS shared credentials file (~/.aws/credentials), then container credentials or the instance profile when running on EC2/ECS/EKS.

MinIO example

[plugin_config]
bucket = "my-bucket"
region = "us-east-1"
endpoint = "http://localhost:9000"
access_key_id = "minioadmin"
secret_access_key = "minioadmin"

Delivery Semantics

The connector runtime commits consumer offsets before consume() runs and does not inspect its return value, so the effective guarantee is at-most-once: upload failures are retried only by the sink's internal retry loop, and a crash loses whatever is buffered in memory but not yet flushed to S3 (there is no write-ahead log or dead-letter queue). Size your rotation thresholds accordingly: smaller files bound the loss window, larger files upload more efficiently.

Transforms

Transforms can be applied before writing to S3. See the transforms documentation for the available types and their configuration.

S3 Sink