# S3 Sink

> Write messages from Iggy streams to Amazon S3 and S3-compatible object stores such as MinIO, R2, Spaces and B2.

Rendered page: https://iggy.apache.org/docs/connectors/sinks/s3/

Source: https://github.com/apache/iggy-website/blob/main/content/docs/connectors/sinks/s3.mdx

The S3 sink connector writes messages from Iggy streams to Amazon S3 and S3-compatible object stores (MinIO, Cloudflare R2, DigitalOcean Spaces, Backblaze B2). Messages are buffered in memory and uploaded as files, rotated by size or message count, under configurable path templates with offset ranges in the object names.

This page is a curated subset of the documentation. The canonical reference is the upstream [s3_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/s3_sink) in the `apache/iggy` repository.

## Configuration

```toml
type = "sink"
key = "s3"
enabled = true
version = 0
name = "S3 sink"
path = "target/release/libiggy_connector_s3_sink"
plugin_config_format = "toml"

[[streams]]
stream = "application_logs"
topics = ["api_requests", "errors"]
schema = "json"
batch_length = 1000
poll_interval = "100ms"
consumer_group = "s3_sink"

[plugin_config]
bucket = "my-data-lake"
prefix = "iggy/raw"
region = "us-east-1"
path_template = "{stream}/{topic}/{date}/{hour}"
file_rotation = "size"
max_file_size = "8MiB"
output_format = "json_lines"
```

### Common Options

| Option | Type | Default | Description |
| ------ | ---- | ------- | ----------- |
| `bucket` | string | **required** | S3 bucket name |
| `region` | string | **required** | AWS region, e.g. `us-east-1` (`auto` for R2) |
| `prefix` | string | none | Key prefix prepended to all objects |
| `endpoint` | string | none | Custom S3 endpoint for MinIO, R2, and other compatible stores |
| `path_template` | string | `{stream}/{topic}/{date}/{hour}` | Directory structure for S3 keys |
| `file_rotation` | string | `size` | `size` or `messages` |
| `max_file_size` | string | `8MiB` | Rotation threshold when `file_rotation = "size"` |
| `max_messages_per_file` | u64 | none | Rotation threshold, required when `file_rotation = "messages"` |
| `output_format` | string | `json_lines` | `json_lines`, `json_array`, or `raw` |
| `include_metadata` | bool | `true` | Include stream/topic/partition/offset in the output |
| `include_headers` | bool | `false` | Include message headers in the output |

Further options cover credentials (`access_key_id` / `secret_access_key`), retries (`max_attempts` `3`, `retry_delay` `1s` with exponential backoff and jitter), and `path_style` addressing (auto-enabled when `endpoint` is set). See the upstream [s3_sink README](https://github.com/apache/iggy/tree/master/core/connectors/sinks/s3_sink) for the full reference.

Path templates support `{stream}`, `{topic}`, `{partition}`, `{date}`, `{hour}`, and `{timestamp}`. The time-based variables derive from the first message timestamp in each buffer. File names embed the partition (zero-padded to 5 digits, to prevent cross-partition collisions) and the offset range (zero-padded to 20 digits), e.g. `application_logs/api_requests/2026-03-16/14/00000-00000000000000000000-00000000000000000999.jsonl`.

## Credentials

Credentials resolve in order of precedence: explicit `access_key_id` + `secret_access_key` in the config (both together or neither), the standard `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_SESSION_TOKEN` environment variables, the AWS shared credentials file (`~/.aws/credentials`), then container credentials or the instance profile when running on EC2/ECS/EKS.

### MinIO example

```toml
[plugin_config]
bucket = "my-bucket"
region = "us-east-1"
endpoint = "http://localhost:9000"
access_key_id = "minioadmin"
secret_access_key = "minioadmin"
```

## Delivery Semantics

The connector runtime commits consumer offsets before `consume()` runs and does not inspect its return value, so the effective guarantee is **at-most-once**: upload failures are retried only by the sink's internal retry loop, and a crash loses whatever is buffered in memory but not yet flushed to S3 (there is no write-ahead log or dead-letter queue). Size your rotation thresholds accordingly: smaller files bound the loss window, larger files upload more efficiently.

## Transforms

Transforms can be applied before writing to S3. See the [transforms documentation](/docs/connectors/transforms) for the available types and their configuration.