Observability

The connector runtime exposes its health through logs, Prometheus metrics, a stats endpoint, OpenTelemetry export, and per-source state files.

Logging

The runtime logs via the tracing crate. The output format is configured in the [logging] section:

[logging]
format = "text" # "text" (default) or "json"

Two per-connector flags in the connector configuration files add more detail:

• verbose: logs additional per-batch details for the connector.
• benchmark: emits per-batch timing events on the iggy_connectors::benchmark tracing target.

type = "source"
key = "random"
# ...
verbose = false
benchmark = true

Prometheus metrics

Metrics are served by the runtime HTTP API when the [http.metrics] section is enabled (the HTTP API itself is on by default at 127.0.0.1:8081):

[http.metrics]
enabled = false
endpoint = "/metrics"

Runtime-level gauges:

• iggy_connectors_sources_total - configured source connectors.
• iggy_connectors_sources_running - sources in the Running status.
• iggy_connectors_sinks_total - configured sink connectors.
• iggy_connectors_sinks_running - sinks in the Running status.

Per-connector counters, labeled with connector_key and connector_type (source or sink):

• iggy_connector_messages_produced_total - messages received from a source plugin's poll.
• iggy_connector_messages_sent_total - messages sent to Iggy (source).
• iggy_connector_messages_consumed_total - messages consumed from Iggy (sink).
• iggy_connector_messages_processed_total - messages processed and delivered to the sink plugin.
• iggy_connector_messages_filtered_total - messages intentionally dropped by transforms.
• iggy_connector_errors_total - errors encountered.

Per-batch stage timings are recorded in the iggy_connector_stage_duration_seconds histogram, labeled with connector_key, connector_type, and stage (prepare, ffi, decode, iggy_send, state_save, total), with buckets ranging from 50 microseconds to 5 seconds.

Runtime stats

GET /stats on the HTTP API returns a JSON snapshot of the runtime: version, process ID, CPU and memory usage (cgroup-aware in containers), uptime, the source/sink totals, and a connectors array with per-connector details - key, name, type, plugin version, enabled, message counters, error count, and the current status.

The connector statuses are Starting, Running, Stopping, Stopped, and Error. A connector whose plugin fails to load (missing library, bad configuration) is reported with the Error status and the failure message, without blocking the remaining connectors.

OpenTelemetry

The [telemetry] section enables OTLP export of logs and traces:

[telemetry]
enabled = false
service_name = "iggy-connectors"

[telemetry.logs]
transport = "grpc" # "grpc" or "http"
endpoint = "http://localhost:4317"

[telemetry.traces]
transport = "grpc" # "grpc" or "http"
endpoint = "http://localhost:4317"

State files

Source connectors can persist their position (see the SDK documentation) into the directory configured in the [state] section:

[state]
path = "local_state"

Each source gets one file named source_{key}.state (e.g. local_state/source_random.state) holding the raw state bytes - by convention MessagePack produced by the SDK helpers. Sinks have no state files. Their progress lives in Iggy consumer groups.

The write is crash-safe: the state is written and fsynced to a .tmp file, atomically renamed over the previous file, and the directory entry is fsynced. Files are created with owner-only permissions (0600), since state may carry cursors or tokens. An empty or missing state file means the source starts fresh - deleting a state file resets that source's position.