Monitoring
Scrape the server's Prometheus metrics: the token the endpoint needs, what it exports, and which metrics are worth alerting on.
The server exports Prometheus metrics on its HTTP API at /metrics. Both the HTTP API and the metrics endpoint are on by default. See [http.metrics] to change the path or turn it off.
This page covers the server's own metrics. The connectors runtime has a separate endpoint with its own metric names. See Connectors observability.
Get a token for the scraper
/metrics needs a bearer token, like every other read on the HTTP API. Any logged-in user can scrape it and gets the aggregate gauges. The per-topic series need the read_servers or manage_servers permission. There's no separate monitoring permission. Prometheus can't log in, so create a personal access token and put it in the scrape config:
iggy pat create prometheus 90dThe token is printed once. The Prometheus scrape config is in [http.metrics].
The token can read every metric the server exports, so store it like any other secret: in a Kubernetes Secret or your secrets manager, not in a file you commit. With the Helm chart, set it through server.serviceMonitor.authorization. See Docker & Helm.
What's exported
The core metrics are below. The gauges are read fresh on every scrape.
| Metric | Type | What it counts |
|---|---|---|
http_requests_total | counter | HTTP API requests |
streams, topics, partitions, segments | gauge | Current number of each |
messages | gauge | Messages stored |
users | gauge | Users |
clients | gauge | Connected clients |
The names have no iggy_ prefix, so watch for clashes if Prometheus scrapes other exporters with similar names.
Each shard also exports its own lower-level metrics, labelled shard="<id>". These cover the write-ahead log (the partition_wal_* series), partition housekeeping (partitions_*), and requests or frames the server refused or dropped. To see the full list with descriptions, scrape a running server:
curl -s -H "Authorization: Bearer <token>" http://localhost:3000/metrics | grep '^# HELP'What to alert on
frame_drops_total{variant="forward_client_send"}. Any increase means a client never got a reply to a request. Most otherframe_drops_totalvariants are retried or repaired automatically. This series only appears after the first drop, so on a healthy server it won't be in the scrape at all. Write the alert so that a missing series counts as no drops.stats_rollup_underflows_total. It goes up when a stream, topic or partition total would have dropped below zero and was held at zero instead. A few during deletes or restores are expected. A steady rise at other times is worth a look.
Grafana and OpenTelemetry
Add Prometheus as a data source in Grafana, then build panels from the metric names above.
The server doesn't export metrics over OTLP. To bring them into an OpenTelemetry pipeline, scrape /metrics with the collector's Prometheus receiver, using the same bearer token. Server export of logs and traces over OTLP isn't available yet. See [telemetry].
Topic usage and replica progress
topic_size_bytes, topic_messages and topic_max_size_bytes use stream and topic labels. The size-cap series is absent for unlimited topics. Renamed or deleted topics disappear from the next scrape. Retention can keep more bytes than the configured cap, so a usage ratio is not a hard storage boundary.
Replica counters can lag a successful send until a prepare or commit broadcast updates them. Compare node-local observations with routing and view-change events before treating a transient difference as data loss. Repair-ring metrics report retained batch bytes and released-entry accounting, not total process memory.
