Kafka Operations and Monitoring

Every tool below ships in Kafka’s bin/ directory (bin/kafka-.sh on Linux/macOS, bin\windows\kafka-.bat on Windows) and, since Kafka 3.0, talks to the cluster directly over the Kafka protocol via --bootstrap-server — none of them need direct access to the controller quorum’s storage.

The LAG column above — log-end offset minus the group’s committed offset, per partition — is exactly what "Consumer lag" below defines, expressed as a one-off snapshot instead of a continuously scraped metric; running this command is the fastest way to check a specific group by hand.

Kafka Architecture already covers what a partition is — the unit of ordering, parallelism, and per-consumer-group throughput; this is the operational question of how many a topic should actually have. Three factors usually decide it: the target aggregate throughput divided by what one partition can sustain (a few or a few dozen MB/s depending on hardware, replication factor, and compression — benchmark rather than assume); the number of consumer instances that must be able to read the topic in parallel within a single group, since a topic with P partitions can never keep more than P consumers in one group busy at once (see Kafka Consumers); and per-partition overhead on the brokers and controller — open file handles, replication traffic, and metadata records — which is what stops "just create thousands of partitions to be safe" from being free. kafka-topics.sh --alter --partitions can only increase a topic’s partition count, never reduce it, and increasing it reshuffles which partition a given key hashes to for any producer using the default partitioner — so it is worth erring generously (over-provisioning within reason) up front rather than treating the initial count as easily reversible later.

Most topic-, broker-, and client-level settings can be changed on a live cluster with kafka-configs.sh --alter, without restarting a single process — the change is written to the metadata log and picked up by every broker that needs it. This is what makes tuning retention, throttling a noisy client, or fixing a too-low min.insync.replicas an operational action rather than a deployment.

Not every setting is dynamic — some, such as a broker’s listener list or log directories, are read once at startup — but the common levers operators reach for repeatedly (retention, compaction thresholds, min.insync.replicas, per-client quotas) all are.

Kafka exposes its internal state as JMX MBeans; whatever metrics pipeline scrapes them (Prometheus’s JMX exporter, Datadog, CloudWatch agent, and so on), these are the ones worth an alert rather than just a dashboard panel.

Lag is a partition’s log-end offset minus a consumer group’s last committed offset on that partition — how many records the group still has to catch up on. It is the single most useful number for answering "is this consumer keeping up," independent of whether the consumer process looks healthy: a consumer can be alive, polling, and heartbeating normally while still falling further behind every minute if it can’t process records as fast as they arrive. kafka-consumer-groups.sh --describe (shown above) reports it as a point-in-time snapshot; the same figure is available continuously per consumer instance as the records-lag-max client metric (kafka.consumer:type=consumer-fetch-manager-metrics,client-id=…​), which is what an alerting pipeline should scrape rather than polling the CLI tool on a cron.

Quotas cap how much of a broker’s resources one client is allowed to consume, so a single noisy or misbehaving client can’t starve every other client sharing the cluster. Kafka enforces three kinds, all set the same way as any other dynamic configuration via kafka-configs.sh --entity-type clients|users --alter --add-config: a produce byte-rate quota (producer_byte_rate), a consume byte-rate quota (consumer_byte_rate), and a request quota (request_percentage) that limits the fraction of a broker’s request-handling threads a client may occupy, independent of the bytes it moves. Quotas can be scoped to a client-id, a user (once authentication is in place — see Kafka Security), or the combination of both, and a default can be set at any of those levels with --entity-default so that new clients get a sane cap automatically instead of running unthrottled until someone notices.

MirrorMaker 2 replicates topics, consumer group offsets, topic configuration, and ACLs from a source cluster to a target cluster, running as a set of connectors (MirrorSourceConnector, MirrorCheckpointConnector, MirrorHeartbeatConnector) on top of Kafka Connect's distributed-mode worker framework rather than as a bespoke replication process. The original MirrorMaker (kafka-mirror-maker.sh, sometimes called "MirrorMaker 1"), a much simpler tool that only copied records and predates this connector-based architecture, was deprecated in Kafka 3.0 under KIP-720 and removed entirely in Kafka 4.0 — MirrorMaker 2 is the only mirroring tool Kafka itself ships today.

A replication flow is configured as <source-cluster>→<target-cluster>.enabled=true, and which topics it covers is a regex allowlist (topics). Two topologies cover most needs: active/passive, a single one-way flow used for disaster recovery, where the target cluster is a standby that is promoted only on failover; and active/active, a pair of flows running in both directions so that either cluster can serve local reads and writes with the other kept in sync — typically for geographic locality (clients in each region talking to their nearest cluster). Active/active needs a way to stop a record from bouncing back and forth forever between the two flows: MirrorMaker 2’s default DefaultReplicationPolicy renames a topic as it crosses — a topic named orders on cluster us-west arrives on us-east as us-west.orders — so a flow never re-mirrors a topic that already carries the other cluster’s prefix.

MirrorCheckpointConnector additionally replicates consumer group offsets (translated between the two clusters' offset spaces) so a consumer failing over from one cluster to the other can resume close to where it left off, and MirrorHeartbeatConnector writes a periodic heartbeat topic used to monitor whether a flow is actually making progress. See Apache Kafka Documentation — Geo-Replication (Cross-Cluster Data Mirroring) for the full connector configuration reference, offset translation details, and fan-out/aggregation topologies beyond the two covered here.

Kafka Architecture introduced tiered storage’s local/remote split conceptually; operating it is two separate switches. The cluster-wide remote.log.storage.system.enable flag is a broker configuration (set in server.properties, taking effect on that broker’s next restart, not a live kafka-configs.sh change) that turns the feature on for the cluster; enabling it per topic is then an ordinary dynamic change:

Once enabled, reads for offsets still in the local tier are served exactly as before; reads for older offsets are fetched transparently from the remote tier through the pluggable RemoteStorageManager — a kafka-console-consumer.sh --from-beginning against a long-retention topic is a simple way to confirm both tiers are being read correctly end to end. See Apache Kafka Documentation — Tiered Storage for the full configuration reference, including how disabling tiered storage on a topic that already has remote data is handled.

Kafka’s own documentation walks running a broker via the official images: apache/kafka (JVM-based, published from Kafka 3.7 on) and the experimental, GraalVM-native apache/kafka-native (from 3.8 on, explicitly called out as suitable for local development and testing only, not production) — Getting Started with Messaging already uses the former for the single-node docker run quickstart; see Apache Kafka Documentation — Docker for both images and the multi-broker Docker Compose examples in Kafka’s own repository it links to.

Running and upgrading a cluster of brokers by hand — coordinating rolling restarts, certificate rotation, storage, and scaling — is a substantial operational burden most teams don’t take on directly. Strimzi is the most common answer for running Kafka on Kubernetes: an operator that manages brokers, controllers, topics, users, and MirrorMaker 2 flows as Kubernetes custom resources (Kafka, KafkaTopic, KafkaUser, KafkaMirrorMaker2), reconciling the running cluster to match them the way any Kubernetes operator does. Fully managed services — Confluent Cloud, Amazon MSK, and similar offerings from other cloud providers — go a step further and remove broker operation from the team entirely, at the cost of losing direct control over broker configuration and upgrade timing. Which of the three (self-managed, Strimzi-on-Kubernetes, fully managed) fits best is mostly a question of how much operational capacity a team already has and how much control over broker internals it actually needs — all three run the same Kafka protocol described in Protocols, so client code does not change between them.