Listing Kafka Consumers

1. Overview

In this quick tutorial, we’ll learn how to list Kafka consumer groups and also take a peek at their details.

2. Prerequisites

To run the examples in this tutorial, we’ll need a Kafka cluster to send our requests to. This can be a full-blown Kafka cluster running on a production environment, or it can be a test-specific, single-instance Kafka cluster.

For the sake of simplicity, we’re going to assume that we have a single-node cluster listening to port 9092 with a Zookeeper instance listening to the 2181 port on the localhost.

Furthermore, note that we’re running all example commands from the Kafka installation directory.

3. Adding Topics and Consumers

Before listing the consumers on a particular Kafka cluster, let’s add a few topics first using the kafka-topics.sh shell script:

$ ./bin/kafka-topics.sh --create --topic users.registrations --replication-factor 1 \ 
  --partitions 2 --zookeeper localhost:2181
$ ./bin/kafka-topics.sh --create --topic users.verfications --replication-factor 1 \ 
  --partitions 2 --zookeeper localhost:2181

Now, we need to add a few consumer groups, too. The simplest way is to use the console consumer bundled in Kafka distributions:

$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic users.registrations --group new-user
$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic users.registrations --group new-user

Here, we’ve used the kafka-console-consumer.sh shell script to add two consumers listening to the same topic. These consumers are in the same group, so the messages from topic partitions will be spread across the members of the group. This way we can implement the competing consumers pattern in Kafka.

Let’s consume from another topic, too:

$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic users.verifications

Since we didn’t specify a group for the consumer, the console consumer created a new group, with itself as the lone member.

We’ll see this new group in the next section, where we’ll learn how to list consumers and consumer groups on the Kafka cluster.

4. Listing Consumers

To list the consumers in the Kafka cluster, we can use the kafka-consumer-groups.sh shell script. The –list option will list all the consumer groups:

$ ./bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092
new-user
console-consumer-40123

In addition to the –list option, we’re passing the –bootstrap-server option to specify the Kafka cluster address. We have three individual consumers in two groups, so the result contains only two groups.

To see the members of the first group, we can use the “–group <name> –describe –members” options:

$ ./bin/kafka-consumer-groups.sh --describe --group new-user --members --bootstrap-server localhost:9092
GROUP           CONSUMER-ID                    HOST            CLIENT-ID            #PARTITIONS
new-user        consumer-new-user-1-b90...     /127.0.0.1      consumer-new-user-1  1
new-user        consumer-new-user-1-af8...     /127.0.0.1      consumer-new-user-1  1

Here, we can see that there are two individual consumers in our new-user group, each consuming from one partition.

If we omit the –members option, it’ll list the consumers in the group, the partition number each is listening to, and their offsets:

$ ./bin/kafka-consumer-groups.sh --describe --group new-user --bootstrap-server localhost:9092
GROUP           TOPIC                       PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG          
new-user        users.registrations         1          3               3               0              
new-user        users.registrations         0          5               5               0

One more thing to note is that the cluster or bootstrap server address is required for this command. If we omit the cluster connection information, the shell script will throw an error:

$ ./bin/kafka-consumer-groups.sh --list
Missing required argument "[bootstrap-server]"
// truncated

5. Conclusion

In this short tutorial, we added a few Kafka topics and consumer groups at first. Then, we learned how to list consumer groups and view the details for each group.

The post Listing Kafka Consumers first appeared on Baeldung.