{"id":65356,"date":"2020-08-31T15:46:12","date_gmt":"2020-08-31T15:46:12","guid":{"rendered":"https:\/\/www.baeldung.com\/?p=86420"},"modified":"2020-08-31T15:46:12","modified_gmt":"2020-08-31T15:46:12","slug":"listing-kafka-consumers","status":"publish","type":"post","link":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/2020\/08\/31\/listing-kafka-consumers\/","title":{"rendered":"Listing Kafka Consumers"},"content":{"rendered":"

<\/p>\n

1. Overview<\/h2>\n
<\/div>\n

In this quick tutorial, we’ll learn how to list Kafka consumer groups and also take a peek at their details.<\/p>\n

2. Prerequisites<\/h2>\n
<\/div>\n

To run the examples in this tutorial, we’ll need a Kafka cluster to send our requests to. This can be a full-blown Kafka cluster running on a production environment, or it can be a test-specific, single-instance Kafka cluster<\/a>.<\/p>\n

For the sake of simplicity, we’re going to assume that we have a single-node cluster listening to port 9092 with a Zookeeper<\/a> instance listening to the 2181 port on the localhost.<\/p>\n

Furthermore, note that we’re running all example commands from the Kafka installation directory.<\/p>\n

3. Adding Topics and Consumers<\/h2>\n
<\/div>\n

Before listing the consumers on a particular Kafka cluster, let’s add a few topics<\/a> first using the kafka-topics.sh<\/em> shell script:<\/p>\n

$ .\/bin\/kafka-topics.sh --create --topic users.registrations --replication-factor 1 \\ \r\n  --partitions 2 --zookeeper localhost:2181\r\n$ .\/bin\/kafka-topics.sh --create --topic users.verfications --replication-factor 1 \\ \r\n  --partitions 2 --zookeeper localhost:2181<\/code><\/pre>\n
Now, we need to add a few consumer groups<\/strong>, too. The simplest way is to use the console consumer bundled in Kafka distributions<\/strong>:<\/p>\n
$ .\/bin\/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic users.registrations --group new-user\r\n$ .\/bin\/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic users.registrations --group new-user<\/code><\/pre>\n
Here, we’ve used the kafka-console-consumer.sh\u00a0<\/em>shell script to add two consumers listening to the same topic. These consumers are in the same group, so the messages from topic partitions will be spread across the members of the group<\/strong>. This way we can implement the competing consumers<\/a>\u00a0pattern in Kafka.<\/p>\n
Let’s consume from another topic, too:<\/p>\n
$ .\/bin\/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic users.verifications<\/code><\/pre>\n
Since we didn’t specify a group for the consumer, the console consumer created a new group, with itself as the lone member<\/strong>.<\/p>\n
We’ll see this new group in the next section, where we’ll learn how to list consumers and consumer groups on the Kafka cluster.<\/p>\n
4. Listing Consumers<\/h2>\n
<\/div>\n
To list the consumers in the Kafka cluster, we can use the kafka-consumer-groups.sh\u00a0<\/em>shell script. The\u00a0–list\u00a0<\/em>option will list all the consumer groups<\/strong>:<\/p>\n
$ .\/bin\/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092\r\nnew-user\r\nconsole-consumer-40123<\/code><\/pre>\n
In addition to the\u00a0–list\u00a0<\/em>option, we’re passing the\u00a0–bootstrap-server\u00a0<\/em>option to specify the Kafka cluster address. We have three individual consumers in two groups, so the result contains only two groups.<\/p>\n
To see the members of the first group<\/strong>, we can use the\u00a0“–group <name> –describe –members” <\/em>options:<\/p>\n
$ .\/bin\/kafka-consumer-groups.sh --describe --group new-user --members --bootstrap-server localhost:9092\r\nGROUP           CONSUMER-ID                    HOST            CLIENT-ID            #PARTITIONS\r\nnew-user        consumer-new-user-1-b90...     \/127.0.0.1      consumer-new-user-1  1\r\nnew-user        consumer-new-user-1-af8...     \/127.0.0.1      consumer-new-user-1  1<\/code><\/pre>\n
Here, we can see that there are two individual consumers in our new-user<\/em> group, each consuming from one partition.<\/p>\n
If we omit the –members\u00a0<\/em>option, it’ll list the consumers in the group, the partition number each is listening to, and their offsets:<\/p>\n
$ .\/bin\/kafka-consumer-groups.sh --describe --group new-user --bootstrap-server localhost:9092\r\nGROUP           TOPIC                       PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG          \r\nnew-user        users.registrations         1          3               3               0              \r\nnew-user        users.registrations         0          5               5               0            <\/code><\/pre>\n
One more thing to note is that the cluster or bootstrap server address is required for this command. If we omit the cluster connection information, the shell script will throw an error<\/strong>:<\/p>\n
$ .\/bin\/kafka-consumer-groups.sh --list\r\nMissing required argument \"[bootstrap-server]\"\r\n\/\/ truncated<\/code><\/pre>\n
5. Conclusion<\/h2>\n
<\/div>\n
In this short tutorial, we added a few Kafka topics and consumer groups at first. Then, we learned how to list consumer groups and view the details for each group.<\/p>\n
The post Listing Kafka Consumers<\/a> first appeared on Baeldung<\/a>.<\/p>\n
\"\"<\/p>\n
<\/a> <\/a> <\/a> <\/a> <\/a> <\/a> <\/a> <\/a><\/p>\n
Related Stories<\/h3>\n