In this quick tutorial, we’re going to see how we can list all topics in an Apache Kafka cluster.<\/p>\n
First, we’ll set up a single-node Apache Kafka and Zookeeper<\/a> cluster. Then, we’ll ask that cluster about its topics.<\/p>\n Before listing all the topics in a Kafka cluster, let’s set up a test single-node Kafka cluster in three steps:<\/p>\n First, we should make sure to download the right Kafka version from the Apache<\/a>\u00a0site.\u00a0Once the download finishes, we should extract the downloaded archive:<\/p>\n Kafka is using Apache Zookeeper to manage its cluster metadata, so we need a running Zookeeper cluster.<\/p>\n For test purposes, we can run a single-node Zookeeper instance using the zookeeper-server-start.sh<\/em>\u00a0script in the\u00a0bin<\/em>\u00a0directory:<\/p>\n This will start a Zookeeper service listening on port 2181. After this, we can use another script to run the Kafka server:<\/p>\n After a while, a Kafka broker will start. Let’s add a few topics<\/a> to this simple cluster:<\/p>\n Now that everything is ready, let’s see how we can list Kafka topics.<\/p>\n To list all Kafka topics in a cluster, we can use the\u00a0bin\/kafka-topics.sh\u00a0<\/em>shell script bundled in the downloaded Kafka distribution. All we have to do is to pass the\u00a0–list\u00a0<\/em>option along with the information about the cluster<\/strong>. For instance, we can pass the Zookeeper service address:<\/p>\n As shown above,\u00a0the\u00a0–list<\/em> option tells the kafka-topics.sh<\/em>\u00a0shell script to list all the topics. In this case, we have two topics to store user-related events. If there is no topic in the cluster, then the command will return silently without any result.<\/p>\n Also, in order to talk to the Kafka cluster, we need to pass the Zookeeper service URL using the –zookeeper\u00a0<\/em>option<\/strong>.<\/p>\n It’s even possible to pass the Kafka cluster address directly using the –bootstrap-server<\/em> option<\/strong>:<\/p>\n Our single-instance Kafka cluster listens to the 9092 port, so we specified “localhost:9092”<\/em>\u00a0as the bootstrap server. Put simply, bootstrap servers are Kafka brokers.<\/p>\n If we don’t pass the information necessary to talk to a Kafka cluster, the kafka-topics.sh<\/em>\u00a0shell script will complain with an error:<\/p>\n As shown above, the shell scripts require us to pass either the –bootstrap-server<\/em>\u00a0or\u00a0–zookeeper<\/em>\u00a0option.<\/p>\n Once we’ve found a list of topics, we can take a peek at the details of one specific topic. To do that, we can use the\u00a0“<\/em>–describe –topic <topic name>”<\/em>\u00a0combination of options<\/strong>:<\/p>\n These details include information about the specified topic such as the number of partitions and replicas, among others. Similar to other commands, we must pass the cluster information or Zookeeper address. Otherwise, we won’t be able to talk to the cluster.<\/p>\n In this short tutorial, we learned how to list all topics in a Kafka cluster. Along the way, we saw how to set up a simple, single-node Kafka cluster.<\/p>\n Currently, Apache Kafka is using Zookeeper to manage its cluster metadata. This, however, will change shortly as part of KIP-500<\/a>, as Kafka is going to have its own metadata quorum.<\/p>\n Learn how to list all kafka topics using the command line.<\/p>\n2. Setting Up Kafka<\/h2>\n
\n
$ tar xvf kafka_2.13-2.6.0.tgz<\/code><\/pre>\n$ cd kafka_2.13-2.6.0 # extracted directory\r\n$ .\/bin\/zookeeper-server-start.sh config\/zookeeper.properties<\/code><\/pre>\n$ .\/bin\/kafka-server-start.sh config\/server.properties<\/code><\/pre>\n$ bin\/kafka-topics.sh --create --topic users.registrations --replication-factor 1 \\\r\n --partitions 2 --zookeeper localhost:2181\r\n$ bin\/kafka-topics.sh --create --topic users.verfications --replication-factor 1 \\\r\n --partitions 2 --zookeeper localhost:2181<\/code><\/pre>\n3. Listing Topics<\/h2>\n
$ bin\/kafka-topics.sh --list --zookeeper localhost:2181\r\nusers.registrations\r\nusers.verfications<\/code><\/pre>\n$ .\/bin\/kafka-topics.sh --bootstrap-server=localhost:9092 --list\r\nusers.registrations\r\nusers.verfications<\/code><\/pre>\n$ .\/bin\/kafka-topics.sh --list\r\nException in thread \"main\" java.lang.IllegalArgumentException: Only one of --bootstrap-server or --zookeeper must be specified\r\n at kafka.admin.TopicCommand$TopicCommandOptions.checkArgs(TopicCommand.scala:721)\r\n at kafka.admin.TopicCommand$.main(TopicCommand.scala:52)\r\n at kafka.admin.TopicCommand.main(TopicCommand.scala)<\/code><\/pre>\n4. Topic Details<\/h2>\n
$ .\/bin\/kafka-topics.sh --bootstrap-server=localhost:9092 --describe --topic users.registrations\r\nTopic: users.registrations PartitionCount: 2 ReplicationFactor: 1 Configs: segment.bytes=1073741824\r\n Topic: users.registrations Partition: 0 Leader: 0 Replicas: 0 Isr: 0\r\n Topic: users.registrations Partition: 1 Leader: 0 Replicas: 0 Isr: 0<\/code><\/pre>\n5. Conclusion<\/h2>\n
<\/p>\n
![](\"https:\/\/assets.feedblitz.com\/i\/fblike20.png\")
<\/a> ![](\"https:\/\/assets.feedblitz.com\/i\/googleplus20.png\")
<\/a> ![](\"https:\/\/assets.feedblitz.com\/i\/pinterest20.png\")
<\/a> ![](\"https:\/\/assets.feedblitz.com\/i\/twitter20.png\")
<\/a> ![](\"https:\/\/assets.feedblitz.com\/i\/email20.png\")
<\/a> ![](\"https:\/\/assets.feedblitz.com\/i\/rss20.png\")
<\/a> ![](\"https:\/\/assets.feedblitz.com\/i\/comments20.png\")
<\/a> ![](\"https:\/\/assets.feedblitz.com\/i\/commentsrss20.png\")
<\/a> <\/div>\n\n![\\\"IT\u96fb\u8166\u88dc\u7fd2](\\\"http:\/\/itteacheritfreelance.hk\/test\/wordpress\/wp-content\/uploads\/2016\/05\/logo2-2.png\\\"){kind=logo}
\r\nwww.itsec.hk<\/a>
\r\nwww.itsec.vip<\/a>
\r\nwww.itseceu.uk<\/a>
<\/p>","protected":false},"excerpt":{"rendered":"<\/a> <\/a> <\/a> <\/a> <\/a> <\/a> <\/a> <\/a> <\/div>\n<\/div>","protected":false},"author":760,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"slim_seo":{"title":"Listing Kafka Topics - ITTeacherITFreelance.hk","description":"Learn how to list all kafka topics using the command line. "},"footnotes":""},"categories":[6],"tags":[4913],"_links":{"self":[{"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/63375"}],"collection":[{"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/users\/760"}],"replies":[{"embeddable":true,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/comments?post=63375"}],"version-history":[{"count":8,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/63375\/revisions"}],"predecessor-version":[{"id":66049,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/posts\/63375\/revisions\/66049"}],"wp:attachment":[{"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/media?parent=63375"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/categories?post=63375"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/itteacheritfreelance.hk\/wordpress\/index.php\/wp-json\/wp\/v2\/tags?post=63375"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}