Kafka

Apache Kafka

It is a high throughput messaging system

1. Download Apache Kafka from http://kafka.apache.org/downloads.html

2. Extract the zip contents to a local folder

3. Go to the Kafka installation directory/bin/windows 

4. Run Zookeeper by the following command

C:\kafka_2.11-0.8.2.1\bin\windows> zookeeper-server-start.bat ..\..\config\zookeeper.properties

5. Now run Kafka by using command prompt

C:\kafka_2.11-0.8.2.1\bin\windows> kafka-server-start.bat ..\..\config\server.properties

Creating a Kafka Topic

1.       Try to create a topic named  “test” 

2.       The replication factor 1 since there is only one Kafka server running. If you have a cluster with more than 1 Kafka servers running, you can increase the replication-factor accordingly which will increase the data availability

3.       Open a new command prompt in the location C:\kafka_2.11-0.8.2.1\bin\windows

4.       Type the command 

kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Communication between Producer and Consumer

1.       Open a new command prompt in the location C:\kafka_2.11-0.8.2.1\bin\windows

2.       To start a producer type command “kafka-console-producer.bat --broker-list localhost:9092 --topic test”.

3.       Again open a new command prompt in the same location as C:\kafka_2.11-0.8.2.1\bin\windows

4.       Now start a consumer by typing command “kafka-console-consumer.bat --zookeeper localhost:2181 --topic test”.

Other Commonly used Commands

1.       List Topics: kafka-topics.bat --list --zookeeper localhost:2181

2.       Describe Topic: kafka-topics.bat --describe --zookeeper localhost:2181 --topic [Topic Name]

3.       Read messages from beginning: kafka-console-consumer.bat --zookeeper localhost:2181 --topic [Topic Name] --from-beginning

4.       Delete Topic: kafka-run-class.bat kafka.admin.TopicCommand --delete --topic [topic_to_delete] --zookeeper localhost:2181