Steps to setup Kafka cluster on Local And Docker

In this article, we’ll see the steps to install and run Kafka cluster on Docker (Docker toolbox) or Local machine.

Apache Kafka: Quick start
Kafka Connect: Quick Start

Note: Here we are setting up a Kafka cluster with 3 brokers & one Zookeeper instance.

Setup Kafka on local machine (windows):

Download Kafka from the below link:

Apache Kafka

Once kafka is downloaded on the local machine, extract Kafka on to the directory and create couple of directories to save logs for Zookeeper and Kafka broker’s as below.

Kafka CLI commands to start Kafka cluster in local/docker (windows)
Windows commands path: ../kafka_2.12v/bin/windows

1. To start/stop Zookeeper
   - zookeeper-server-start.bat
   - zookeeper-server-stop.bat
2. To start/stop Kafka broker
   - kafka-server-start.bat
   - kafka-server-stop.bat
3. For Kafka topics
   - kafka-topics.bat
4. To create console producer
   - kafka-console-producer.bat
5. To create console consumer
   - kafka-console-consumer.bat
6. To create console consumer group
   - kafka-consumer-groups.bat

Note: The commands are same for linux machine under : ../kafka_2.12v/bin

Lets start Zookeeper and Kafka Broker/Server with Kafka CLI commands as below

## START ZOOKEEPER
zookeeper-server-start <LOCAL-KAFKA-PATH>/etc/kafka/zookeeper.properties
## START 2 KAFKA BROKERS (on port - 9091, 9092: configured in server properties)
kafka-server-start <LOCAL-KAFKA-PATH>/etc/kafka/server-0.properties
kafka-server-start <LOCAL-KAFKA-PATH>/etc/kafka/server-1.properties
## CREATE TOPIC - NUMBERS
kafka-topics --bootstrap-server localhost:9091 --create --topic squaredNumbers --partitions 3 --replication-factor 2
## TOPICS LIST FROM CURRENT CLUSTER
kafka-topics --bootstrap-server localhost:9091 --list
## DESCRIBE TOPIC "NUMBERS"
kafka-topics --bootstrap-server localhost:9091 --topic  numbers --describe

Setup Kafka Cluster on Docker Desktop/Docker Toolbox:

Docker installation:
https://docs.docker.com/docker-for-windows/install/
https://docs.docker.com/docker-for-windows/install-windows-home/
https://docs.docker.com/docker-for-mac/install/

Execute any of the below commands to verify docker installation

docker -v
docker --version
docker version

Let’s get into Docker terminal or command prompt and use docker pull command with kafka-image-name “cp-kafka”

Docker hub — kafka image

Start Zookeeper and Kafka Broker on docker container

Lets start by creating a topic “numbers” with partitions — 3 and replication factor as 2.

Note: “kafka-1” is a docker container name on which Kafka broker is running.

IP: 192.168.99.100 : Docker toolbox default IP

## CREATE TOPIC - NUMBERS
docker exec kafka-1 kafka-topics --bootstrap-server 192.168.99.100:9092 --create --topic numbers --partitions 3 --replication-factor 2
## TOPICS LIST FROM CURRENT CLUSTER
docker exec kafka-1 kafka-topics --bootstrap-server 192.168.99.100:9092 --list
## DESCRIBE NUMBERS TOPIC
docker exec kafka-1 kafka-topics --bootstrap-server 192.168.99.100:9091 --topic  numbers --describe

References:

• Setup Kafka on local
• Setup Kafka on Docker