Introduction:

Apache Kafka is a popular distributed streaming platform used for building real-time data pipelines and streaming applications. Running Kafka locally for development or testing can be a hassle, but Docker Compose makes it easy. In this guide, we’ll walk you through setting up Kafka and ZooKeeper containers using Docker Compose.

Prerequisites:

1. Docker installed on your system.
2. Basic knowledge of Docker and Kafka.

Step 1: Create a Docker Compose File

We start by creating a docker-compose.yml file. This file defines the Kafka and ZooKeeper services. You can use your favorite text editor to create this file.

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

In this file, we define two services: zookeeper and kafka. The kafka service depends on zookeeper.

Step 2: Start Kafka Containers

Open a terminal and navigate to the directory where your docker-compose.yml file is located. Run the following command to start the Kafka containers:

docker-compose up -d

The -d flag runs the containers in detached mode, allowing you to continue working in the same terminal.

Step 3: Stopping Kafka Containers

To stop the Kafka containers, return to the terminal and run:

docker-compose down

This command will stop and remove the containers.

Conclusion

Running Apache Kafka with Docker Compose is a convenient way to set up a local Kafka environment for development and testing. It eliminates the need for manual installation and configuration. By following the steps in this guide, you can easily run Kafka and ZooKeeper in Docker containers on your system. Happy streaming!