Apache Kafka is a popular distributed streaming platform used for building real-time data pipelines and streaming applications. Running Kafka locally for development or testing can be a hassle, but Docker Compose makes it easy. In this guide, we’ll walk you through setting up Kafka and ZooKeeper containers using Docker Compose.
- Docker installed on your system.
- Basic knowledge of Docker and Kafka.
Step 1: Create a Docker Compose File
We start by creating a
docker-compose.yml file. This file defines the Kafka and ZooKeeper services. You can use your favorite text editor to create this file.
version: '3' services: zookeeper: image: confluentinc/cp-zookeeper:latest environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 ports: - 22181:2181 kafka: image: confluentinc/cp-kafka:latest depends_on: - zookeeper ports: - 29092:29092 environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
In this file, we define two services:
kafka service depends on
Step 2: Start Kafka Containers
Open a terminal and navigate to the directory where your
docker-compose.yml file is located. Run the following command to start the Kafka containers:
docker-compose up -d
-d flag runs the containers in detached mode, allowing you to continue working in the same terminal.
Step 3: Stopping Kafka Containers
To stop the Kafka containers, return to the terminal and run:
This command will stop and remove the containers.
Running Apache Kafka with Docker Compose is a convenient way to set up a local Kafka environment for development and testing. It eliminates the need for manual installation and configuration. By following the steps in this guide, you can easily run Kafka and ZooKeeper in Docker containers on your system. Happy streaming!