How to Run Kafka Locally

Hey there, fellow Kafka enthusiasts! Running Kafka locally can be a bit of an adventure, and I’m here to make it a fun one. I’ve spent countless hours troubleshooting the ins and outs so that you don’t have to. Buckle up and let’s set up Kafka together in a way that’s simple, understandable, and, most importantly, actually works!

You might find lots of tutorials out there promising an easy setup for Kafka, but trust me, many of those come with hidden gremlins—from outdated versions to quirky networking issues. I’ve fought the battles, and now I’m sharing the magic potion: a simple Docker Compose setup that actually works without headaches. But keep in mind, this is strictly for local development and testing purposes—not for production!

What Are We Doing Here?

So, what are we building today? A local Kafka environment that includes:

• Zookeeper: Think of it as Kafka’s coordinator, keeping everything in sync.
• Kafka Broker: The main guy, storing and serving messages.
• AKHQ: A lovely UI for managing your Kafka topics—no peeking into the command line if you don’t want to!

How Does It All Work?

We’ll use Docker Compose to spin up these services. If you’re new to Docker, don’t worry—just think of it as a way to manage virtual boxes for each service we need. All services will live in a shared network called myNetwork, which lets them talk to each other seamlessly.

I’ve included AKHQ to add a bit of jolliness to the mix—it’s a neat UI tool to peek at your Kafka setup, manage topics, and see messages in real-time without having to memorize command-line Kafka commands. Trust me, it’s a lifesaver.

Here’s the setup:

docker-compose.yml

version: "3"
networks:
  myNetwork:

services:

  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
    networks:
      - myNetwork

  kafka:
    image: 'bitnami/kafka:latest'
    user: root
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_LISTENERS=PLAINTEXT://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - myNetwork
    depends_on:
      - zookeeper

  akhq:
    image: tchiotludo/akhq
    container_name: akhq
    ports:
      - "9096:8080"
    environment:
      AKHQ_CONFIGURATION: |
        akhq:
          connections:
            local:
              properties:
                bootstrap.servers: "kafka:9092"
    networks:
      - myNetwork
    depends_on:
      - kafka

Copy, Paste, and Spin It Up!

To get started, copy the above docker-compose.yml to a file in your project directory. Once that’s done, navigate to that directory in your terminal and run the following command:

docker-compose up -d

This command will download the necessary Docker images, spin up Zookeeper, Kafka, and AKHQ, and have everything up and running in just a few minutes. You can then head to http://localhost:9096 in your browser to access AKHQ, where you’ll see your Kafka instance and start creating topics, sending messages, and more.

Why This Setup Works

I’ve pieced this together because many guides out there are full of almost correct information—small version mismatches, weird network configs, or incomplete examples that leave you scratching your head. I’ve been through all of that so you don’t have to. This setup works smoothly for local development and allows you to get productive right away, without diving deep into Kafka internals unless you want to.

Keep in Mind

One last reminder—this setup is only recommended for local development and testing purposes. If you’re setting up Kafka for a real-world use case, you’d want more robust configuration, proper security, persistent data storage, and monitoring.

Now It’s Your Turn!

Give it a whirl and see how easily you can get Kafka running locally with this setup. Have fun creating topics, pushing messages, and maybe even throwing a few confetti GIFs into the mix.

Did this setup work well for you? Did you run into any issues? Drop a comment and let’s keep this Kafka journey exciting together!