I’m working on macOS, and have the following docker-compose.yaml to get Kafka up and running:

version: '3'

services:

  kafka:
    image: 'bitnami/kafka:latest'
    container_name: kafka_broker
    ports:
      - '9092:9092'
      - '9094:9094'
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_NODE_ID=1
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka:9093
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER//:9093,EXTERNAL://:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,EXTERNAL://127.0.0.1:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_BROKER_ID=1
      - [email protected]
      - KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
      - KAFKA_CFG_NUM_PARTITIONS=2

    volumes:
      - ./kafka/data:/bitnami/kafka

I execute this with docker-composer run kafka bash.

I want to set up a Kafka Producer using the Python kafka library, so I set up the following in sample.py:

producer = KafkaProducer(bootstrap_servers=["localhost:9092"],
                         value_serializer=lambda m: json.dumps(m).encode('ascii'))

I open up another tab on my terminal, and when I run sample.py, I get the following error: kafka.errors.NoBrokersAvailable: NoBrokersAvailable. I go check on DockerDesktop and see that I do indeed have the container running. In the Inspect tab of DockerDesktop, I see that "Config.ExposedPorts" has a value of "9092/tcp": {}.

Changing the port to 9093 or 9094 doesn’t help. The articles about doing this task which I saw didn’t seem to think that the IP address of the container was important, but I gave replacing localhost with the container IP address a try and had the same error.

I am running the python script from within a virtual environment, but I don’t believe that to be the issue.

When I look around, I see some examples of compose files where most of the environment fields begin with KAFKA_CFG, and other examples where they begin with just KAFKA, but most discussions I’ve seen range from 2017 to 2021, and I’m not sure if this is reflective of some major syntax overhaul or what.

I am not that familiar with Docker, so I am at a loss at this point.