With some help from the comments and this example, I’ve got it working.

Things I learned:

• For KRaft, the compose environment needs to configure a “controller” listener, in addition to any other listeners.

• Kafka will treat the config as Zookeeper unless it sees KAFKA_CONTROLLER_QUORUM_VOTERS.

• These last 4 vars on my broker image are absolutely necessary. I honestly don’t know what they do right now, but the default values didn’t work (subscriptions always failed), and the error messages I was getting were 100% not-helpful for me with my present knowledge:

  KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
  KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
  KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
  

My working compose config

The following compose config configures 3 listeners:

• a controller listener on docker-internal port 29092
• a broker listener on docker-internal port 19092
• a broker listener on host-exposed port 9092

The kafka1 service is the broker, and kafka-init creates topics on it.

services:
  kafka1:
    container_name: dotnetbus-kafka
    image: apache/kafka:3.8.0
    ports:
      - 9092:9092
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,INTERBROKER:PLAINTEXT,HOST:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'HOST://localhost:9092,INTERBROKER://kafka1:19092'
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:29092'
      KAFKA_LISTENERS: 'HOST://:9092,INTERBROKER://:19092,CONTROLLER://:29092'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'INTERBROKER'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

  kafka-init:
    container_name: dotnetbus-init-kafka
    image: apache/kafka:3.8.0
    depends_on:
      - kafka1
    command:
      - /bin/sh
      - -c
      - |
        sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --list
        echo -e 'Creating kafka topics'
        sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --create --if-not-exists --topic trade-request --replication-factor 1 --partitions 1
        sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --create --if-not-exists --topic trade-status --replication-factor 1 --partitions 1
        echo -e 'Successfully created the following topics:'
        sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --list

(Note: The depends_on ought to accompanied by a good health check, which I clearly haven’t done in this config.)

Your kafka listener needs configuring. See here for a working example.

Something like

  kafka:
    container_name: dotnetbus-kafka
    image: apache/kafka:3.8.0
    ports:
      - 9092:9092
    environment:
      KAFKA_LISTENERS: PLAIN://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAIN://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAIN:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAIN
      KAFKA_ZOOKEEPER_CONNECT: zookeeper
      KAFKA_BROKER_ID: 1
      KAFKA_SECURITY_PROTOCOL: PLAINTEXT