Our team is using Kafka Streams version 3.6.0. We are trying to run a Kafka Streams topology over some externally managed brokers. I’m getting the following error when the Kafka Streams application tries to create its internal changelog topic.

org.apache.kafka.streams.errors.StreamsException: Could not create topic <application.id>-KSTREAM-OUTEROTHER-0000000007-store-changelog.
    at org.apache.kafka.streams.processor.internals.InternalTopicManager.getTopicPartitionInfo(InternalTopicManager.java:623)
    at org.apache.kafka.streams.processor.internals.InternalTopicManager.getNumPartitions(InternalTopicManager.java:641)
    at org.apache.kafka.streams.processor.internals.InternalTopicManager.validateTopics(InternalTopicManager.java:657)
    at org.apache.kafka.streams.processor.internals.InternalTopicManager.makeReady(InternalTopicManager.java:472)
    at org.apache.kafka.streams.processor.internals.ChangelogTopics.setup(ChangelogTopics.java:97)
    ...
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159)
    at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:1014)
    at org.apache.kafka.streams.processor.internals.StreamThread.pollPhase(StreamThread.java:962)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:766)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:617)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:579)

If we don’t have a way to give our underlying Kafka client admin permissions…
Can we provide a stable name for Kafka Streams internal changelog topic?
Setting the topic prefix doesn’t seem to quite do the trick, as we have some internal topics that are like testapplication-KSTREAM-REDUCE-STATE-STORE-0000000008-repartition.

Prior research

• Kafka Streams Internal Topic Naming
• https://kafka.apache.org/37/documentation/streams/developer-guide/dsl-topology-naming.html
• https://forum.confluent.io/t/manually-generate-internal-topic-list-based-on-topology/7638
• https://lists.apache.org/thread/ftqfz7q8qm0255krxyxp145w3l1pxpkg