I’m currently using the Confluent Kubernetes Operator for managing a Kafka cluster, along with a Kafka Connect standalone node. Within this setup, I’ve configured a Debezium PostgreSQL source connector with the heartbeat option enabled. The purpose of enabling the heartbeat option is to facilitate the cleanup of the PostgreSQL Write-Ahead Log (WAL).

Here is the relevant configuration snippet:

heartbeat.action.query: "INSERT INTO engineering.debezium_heartbeat (id, timestamp) VALUES (1,NOW()) ON CONFLICT(id) DO UPDATE SET timestamp=NOW();"
heartbeat.interval.ms: 10000

According to the Debezium documentation, a topic with the naming convention __debezium-heartbeat.<topic.prefix> should be created to handle these heartbeats. However, despite this configuration, I cannot find this topic anywhere in the Confluent UI or CLI.

Additionally, I’m encountering the following error in the logs of the Debezium PostgreSQL connector:

[ERROR] 2024-04-25 06:38:28,427 [task-thread-postgress-stg-source-0] org.apache.kafka.connect.runtime.errors.LogReporter report - Error encountered in task postgress-stg-source-0. Executing stage 'TRANSFORMATION' with class 'org.apache.kafka.connect.runtime.TransformationStage', where source record is = SourceRecord{sourcePartition={server=stg}, sourceOffset={transaction_id=null, lsn_proc=46912319294480, messageType=UPDATE, lsn_commit=46912319294480, lsn=46912319294480, txId=235351679, ts_usec=1714027098071505}} ConnectRecord{topic='__debezium-heartbeat.stg', kafkaPartition=0, key=Struct{serverName=stg}, keySchema=Schema{io.debezium.connector.common.ServerNameKey:STRUCT}, value=Struct{ts_ms=1714027108196}, valueSchema=Schema{io.debezium.connector.common.Heartbeat:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}.

Why can’t I find the __debezium-heartbeat.<topic.prefix> topic in the Confluent UI or CLI despite configuring the heartbeat option in the Debezium PostgreSQL connector?

From other threads i have seen that this topic created automaticly