We have many projects related to Kafka. On one of them, a couple of years ago they raised a cluster of 3 Kafka with version 3.3.1. And everything in general was fine, when suddenly I recently planned to raise a connector on it from the database and before that I decided to look at the logs.. And so there is an endless change of controllers and errors, and so on endlessly.. At the same time, I can create a topic and delete it too. Clients and the sender interact without problems, they do not observe any errors.

May 31 18:53:53 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:53,651] WARN [Controller 0] createTopics: failed with unknown server exception NoSuchElementException at epoch 576140 in 231 us.  Renouncing leadership and reverting to the last committed offset 71861715. (org.apache.kafka.controller.QuorumController)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: java.util.NoSuchElementException
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: #011at org.apache.kafka.timeline.SnapshottableHashTable$CurrentIterator.next(SnapshottableHashTable.java:167)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: #011at org.apache.kafka.timeline.SnapshottableHashTable$CurrentIterator.next(SnapshottableHashTable.java:139)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: #011at org.apache.kafka.timeline.TimelineHashSet$ValueIterator.next(TimelineHashSet.java:120)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: #011at org.apache.kafka.controller.ReplicationControlManager.validateNewTopicNames(ReplicationControlManager.java:799)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: #011at org.apache.kafka.controller.ReplicationControlManager.createTopics(ReplicationControlManager.java:567)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: #011at org.apache.kafka.controller.QuorumController.lambda$createTopics$7(QuorumController.java:1832)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: #011at org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:767)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: #011at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: #011at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: #011at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: #011at java.base/java.lang.Thread.run(Thread.java:829)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:53,651] INFO [RaftManager nodeId=0] Received user request to resign from the current epoch 576140 (org.apache.kafka.raft.KafkaRaftClient)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:53,652] INFO [RaftManager nodeId=0] Completed transition to ResignedState(localId=0, epoch=576140, voters=[0, 1, 2], electionTimeoutMs=1917, unackedVoters=[1, 2], preferredSuccessors=[2, 1]) (org.apache.kafka.raft.QuorumState)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:53,662] INFO [RaftManager nodeId=0] Completed transition to Unattached(epoch=576141, voters=[0, 1, 2], electionTimeoutMs=1724) (org.apache.kafka.raft.QuorumState)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:53,662] INFO [Controller 0] In the new epoch 576141, the leader is (none). (org.apache.kafka.controller.QuorumController)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:53,665] INFO [RaftManager nodeId=0] Completed transition to Voted(epoch=576141, votedId=2, voters=[0, 1, 2], electionTimeoutMs=1652) (org.apache.kafka.raft.QuorumState)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:53,665] INFO [RaftManager nodeId=0] Vote request VoteRequestData(clusterId='NmY4ODViNzcxZjVkNGY0MQ', topics=[TopicData(topicName='__cluster_metadata', partitions=[PartitionData(partitionIndex=0, candidateEpoch=576141, candidateId=2, lastOffsetEpoch=576140, lastOffset=71863380)])]) with epoch 576141 is granted (org.apache.kafka.raft.KafkaRaftClient)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:53,686] ERROR [Controller 0] createTopics: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:53,930] INFO [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested disconnect from node 2 (org.apache.kafka.clients.NetworkClient)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:53,967] INFO [RaftManager nodeId=0] Completed transition to FollowerState(fetchTimeoutMs=2000, epoch=576141, leaderId=2, voters=[0, 1, 2], highWatermark=Optional.empty, fetchingSnapshot=Optional.empty) (org.apache.kafka.raft.QuorumState)
May 31 18:53:53 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:53,968] INFO [Controller 0] In the new epoch 576141, the leader is 2. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,031] INFO [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new controller, from now on will use broker server_name.frnx:9093 (id: 2 rack: null) (kafka.server.BrokerToControllerRequestThread)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,033] INFO [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested disconnect from node 2 (org.apache.kafka.clients.NetworkClient)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,033] INFO [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new controller, from now on will use broker server_name.frnx:9093 (id: 2 rack: null) (kafka.server.BrokerToControllerRequestThread)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,083] INFO [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new controller, from now on will use broker server_name.frnx:9093 (id: 2 rack: null) (kafka.server.BrokerToControllerRequestThread)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,086] INFO [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested disconnect from node 2 (org.apache.kafka.clients.NetworkClient)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,086] INFO [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new controller, from now on will use broker server_name.frnx:9093 (id: 2 rack: null) (kafka.server.BrokerToControllerRequestThread)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,136] INFO [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new controller, from now on will use broker server_name.frnx:9093 (id: 2 rack: null) (kafka.server.BrokerToControllerRequestThread)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,139] INFO [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested disconnect from node 2 (org.apache.kafka.clients.NetworkClient)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,139] INFO [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new controller, from now on will use broker server_name.frnx:9093 (id: 2 rack: null) (kafka.server.BrokerToControllerRequestThread)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,200] INFO [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new controller, from now on will use broker server_name.frnx:9093 (id: 2 rack: null) (kafka.server.BrokerToControllerRequestThread)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,203] INFO [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested disconnect from node 2 (org.apache.kafka.clients.NetworkClient)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,203] INFO [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new controller, from now on will use broker server_name.frnx:9093 (id: 2 rack: null) (kafka.server.BrokerToControllerRequestThread)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,253] INFO [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new controller, from now on will use broker server_name.frnx:9093 (id: 2 rack: null) (kafka.server.BrokerToControllerRequestThread)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,268] INFO [RaftManager nodeId=0] Become candidate due to fetch timeout (org.apache.kafka.raft.KafkaRaftClient)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,272] INFO [RaftManager nodeId=0] Completed transition to CandidateState(localId=0, epoch=576142, retries=1, electionTimeoutMs=1297) (org.apache.kafka.raft.QuorumState)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,272] INFO [Controller 0] In the new epoch 576142, the leader is (none). (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,273] INFO [RaftManager nodeId=0] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,273] WARN [RaftManager nodeId=0] Connection to node 1 (server_name.frnx/212.224.118.39:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,286] INFO [RaftManager nodeId=0] Completed transition to Leader(localId=0, epoch=576142, epochStartOffset=71863381, highWatermark=Optional.empty, voterStates={0=ReplicaState(nodeId=0, endOffset=Optional.empty, lastFetchTimestamp=-1, lastCaughtUpTimestamp=-1, hasAcknowledgedLeader=true), 1=ReplicaState(nodeId=1, endOffset=Optional.empty, lastFetchTimestamp=-1, lastCaughtUpTimestamp=-1, hasAcknowledgedLeader=false), 2=ReplicaState(nodeId=2, endOffset=Optional.empty, lastFetchTimestamp=-1, lastCaughtUpTimestamp=-1, hasAcknowledgedLeader=false)}) (org.apache.kafka.raft.QuorumState)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,492] ERROR [Controller 0] createTopics: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,524] ERROR [Controller 0] processBrokerHeartbeat: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,546] ERROR [Controller 0] createTopics: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,577] ERROR [Controller 0] processBrokerHeartbeat: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,599] ERROR [Controller 0] createTopics: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,629] ERROR [Controller 0] processBrokerHeartbeat: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,682] ERROR [Controller 0] createTopics: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,682] ERROR [Controller 0] processBrokerHeartbeat: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,716] INFO [RaftManager nodeId=0] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,717] WARN [RaftManager nodeId=0] Connection to node 1 (server_name.frnx/212.224.118.39:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,735] ERROR [Controller 0] createTopics: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,492] ERROR [Controller 0] createTopics: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,524] ERROR [Controller 0] processBrokerHeartbeat: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,546] ERROR [Controller 0] createTopics: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,577] ERROR [Controller 0] processBrokerHeartbeat: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,599] ERROR [Controller 0] createTopics: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,629] ERROR [Controller 0] processBrokerHeartbeat: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,682] ERROR [Controller 0] createTopics: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,682] ERROR [Controller 0] processBrokerHeartbeat: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,716] INFO [RaftManager nodeId=0] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,717] WARN [RaftManager nodeId=0] Connection to node 1 (server_name.frnx/212.224.118.39:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,735] ERROR [Controller 0] createTopics: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,737] ERROR [Controller 0] processBrokerHeartbeat: unable to start processing because of NotControllerException. (org.apache.kafka.controller.QuorumController)
May 31 18:53:54 server_name kafka-server-start.sh[1341606]: [2024-05-31 18:53:54,745] INFO [Controller 0] Becoming the active controller at epoch 576142, committed offset 71861715, committed epoch 574475 (org.apache.kafka.controller.QuorumController)

What to do, how to stop these massive errors?

I tried rebooting all the nodes one by one. I tried to synchronize all topics and there are no errors with this.