I have set up a 3-node Kafka KRaft cluster with each node acting as both a broker and a controller. However, when I check the metadata quorum status, each node seems to act as a separate leader instead of forming a single consistent cluster. Below are the details of my setup and configuration:

I have set up a 3-node Kafka KRaft cluster with each node acting as both a broker and a controller. However, when I check the metadata quorum status, each node seems to act as a separate leader instead of forming a single consistent cluster. Below are the details of my setup and configuration:

tl;dr

I want to test my kafka/connect/schema registry configuration i have set up locally with docker-compose. initially – i set up a connect instance with a s3 sink plugin that writes incoming json messages to s3 in avro format. I was able to send messages with the format

Assume a KAFKA PRODUCER application (C++, librdkafka, with enable.idempotence=true) which produce msgs 1,2,3,4 and 5 in order for a single topic partition.
Is there a possiblity of kafka writing (i.e. for a Consumer when reading), these msgs with a gap ?
e.g. is there a possiblity of kafka writing 1,2,5 (i.e. when consumer reads, they get 1,2,5) – 3 and 4 missing.

I am trying to setup a 3 machines Kafka cluster.
I’m currently running Kafka 3.7.1 with Scala 2.13 on premise (downloading the tgz from apache.downloads.com).

We have a kafka topic set with a cleanup.policy of compact.
we currently have the segment.bytes set to 1GB.

If I serialize an object manually as a byte array before producing it in kafka will it be serialized internally by kafka again or the kafka’s serialization will be skipped on receiving a byte array.

In kafka 2.6.1, there is a topic called test with two copies and 30 partitions. Now the partitions are expanded to 40. However, it is found that the disk capacity changes during the expansion process. After looking at the topic, it is found that there are the original 30 partitions. From two copies to three copies, why is this? What is the mechanism?
enter image description here

I am planning to write a blog around searching message(s) based on criteria. I feel there is a lack of tooling / framework in this space, while it’s a routine activity for any Kafka operation team / Development team.