While using Apache Kafka, is it possible to get the offset of the message that has been produced?
From the Kafka documentation page it says:
each message in the partition are assigned a unique sequential id/number called Offset
Now suppose when I am producing any message I want to retrieve the offset that has been assigned to it.
1
You shouldn’t do that from a class that acts as a producer, like in most queue systems, as its responsibility is to just fire and forget the messages. The broker will do the appropriate meta data handling with id’s, offsets, whatever it needs to handle the messages in a correct manner.
You can get the offsets as a consumer of messages from a Kafka broker. If you look in the SimpleConsumer class, you’ll notice it fetches MultiFetchResponse objects that contains offsets as a list. Furthermore when you iterate the Kafka messages you’ll end up with a MessageAndOffset objects that contains both the message sent and it’s offset.
5
The producer response protocol provides the offset of the first message sent in the messageset. Provided the API/Implementation you’re using exposes this information, you should be able to retrieve it from the response to producing your message.
More here:
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceResponse
The javadoc states you can setup a callback for the send method that will be invoked once the send has been acknowledged. The send method returns a RecordMetadata that contains the offset the record was assigned to as well as other metadata. I know this is an old question but I came across it in some Kafka research I was doing. Hope this helps!