I have a scenario where we need to send messages to Kafka using transactional and non-transactional producers simultaneously. In this scenario I also want to monitor the highest offset value for particular partition say 0.Highest offset is nothing but offset of last committed message.

Im using below similar function to get highest offset.

rd_kafka_resp_err_t errrr = rd_kafka_query_watermark_offsets(rks, topic, partition, &low_offset, &highest_offset, 2000);

Assume the current highest offset is 10.First I have started sending messages to Kafka using transactional producer for transaction X.After sending 5 messages using transactional producer the highest offset is still 10 because the transaction X is not yet committed.Now if I send a single message using non-transactional producer im expecting that highest offset should change to 16 or 17.

My question is still highest offset is showing 10 even after sending non-transactional message,Is this expected behaviour?
Is there any way that this highest offset gets incremented after sending non-transactional message while there is one open transaction?