I have a design for an application that read from a Kafka topic processes the messages and writes to another topic.
My design is as such:
Read a batch -> send to worker pool-> worker sends to producer ->
Producer commits messages.
Will this work when using transactions.
In this case messages c,d can be committed before a,b as the processing worker might have finished it first.
I assumed it’s not possible but now confused
I didn’t try anything yet, just a design idea