I have a flink application that has kafka source and iceberg table sink (on hadoop).
It looks something like:
FlinkSink.forRowData(input)
.tableLoader(tableLoader)
.append();
For the example lets say i have only 4 fields:
Id string,
Time timestamp,
Description string,
Version int
My table is partitioned by the time field date, i can receive multiple messages with the same id(and same time field) but with different description and version, i would like to override a row when i receive new message with same id and higher version, i cant guarantee that they will come corresponding to their version, is flink streaming iceberg sink provides any convinient way to do that?
I Searched about upsert and override options but both relay on message order by default and i didnt find a way to change this behaviour, tried looking on equality fields but it seems unrelated
alon steimberg is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.