I am new to Flink and trying to use Flink with Kafka as data input and iceberg for storing the data.
Here are the steps I have done.
- Read A Datastream of java POJOs from kafkaSource (with a Avro schema).
- Convert the Datastream of POJOs to DataStream by using MapFunction wrapper.
- Send the DataStream to Iceberg using FlinkSink.
Everything works but My concern is with the way I am converting the DataStream to DataStream. I have a complex and pretty big POJO reference and to convert it to RowData, I had to use .setField() method for each and handle each dataType transformation like StringData,Map, List etc.
I wanted to check here, if there is a better way to do this.