Customer is not interested in using CDC/kafka approach to load data from database, instead asking to use scheduler approach.

I’m encountering an issue where my Flink job, deployed on Flink Clusters 1.19.0 and 1.19.1, fails to consume from all partitions of a Kafka topic when deployed with a checkpoint or savepoint. Despite the Flink 1.19 documentation not listing an official Kafka connector, I’m using one from Maven, which is supposed to support 1.19 bacause it’s for the 1.19 version.

I’m developing a custom source for Apache Flink 1.18 and need to ensure that the offset for each partition is correctly persisted and restored between pipeline restarts. I can’t store the offset in the data source system, so it needs to be correctly checkpointed within Flink. I’m confused about the separation between SourceSplit and SplitState.

I have the following Pipeline:

Is it possible to use Flink’s FileSource to produce elements in ascending order (by filename or timestamp of creation)?

I’m using old Flink 1.71 but I think this is not version related. I use Streaming API FinkSink to write Arvo GenericRecord to parquet files.

So I have an extremely simple streaming pipeline setup using apache flink, streaming data from kinesis, I don’t think this really matters but I am using AWS’s Managed Flink Environment, that said here is my implementation.

I’m currently working on a custom partitioner but I’m encountering some issues with my code. I’ve tried looking for documentation on the custom partitioner but have been unable to do so. If anyone has the documentation I would be extremely grateful 🙂

in our organization, we got number of systems running on flink 1.16.

I’m trying to bootstrap my operator broadcast state and keyed state using Flink’s State API, and creating a savepoint to initialize my job from.