I have a flink job with Rabbitmq source,keyby, tumbling window and aggregator function, sink.
I’m using rockDB state backend along with incremental checkpoints. Every 10s checkpointing is happening with 5s pause between checkpoints.
Constant load with same keys I used for my test. I did same test 5 times.
Following is the result of same.
1.Why is checkpoint size is increasing for same keys/same load over a period of time? – Does it mean lot of managed memory or disk space still there and compaction didn’t trigger yet?
2.Full checkpoint size has reduced in between. Is it because of compaction?
-
I have also enabled rocks DB native metrics, It shows only pending compaction? Is there any property which will show completed compaction?
-
I am using default rocks DB configuration (Flink version used is 1.18). So write buffer size is 64MB and l0 compaction will create a l1 file which will be of size 64MB.
What is the default multiplier value and also how many levels are totally available in flink’s rocks DB. -
Let’s say before memtable is getting filled with my state (i.e before it becomes 64MB), if operator needs to take snapshot how is it handled? Will it copy inflight data as it is, before memtable is flushed to .sst files? Because I’m seeing only sometimes .sst files are present. Other times a file without extension is present in checkpoints directory. Why?
-
If I reduce my diskspace will compaction run often? If so how to decide how we need for state storage?
-
what is state.backend.rocksdb.compaction.level.max-size-level-base ? document says The upper-bound of the total size of level base files in bytes. ” + “The default value is ‘256MB’.”But it is not clear for me. Is it lmax file sizes? Single lmax file size or complete lmax size?
Thanks in advance.