We have a kafka topic set with a cleanup.policy of compact.
we currently have the segment.bytes set to 1GB.
I have read the documentation but it wasnt 100% clear so I thought I could ask here.
Due to there being lots of config values, will any of these effect a compact policy, in that when a segment fills up, would it ever be eligible for deletion and hence we would loose the old data?
Just wondering what the right setup to keep them forever when the policy is compact.
Below shows our current config values:
1
If you want compaction and deletion, you need to set cleanup.policy=compact,delete
Only then would retention.ms, retention.bytes or similar configs be applied.