Recently Thanos compactor was sharded from 1 to 3 compactors(all of them are working on different labels) to reduce the compaction backlog, but i have started getting these errors in one of the compactors:
{"caller":"compact.go:491","err":"compaction:
group 0@3321434660821436344: compact blocks
[/data/compact/0@3321434660821436344/01Hxxx1DW3
/data/compact/0@3321434660821436344/01HxxxJ5105
/data/compact/0@3321434660821436344/01HxxxRV11Y
/data/compact/0@3321434660821436344/01HxxxVDDJA
/data/compact/0@3321434660821436344/01HxxxG7ZBG
/data/compact/0@3321434660821436344/01Hxxx74QW]:
postings offset table size exceeds 4 bytes: 4591310572",
"level":"error","msg":"critical error detected; halting","ts":"2024-05-13T07:13:50.355467409Z"}
Seems that the only way to fix them is adding no-compact
marker to all the blocks in the error. This is happening for every new compaction. What is the correct way to solve this problem?
Thanos version: v0.32.5
Compactors are running on Kubernetes