I have a **vector.toml** file with the following contents:
[sources.dummy_logs]
type = "syslog"
address = "0.0.0.0:9000"
mode = "udp"
[transforms.parse_logs_clickhouse]
type = "remap"
inputs = ["dummy_logs"]
...
[transforms.parse_logs_s3]
type = "remap"
inputs = ["dummy_logs"]
...
[sinks.s3]
...
buffer.type = "disk"
buffer.max_size = 30000000000 # 30 GB
buffer.when_full = "drop_newest"
[sinks.clickhouse]
...
batch.timeout_secs = 1
buffer.type = "memory"
buffer.max_events = 3000000 # 2 GB
buffer.when_full = "drop_newest"
Imagine the following scenario where both Clickhouse and S3 go down, and the data (logs/events) begins to accumulate in buffers—memory for Clickhouse, and disk for S3. What happens if Vector.dev also crashes? The data in memory will be lost forever, but the data on disk remains. After restarting Vector.dev, the data that remains on the disk does not get sent to S3. How can I send it there?
At first I wanted to find the answer in the documentation. But Vector.dev has quite a large documentation and I couldn’t find the answer to my question in it. I also searched for the answer on the Internet, but I didn’t find it there either
letsgoliquid is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.