I’m using AWS Fargate and log_router containers with a custom Fluent Bit configuration to route logs. Here is my current configuration:
[FILTER]
Name grep
Match cw
Exclude log unwatedtext
[FILTER]
Name modify
Match cw
Remove source
Remove container_id
Remove container_name
Remove ec2_instance_id
Remove ecs_cluster
Remove ecs_task_arn
Remove ecs_task_definition
[OUTPUT]
Name cloudwatch_logs
Match cw
log_key log
region eu-west-1
log_group_name /ecs/container
log_stream_prefix fluent-bit-
auto_create_group On
log_format json/emf
[OUTPUT]
Name s3
bucket mybucket
total_file_size 100M
log_key log
match *
s3_key_format /$TAG/%Y/%m/%d/
use_put_object on
upload_timeout 1M
region eu-west-1
Currently, the unwatedtext is being removed for both CloudWatch and S3 outputs. However, I want to remove it only for CloudWatch and send all logs, including those with unwatedtext, to S3.
How can I modify my Fluent Bit configuration to achieve this?
Additional Information:
- Fluent Bit version: AWS_FOR_FLUENT_BIT_VERSION 2.32.2.20240627
- AWS Fargate setup
- Using log_router containers