I have Kibana, Elasticsearch, and Filebeat running in an AKS cluster.

Filebeat is configured to capture logs from a few applications in my cluster, send to an index created each day, apply an ingest pipeline, and also apply a lifecycle policy.

The ingest pipeline and lifecycle policy are created using terraform, and the index is generated by filebeat.

Part of my filebeat configuration:

output.elasticsearch:
  host: '${NODE_NAME}'
  hosts: '["${ELASTICSEARCH_HOST}"]'
  username: '${ELASTICSEARCH_USERNAME}'
  password: '${ELASTICSEARCH_PASSWORD}'
  protocol: https
  ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]
  index: "${CLUSTER_ENV}-filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
  pipeline: "${CLUSTER_ENV}-cluster-pipeline"

setup:
  ilm: # index lifecycle management
    enabled: true
    policy_name: "${CLUSTER_ENV}-cluster-policy"
    overwrite: true

  template:
    enabled: true
    name: "${CLUSTER_ENV}-filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
    pattern: "${CLUSTER_ENV}-filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"

The thing is: it seems that each time on the creation of a new daily index, the lifecycle policy gets “recreated” losing any customizations done in by the terraform.

I already tried removing the overwrite: true from the ilm block, but I don’t think anything changed.

By default, the documentation says it applies a filebeat lifecycle policy (which gets created), so by specifying a name, I was expecting to use the existing one I created. Then what seems to be happening is that filebeat is re-creating the policy each day, with default parameters.

Also saw some similar questions, which indicate that daily indexes are not encouraged. Could it be related to my problem?