I am using ELK (elasticsearch-8.12.0-1.x86_64) to store kong API gateway logs. I am using ILM (Index Lifecycle Management) policy to manage the index retention and I mentioned it into
logstash pipeline configuration file.
I noticed that the new created indices are created using the below naming convention although they have been created in different days:
kong-2022-11-17-000001
kong-2022-11-17-000002
kong-2022-11-17-000003
kong-2022-11-17-000004
kong-2022-11-17-000005
kong-2022-11-17-000006
How to change the naming convention to include the creation date like the following:
kong-2022-11-17-000001
kong-2022-11-17-000002
kong-2022-11-17-000003
kong-2022-12-25-000001
kong-2023-01-01-000001
/etc/logstash/kong.conf
elasticsearch {
hosts => ["https://elastic01:elastic_port" , "https://elastic02:elastic_port" , "https://elastic03:elastic_port"]
user => "elastic_user"
password => elastic_user_password
ssl => true
ssl_certificate_verification => false
cacert => "/etc/logstash/http_ca.crt"
ilm_rollover_alias => "kong"
ilm_pattern => "{now/d}-000001"
ilm_policy => "kong-index-policy"
kong-index-template
{
"index": {
"lifecycle": {
"name": "kong-index-policy",
"rollover_alias": "kong"
},
"mapping": {
"total_fields": {
"limit": "10000"
}
},
"refresh_interval": "5s"
}
}
kong-index-policy
{
"policy": "kong-index-policy",
"phase_definition": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "180d",
"max_primary_shard_size": "10gb"
},
"set_priority": {
"priority": 100
}
}
},
I tried to configure ILM policy to manage the indices rollover and create the new index using the creation date but it is not working properly.