Configure Log Retention Period in ELK Stack

|
Published:
|
|

How do I change the log retention policy in elk stack? In this tutorial, you will learn how to configure log retention period in ELK stack. Elasticsearch uses ILM (Index Lifecycle Management) policies to define what actions to be applied to indices according to your performance, resiliency, and retention requirements.

Configure Log Retention Period in ELK Stack

By default, if no custom policy is applied to Elasticsearch indices, it will store data as long as possible without deleting them. Long data retention will make management of data cumbersome. There are also implications of cost of required storage and relevancy of the data as it ages. You should however be always guided by your regulatory and compliance policies on how long you should retain your data.

So, how can you configure log retention period in ELK stack?

As already mentioned, Elastic uses ILM policies to define how long the log data can be stored. Some of the index lifecycle policies can trigger actions to permanently remove an index, including all of its data and metadata.

In order to configure Log retention period in ELK stack;

Create Custom Index Lifecycle Management Policy

To begin with, if you are running a cluster, you need to ensure that all nodes are running same version of Elasticsearch.

Next, proceed to create your custom ILM policy to apply to your indices.

You can create custom ILM policy from Kibana or using Elasticsearch API.

  • create ILM Policy from Kibana
  • create ILM policy from Elasticsearch API

Create ILM Policy from Kibana

To create a custom ILM policy from Kibana;

  • Navigate to Kibana menu > Management > Stack Management > Data > Index Lifecycle PoliciesCreate policy.
Configure Log Retention Period in ELK Stack
  • Enter the name of the policy,
  • Define various lifecycle phases. There exists five different phases as described below;
    • Hot: The index is actively being updated and queried. This phase is required.
    • Warm: The index is no longer being updated but is still being queried.
    • Cold: The index is no longer being updated and is queried infrequently. The information still needs to be searchable, but it’s okay if those queries are slower.
    • Frozen: The index is no longer being updated and is queried rarely. The information still needs to be searchable, but it’s okay if those queries are extremely slow.
    • Delete: The index is no longer needed and can safely be removed.
  • You can configure your policy to transition your data across all the phases above. For example, from hot phase > warm phase > cold phase > delete phase, while defining respective settings at each phase. You can even have a policy that transition your index between just hot and delete phase based on some conditions.
  • By default, all phases stores data indefinitely. Configure Log Retention Period in ELK Stack
  • To delete your index after a given period of time, you need to enable delete phase by toggling the Trash can icon on that phase. For example, if you want to move your index data from hot phase into a delete phase after a specific period of time, toggle the Trash can icon on the Hot phase.
  • You should be able to see delete phase enabled.Configure Log Retention Period in ELK Stack
  • Rollover from a phase to phase is determined by size of the primary shard or the maximum/minimum age of an index in a phase:
    • Maximum Age: This is the maximum amount of time an index can remain in a phase before it moves to the next phase.
    • Minimum Age: This is the minimum amount of time an index must remain in a phase before it can move to the next phase.
    • Maximum Size: This is the maximum size an index can be before it moves to the next phase.
  • For demonstration purpose, let’s see how you can be able to configure your data to transition from hot > warm > cold > delete phase.

Configure Hot Phase;

  • Under Hot phase settings, expand Advanced Settings
  • Disable “Use recommended defaults” option to be able to define your own thresholds.
  • Enable rollover to start writing to a new index when the current index reaches a certain size, document count, or age.
  • Set maximum primary shard size. In this demo, we use 1MB so we don’t have to wait for ages to demo the setup.
  • Set the index maximum age. We use 10 minutes here, again for easy demo.
  • You can leave other options with default settings.
Configure Log Retention Period in ELK Stack

Enable and Configure Warm Phase;

  • Choose when the data from Hot phase is moved into Warm phase. We use 5 minutes here for easy demonstrations.
  • We also delete replicas when moving into warm phase by setting number of replica to 0.
Configure Log Retention Period in ELK Stack

Enable and Configure Cold Phase

  • Next, enable and configure cold phase;
  • Define when to move data from warm phase to cold phase. We set it to 5 minutes. The value must be greater than or equal to the warm phase value.
  • Set the data to be deleted after this phase by toggling trash can icon.
  • You can leave other options with default settings.
  • You will need Enterprise license to be able to create a searchable snapshot in this phase.
cold phase settings 1

Configure Delete Phase

  • Enable delete phase after cold phase.
  • We set the data to be moved into this phase after 6 minutes. Again, the value must be greater or equal than the cold phase value.
delete phase settings

Save the Policy and it should now be listed in your policies;

custom ilm policy 1

You can get the ILM policy details from command line;

curl -XGET localhost:9200/_ilm/policy/kifarunix-demo?pretty
{
  "kifarunix-demo" : {
    "version" : 4,
    "modified_date" : "2023-03-23T19:22:16.898Z",
    "policy" : {
      "phases" : {
        "warm" : {
          "min_age" : "5m",
          "actions" : {
            "allocate" : {
              "number_of_replicas" : 0,
              "include" : { },
              "exclude" : { },
              "require" : { }
            }
          }
        },
        "cold" : {
          "min_age" : "5m",
          "actions" : { }
        },
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "1mb",
              "max_age" : "10m"
            }
          }
        },
        "delete" : {
          "min_age" : "5m",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    },
    "in_use_by" : {
      "indices" : [ ],
      "data_streams" : [ ],
      "composable_templates" : [
        "kifarunix-demo"
      ]
    }
  }
}

Create Custom Index Template

An index template is a pre-defined set of rules and settings that define how new indices should be created. Later on, we will configure our new index template to use the custom ILM policy while creating new indices.

Thus, navigate to Kibana menu > Management > Stack Management > Data > Index ManagementIndex Templates > Create Template.

  • Under logistics, set the name and the index pattern of the new index template.
    index template logistics
  • For step 2, 3, 4, 5, we will leave the default settings on.
  • On Step 6, review your index template settings and create it.
Configure Log Retention Period in ELK Stack

You index template should now be listed among other existing templates.

Apply ILM Policy into the Index Template

To use a policy that triggers the rollover action, you need to configure the policy in the index template used to create each new index. You specify the name of the policy and the alias used to reference the rolling indices“.

To apply an ILM Policy into the Index Template, you can use either of these methods;

Apply ILM Policy into the Index Template via Elasticsearch API on Elasticsearch node terminal

Login to Elasticsearch node or any node from which you can access Elasticsearch node and execute the command below;

  • Replace kifarunix-demo with your respective index name/pattern.
  • Change IP of the Elasticsearch accordingly.
curl -X PUT "192.168.58.22:9200/_index_template/kifarunix-demo?pretty" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["kifarunix-demo-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "kifarunix-demo",
      "index.lifecycle.rollover_alias": "kifarunix-demo"
    }
  }
}
'

If all is good, you should get such an output;

{
  "acknowledged" : true
}

Apply ILM Policy into the Index Template via Kibana Dev Tools

Navigate to Kibana menu > Stack Management > Management > Dev Tools > Console.

Paste and execute the command below;

PUT _index_template/kifarunix-demo?pretty
{
  "index_patterns": ["kifarunix-demo-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "kifarunix-demo",
      "index.lifecycle.rollover_alias": "kifarunix-demo"
    }
  }
}
update index template from kibana

You are now ready use your custom indices with your custom policy.

Verify ILM Policy and Data Retention on ELK Stack

In order to verify that our policy and rollover works as expected, we will configure a Filebeat agent to sent logs to the specific index, kifarunix-demo.

Before we can proceed, it is good to note that index lifecycle management (ILM) checks the current state of indices every 10 minutes by default. This value is controlled by the setting, index.lifecycle.poll_interval.

The default value of 10m seem reasonable and thus changing the value of this setting may have an impact on the performance of your Elasticsearch cluster, especially if you set a very short interval.

If you want to change this value temporarily, e.g to 5m, then you can do this by running the following command under Kibana DevTools console;

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": "5m"
  }
}

Or the command below on terminal on Elasticsearch node;

curl -XPUT "http://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d '{
  "transient": {
    "indices.lifecycle.poll_interval": "5m"
  }
}'

When you restart Elasticsearch, defaults are restored. To set permanent value, replace transient keyword with permanent in the above commands.

Next, configure Filebeat agent to sent event data to an index alias of kifarunix-demo. This is what we use in our filebeat config;

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "kifarunix-demo"

Below is a sample filebeat.yml file contents;

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /var/log/auth.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
  hosts: ["192.168.58.22:9200"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "kifarunix-demo"
seccomp:
  default_action: allow
  syscalls:
  - action: allow
    names:
    - rseq

Start filebeat and check indices on Kibana index management;

initial index created

Sample Elasticsearch logs

tail -f /var/log/elasticsearch/elasticsearch.log
...
[2023-03-24T23:09:05,232][INFO ][o.e.c.m.MetadataCreateIndexService] [debian11] [kifarunix-demo-2023.03.24-000001] creating index, cause [api], templates [kifarunix-demo], shards [1]/[1]
[2023-03-24T23:09:05,337][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [kifarunix-demo]
[2023-03-24T23:09:05,534][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [kifarunix-demo]
[2023-03-24T23:09:05,616][INFO ][o.e.c.m.MetadataMappingService] [debian11] [kifarunix-demo-2023.03.24-000001/W8R1iYLeSCiGoQ4Cg72UHg] create_mapping [_doc]
[2023-03-24T23:09:05,711][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [kifarunix-demo]
...

You can use GET <target>/_ilm/explain to show the current lifecycle status for one or more indices.

GET kifarunix-demo/_ilm/explain

{
  "indices" : {
    "kifarunix-demo-2023.03.24-000001" : {
      "index" : "kifarunix-demo-2023.03.24-000001",
      "managed" : true,
      "policy" : "kifarunix-demo",
      "lifecycle_date_millis" : 1679688545230,
      "age" : "4.83m",
      "phase" : "hot",
      "phase_time_millis" : 1679688545397,
      "action" : "rollover",
      "action_time_millis" : 1679688545597,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1679688545597,
      "phase_execution" : {
        "policy" : "kifarunix-demo",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "1mb",
              "max_age" : "10m"
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1679688094268
      }
    }
  }
}

As you can see from the above information, the index is just created.

It is good to note that the rollover may not happen at the exact time specified in the policy due to several factors, such as the size of the data being indexed, the load on the Elasticsearch cluster, and any other factors that may delay processing. But eventually, the rollover will happen based on the size of index/max age of the primary shard.

For example, as per the logs , rollover happened in like 14 minutes after we saw a log about creating index in the logs above;


[2023-03-24T23:23:10,329][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [kifarunix-demo]
[2023-03-24T23:23:10,439][INFO ][o.e.c.m.MetadataCreateIndexService] [debian11] [kifarunix-demo-2023.03.24-000002] creating index, cause [rollover_index], templates [kifarunix-demo], shards [1]/[1]
[2023-03-24T23:23:10,521][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [kifarunix-demo]
[2023-03-24T23:23:10,522][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] to [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] in policy [kifarunix-demo]
[2023-03-24T23:23:10,710][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [kifarunix-demo]
[2023-03-24T23:23:10,710][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] to [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] in policy [kifarunix-demo]
[2023-03-24T23:23:10,711][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] to [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] in policy [kifarunix-demo]
[2023-03-24T23:23:10,871][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [kifarunix-demo]
[2023-03-24T23:23:10,872][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] to [{"phase":"hot","action":"complete","name":"complete"}] in policy [kifarunix-demo]
[2023-03-24T23:23:11,098][INFO ][o.e.c.m.MetadataMappingService] [debian11] [kifarunix-demo-2023.03.24-000002/dz3BAJJMTcixNJANLQrY8A] create_mapping [_doc]
Configure Log Retention Period in ELK Stack

After the rollover, new index kifarunix-demo-2023.03.24-000002 is created. Both indices will stay in the hot phase until the next check conditions are met.

The rolled-over index, kifarunix-demo-2023.03.24-000001, will stay for 5 minutes in rolled-over state or the minimum age defined in the policy, after which it is moved to warm phase.

Since our poll interval is set to 10 minutes, by the time the next check happens, the rolled-over index, kifarunix-demo-2023.03.24-000001, will be older than the min_age (5 minutes) defined in the policy (in our case) and thus, rollover to warm phase happens.

Similarly, it will have also completed the min_age (5 mins) for moving to cold phase. Hence, it will be moved to warm phase then to cold phase immediately.


[2023-03-24T23:33:10,327][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"hot","action":"complete","name":"complete"}] to [{"phase":"warm","action":"allocate","name":"allocate"}] in policy [kifarunix-demo]
[2023-03-24T23:33:10,432][INFO ][o.e.c.m.MetadataUpdateSettingsService] [debian11] updating number_of_replicas to [0] for indices [kifarunix-demo-2023.03.24-000001]
[2023-03-24T23:33:10,516][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"warm","action":"allocate","name":"allocate"}] to [{"phase":"warm","action":"allocate","name":"check-allocation"}] in policy [kifarunix-demo]
[2023-03-24T23:33:10,603][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"warm","action":"allocate","name":"check-allocation"}] to [{"phase":"warm","action":"migrate","name":"branch-check-skip-action"}] in policy [kifarunix-demo]
[2023-03-24T23:33:10,605][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"warm","action":"migrate","name":"branch-check-skip-action"}] to [{"phase":"warm","action":"migrate","name":"migrate"}] in policy [kifarunix-demo]
[2023-03-24T23:33:10,782][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"warm","action":"migrate","name":"migrate"}] to [{"phase":"warm","action":"migrate","name":"check-migration"}] in policy [kifarunix-demo]
[2023-03-24T23:33:10,872][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"warm","action":"migrate","name":"check-migration"}] to [{"phase":"warm","action":"complete","name":"complete"}] in policy [kifarunix-demo]
[2023-03-24T23:33:10,965][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"warm","action":"complete","name":"complete"}] to [{"phase":"cold","action":"allocate","name":"allocate"}] in policy [kifarunix-demo]
[2023-03-24T23:33:11,050][INFO ][o.e.c.m.MetadataUpdateSettingsService] [debian11] updating number_of_replicas to [0] for indices [kifarunix-demo-2023.03.24-000001]
[2023-03-24T23:33:11,050][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"cold","action":"allocate","name":"allocate"}] to [{"phase":"cold","action":"allocate","name":"check-allocation"}] in policy [kifarunix-demo]
[2023-03-24T23:33:11,140][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"cold","action":"allocate","name":"check-allocation"}] to [{"phase":"cold","action":"migrate","name":"branch-check-skip-action"}] in policy [kifarunix-demo]
[2023-03-24T23:33:11,140][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"cold","action":"migrate","name":"branch-check-skip-action"}] to [{"phase":"cold","action":"migrate","name":"migrate"}] in policy [kifarunix-demo]
[2023-03-24T23:33:11,333][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"cold","action":"migrate","name":"migrate"}] to [{"phase":"cold","action":"migrate","name":"check-migration"}] in policy [kifarunix-demo]
[2023-03-24T23:33:11,434][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"cold","action":"migrate","name":"check-migration"}] to [{"phase":"cold","action":"complete","name":"complete"}] in policy [kifarunix-demo]
[2023-03-24T23:33:11,523][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"cold","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [kifarunix-demo]

So, as per our policy, the index kifarunix-demo-2023.03.24-000001, stays in cold phase 5 minutes before being moved to delete phase. At the same time, the second index is rolled-over and process continues.


[2023-03-24T23:38:10,326][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [kifarunix-demo]
[2023-03-24T23:38:10,415][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] to [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] in policy [kifarunix-demo]
[2023-03-24T23:38:10,492][INFO ][o.e.c.m.MetadataCreateIndexService] [debian11] [kifarunix-demo-2023.03.24-000003] creating index, cause [rollover_index], templates [kifarunix-demo], shards [1]/[1]
[2023-03-24T23:38:10,580][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000001] from [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] to [{"phase":"delete","action":"delete","name":"delete"}] in policy [kifarunix-demo]
[2023-03-24T23:38:10,581][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000003] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [kifarunix-demo]
[2023-03-24T23:38:10,583][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] to [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] in policy [kifarunix-demo]
[2023-03-24T23:38:10,764][INFO ][o.e.c.m.MetadataDeleteIndexService] [debian11] [kifarunix-demo-2023.03.24-000001/W8R1iYLeSCiGoQ4Cg72UHg] deleting index
[2023-03-24T23:38:10,853][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000003] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [kifarunix-demo]
[2023-03-24T23:38:10,853][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] to [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] in policy [kifarunix-demo]
[2023-03-24T23:38:10,854][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] to [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] in policy [kifarunix-demo]
[2023-03-24T23:38:11,033][INFO ][o.e.c.m.MetadataMappingService] [debian11] [kifarunix-demo-2023.03.24-000003/3y03bVJHQkOXnOLsBWURPw] create_mapping [_doc]
[2023-03-24T23:38:11,126][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000003] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [kifarunix-demo]
[2023-03-24T23:38:11,126][INFO ][o.e.x.i.IndexLifecycleTransition] [debian11] moving index [kifarunix-demo-2023.03.24-000002] from [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] to [{"phase":"hot","action":"complete","name":"complete"}] in policy [kifarunix-demo]
index after deletion phase

So, the first index has completed the cycle and has been deleted!

And that should suffice to configure log retention period in ELK Stack. Just ensure that you set the right age as per your compliance requirements.

That concludes our simple guide on how to configure log retention period in ELK Stack.

Other Tutorials

How to Integrate ELK Stack with TheHive

Create Custom ELK Ingest Pipeline for Custom Log Processing

SUPPORT US VIA A VIRTUAL CUP OF COFFEE

We're passionate about sharing our knowledge and experiences with you through our blog. If you appreciate our efforts, consider buying us a virtual coffee. Your support keeps us motivated and enables us to continually improve, ensuring that we can provide you with the best content possible. Thank you for being a coffee-fueled champion of our work!

Photo of author
koromicha
I am the Co-founder of Kifarunix.com, Linux and the whole FOSS enthusiast, Linux System Admin and a Blue Teamer who loves to share technological tips and hacks with others as a way of sharing knowledge as: "In vain have you acquired knowledge if you have not imparted it to others".

Leave a Comment