- Index Lifecycle Management (ILM)
- Data Stream Lifecycle (DSL) - newer concept
In cluster settings, data_streams.lifecycle.poll_interval defines how often shall Elasticsearch go over each data stream, check if it is eligible for a rollover and then perform it.
To find this interval value, check the output of
GET _cluster/settings
By default, the GET _cluster/settings command only returns settings that have been manually overridden so if we are using default values, we need to add ?include_defaults=true.
Default interval value is 5 minutes which can be verified by checking cluster's default settings:
GET _cluster/settings?include_defaults=true&filter_path=defaults.data_streams.lifecycle.poll_interval
Output:
{
"defaults": {
"data_streams": {
"lifecycle": {
"poll_interval": "5m"
}
}
}
}
After this interval, Elasticsearch rolls over the write index of the data stream, if it fulfills the conditions defined by cluster.lifecycle.default.rollover. If we are using default cluster settings, we can check its default value:
GET _cluster/settings?include_defaults=true&filter_path=defaults.cluster.lifecycle
Output:
{
"defaults": {
"cluster": {
"lifecycle": {
"default": {
"rollover": "max_age=auto,max_primary_shard_size=50gb,min_docs=1,max_primary_shard_docs=200000000"
}
}
}
}
}
max_age=7d: This is why our indices are rolling over every week.
max_primary_shard_size=50gb: Prevents shards from becoming too large and slow.
max_primary_shard_docs=200000000: A built-in limit to maintain search performance, even if the 50GB size hasn't been reached yet
/**
* When max_age is auto we’ll use the following retention dependent heuristics to compute the value of max_age:
* - If retention is null aka infinite (default), max_age will be 30 days
* - If retention is less than or equal to 1 day, max_age will be 1 hour
* - If retention is less than or equal to 14 days, max_age will be 1 day
* - If retention is less than or equal to 90 days, max_age will be 7 days
* - If retention is greater than 90 days, max_age will be 30 days
*/
So, max age of backing index before rollover depends on how long we want to keep data overall in our data stream. For example, if it's 90 days, Elasticsearch will perform rollover and create a new backing index every 7 days.
Instead of a single fixed value for every data stream, auto adjusts the rollover age to ensure that indices aren't kept too long or rolled over too frequently for their specific retention settings.
max_age=auto is a "smart" setting designed to prevent "small index bloat" while ensuring data is deleted on time. It ensures our max_age is always a fraction of our total retention so that we have several backing indices to delete sequentially as they expire.
Data Stream Lifecycle (DSL)
This is a streamlined, automated alternative to the older Index Lifecycle Management (ILM).
While ILM focuses on "how" data is stored (tiers, hardware, merging), the lifecycle block focuses on "what" happens to the data based on business needs, primarily focusing on retention and automated optimization.
How to find out if data stream is managed by Index Lifecycle Management (ILM) or Data Stream Lifecycle (DSL)?
Get the data stream's details and look at template, lifecycle, next_generation_managed_by and prefer_ilm attributes. Example:
GET _data_stream/ilm-history-7
Output snippet:
"template": "ilm-history-7",
"lifecycle": {
"enabled": true,
"data_retention": "90d",
"effective_retention": "90d",
"retention_determined_by": "data_stream_configuration"
},
"next_generation_managed_by": "Data stream lifecycle",
"prefer_ilm": true,
lifecycle block in our data stream's index template refers to the Data Stream Lifecycle (DSL).
Inside that lifecycle block, we typically see these children:
- enabled (Boolean):
- Interpretation: Determines if Elasticsearch should actively manage this data stream using DSL.
- Behavior: When set to true, Elasticsearch automatically handles rollover (based on cluster defaults) and deletion (based on our retention settings). If this is missing but other attributes are present, it often defaults to true.
- data_retention (String):
- Interpretation: The minimum amount of time Elasticsearch is guaranteed to store our data.
- Format: Uses time units like 90d (90 days), 30m (30 minutes), or 1h (1 hour).
- Behavior: This period is calculated starting from the moment a backing index is rolled over (it becomes "read-only"), not from its creation date.
- effective_retention
- This is the final calculated value that Elasticsearch actually uses to delete data.
- What it represents: It is the minimum amount of time our data is guaranteed to stay in the cluster after an index has rolled over.
- Why it might differ from our setting: We might set data_retention: "90d", but the cluster might have a global "max retention" or "default retention" policy that overrides our specific request
- retention_determined_by
- This attribute identifies the source of the effective_retention value. Common values include:
- data_stream_configuration: The retention is coming directly from the data_retention we set in our index template or data stream.
- default_retention: We didn't specify a retention period, so Elasticsearch is using the cluster-wide default (e.g., data_streams.lifecycle.retention.default).
- max_retention: We tried to set a very long retention (e.g., 1 year), but a cluster admin has capped all streams at a lower value (e.g., 90 days) using data_streams.lifecycle.retention.max
- downsampling (Object/Array):
- Interpretation: Configures the automatic reduction of time-series data resolution over time.
- Behavior: It defines when (e.g., after 7 days) and how (e.g., aggregate 1-minute metrics into 1-hour blocks) data should be condensed to save storage space while keeping historical trends searchable.
Elasticsearch determines the final retention value using this priority:
- If a Max Retention is set on the cluster and our setting exceeds it, Max Retention wins.
- If we have configured Data Retention on the stream, it is used (as long as it's under the max).
- If we have not configured anything, the Default Retention for the cluster is used.
- If no defaults or maxes exist and we haven't set a value, retention is Infinite.
prefer_ilm setting is a transition flag used when a data stream has both an Index Lifecycle Management (ILM) policy and a Data Stream Lifecycle (DSL) configuration. It tells Elasticsearch which of the two management systems should take control of the data stream. Value options are:
- true: Elasticsearch will use the ILM policy defined in index template setting index.lifecycle.name. It will ignore the lifecycle block (DSL). Use this if you need granular control over shard allocation, force merging, or specific rollover ages that DSL doesn't offer.
- false (or unset): Elasticsearch will prioritize the Data Stream Lifecycle (DSL) block. It will ignore the ILM policy for rollover and retention. This is the default behavior in modern 2026 clusters to encourage the use of the simpler DSL.
What if data retention is set both in lifecycle (DSL) and in ILM associated to the index template used for data stream?
If we see retention-related settings in both the lifecycle block and the settings block of an index template, the lifecycle block takes precedence because it is the native configuration for the Data Stream Lifecycle (DSL). This is the modern way to manage data streams. When the lifecycle block is present and enabled: true, Elasticsearch ignores any traditional ILM "Delete" phase settings. It manages the retention of the data stream indices exclusively through the DSL background process.
If a data stream has both a lifecycle block and an ILM policy in data stream index template like:
"settings": {
"index.lifecycle.name": "my-ilm-policy"
}
...then:
- The lifecycle block wins: Elasticsearch will prioritize the Data Stream Lifecycle (DSL) for retention and rollover.
- The ILM policy is ignored: We will often see a warning in the logs or the "Explain" API indicating that the ILM policy is being bypassed because DSL is active.
If we have a custom setting in the settings block (like a metadata field or a legacy retention setting) index.lifecycle.retention: It is ignored for logic: DSL only looks at the lifecycle object. Any other setting is treated as a static index setting and will not trigger the deletion of indices.
How do we associate ILM policy with data stream?
Associating an ILM policy with a data stream requires configuring the data stream's backing index template. Because your current template uses the newer Data Stream Lifecycle (DSL), you must also explicitly tell Elasticsearch to favor ILM.
1. Update the Index Template
To associate a policy, you must add the policy name to the index settings within the template that matches your data stream.
Using the API:
PUT _index_template/<your-template-name>
{
"index_patterns": ["ilm-history-7*"],
"template": {
"settings": {
"index.lifecycle.name": "your-ilm-policy-name",
"index.lifecycle.prefer_ilm": true
}
},
"data_stream": { }
}
index.lifecycle.name: Specifies which ILM policy to use.
index.lifecycle.prefer_ilm: true: This is critical if your template still has a lifecycle block. It forces Elasticsearch to use the ILM policy instead of DSL.
2. Apply to Existing Backing Indices
Updating a template only affects future indices created by the data stream. To apply the policy to the 14 indices already in your stream, you must update their settings directly:
PUT .ds-ilm-history-7-*/_settings
{
"index": {
"lifecycle": {
"name": "your-ilm-policy-name",
"prefer_ilm": true
}
}
}
Note: Use a wildcard like .ds-ilm-history-7-* to target all existing backing indices at once.
If you are moving back to ILM because you need a specific max_age (e.g., rollover every 1 day instead of 7), ensure your new ILM policy has the rollover action defined in its Hot phase. Once applied, the ilm-history-7 stream will immediately begin following the custom timings in your ILM policy instead of the cluster-wide DSL defaults.
What if index template has lifecycle attribute but no index.lifecycle.name?
If an index template contains a lifecycle block, it is configured to use Data Stream Lifecycle (DSL).
If you want to associate a specific ILM policy (to gain granular control over rollover max_age, for example) while this block exists, you must handle the conflict as follows:
1. DSL vs. ILM Precedence
The presence of "lifecycle": { "enabled": true } tells Elasticsearch to ignore traditional ILM. To force the use of an ILM policy instead, you must add index.lifecycle.prefer_ilm: true to the settings block.
Without that setting, the lifecycle block will "win," and your ILM policy will be ignored.
2. How to associate the ILM Policy
To properly link an ILM policy to this specific template, you should update it to look like this:
{
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "your-ilm-policy-name", // 1. Point to your ILM policy
"prefer_ilm": true // 2. Tell ES to favor ILM over the DSL block
},
"number_of_shards": "1",
"auto_expand_replicas": "0-1",
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_hot"
}
}
}
}
},
"mappings": { ... },
"lifecycle": { // This block can remain, but will be ignored
"enabled": true, // because prefer_ilm is true above.
"data_retention": "90d"
}
}
}
Key Interpretations
- The lifecycle block at the root: This governs retention (90 days) and rollover (defaulted to auto, which usually means 7 days or 30 days depending on retention).
- The settings.index block: This is where you define the ILM link.
- Conflict Resolution: If you don't add prefer_ilm: true, Elasticsearch 2026 defaults to using the lifecycle block. Your data stream will continue rolling over every 7–30 days based on the auto logic, even if you put an ILM policy name in the settings.
Recommendation
If you want to use an ILM policy, the cleanest approach is to remove the lifecycle block from the template entirely and only use index.lifecycle.name in the settings. This eliminates any ambiguity for the orchestration engine.
How to fix a Shard Explosion?
Shard Explsion happens when there are more than ~20 shards per GB of heap (Elasticsearch node heap - the JVM heap allocated to the Elasticsearch process, not the Kubernetes node's total memory.)
Force merge does not reduce the number of backing indices in a data stream.
A force merge (_forcemerge) acts on the segments within the shards of the backing indices, not on the backing indices themselves.
Here is the breakdown of what force merge does and how to reduce backing indices:
What Force Merge Does
- Merges Segments: It reduces the number of Lucene segments in each shard, ideally to one, which improves search performance.
- Cleans Deleted Docs: It permanently removes (expunges) documents that were soft-deleted, freeing up disk space.
- Targets Shards: The operation is performed on the shards of one or more indices (or data stream backing indices).
What Reduces Backing Indices
To reduce the number of backing indices for a data stream, you must use other strategies:
- Rollover (ILM): Index Lifecycle Management (ILM) creates new backing indices and can automatically delete old ones based on age or size.
- Data Stream Lifecycle: This automates the deletion of backing indices that are older than the defined retention period.
- Shrink API: While not typically used to combine multiple daily indices into one, it can be used to reduce the primary shard count of a specific, read-only backing index.
- Delete Index API: You can manually delete older backing indices (except the current write index).
Summary:
- Force Merge = Reduces segments inside shards (better search, less disk space).
- Rollover/Delete = Reduces the total number of backing indices (fewer indices overall).
---
