Elastic Fleet

Elastic Agents do not strictly require Fleet and Fleet Server. You can deploy them in standalone mode, which allows you to manually configure and manage them using local configuration files.

However, running Elastic Agents in Fleet-managed mode with a Fleet Server is the recommended best practice for most enterprise environments.

Here is how the two approaches compare:

1. Fleet-Managed Mode (Recommended)

In this setup, you use the Fleet UI in Kibana to centrally manage agent policies, roll out upgrades, and apply integrations.

• How it works: Agents connect to a Fleet Server (which is just a specialized Elastic Agent process), which then receives policies from Elasticsearch and pushes them to your endpoints.
• Best for: Large-scale deployments, continuous monitoring, and Elastic Security/Endpoint integrations.
• Action: To set this up, refer to the official Elastic Agent Installation Guide.

2. Standalone Mode

In this setup, you manually install the agent and define its inputs, outputs, and integrations directly in a local YAML configuration file.

• How it works: The agent connects directly to outputs like Elasticsearch or Logstash without a Fleet Server intermediary.
• Limitations: Central management, automated upgrades, and certain advanced Endpoint Security features are disabled.
• Best for: Edge cases, highly air-gapped networks, or evaluating specific integrations.
• Action: To configure this, use the steps outlined in the Standalone Elastic Agent Tutorial.

Example of how an Elastic Fleet can actually be wired

Components & where they live

Everything is in cluster company-prod-elastic-eks (EKS, us-east-2, now on v1.36.2), namespace elastic-system, across 5 nodes in 3 AZs:

Node            AZ  Node Group             Runs

====            ==  ==========             ====

ip-10-99-44-1   2a  es MNG (m7g.2xlarge)   agent

ip-10-99-55-34  2b  default MNG (m5.large) agent + Fleet Server pod

ip-10-99-55-89  2b  es MNG (m7g.2xlarge)   agent

ip-10-99-66-51  2c  es MNG (m7g.2xlarge)   agent

ip-10-99-66-78  2c  default MNG (m5.large) agent

• Fleet Server — Deployment fleet-server-prod, replicas: 1 (single pod, currently on the 2b node). Listens on :8220 HTTPS. Exposed by a NodePort service fleet-server-prod-agent-http (8220 --> 32202 on every node).
• Elastic Agents — DaemonSet agent-prod-eck-agent, one pod per node (5 total), spread across all 3 AZs. hostNetwork: true, mode: fleet, FLEET_INSECURE=true. They collect node/pod logs + metrics (hostPath mounts of /var/log/...).
• ALB k8s-elastics-eckfleet-a1b2c3d4e5 — internal (not internet-facing), spanning subnets in 2a / 2b / 2c. Created by the AWS LB controller from Ingress eck-fleet-server-prod-ingress. Targets = all 5 nodes' NodePort 32202 (instance mode), backend-protocol: HTTPS, idle_timeout: 300s.
• DNS — eck-fleet-server.internal-domain.local (Route53 via external-dns) -->  the internal ALB. Elasticsearch is a separate endpoint, elasticsearch.internal-domain.local:443.

Who listens, who initiates

The key thing: every connection is initiated by the agent (outbound). Nothing is ever pushed to the agents — that's why this works with agents behind hostNetwork and no inbound rules of their own.

                          

                           ┌───────────────────────────────────────────────┐

                           │ cluster company-prod-elastic-eks / elastic-sys│

     ┌──────────┐          │                                               │

     │ Agent    │ check-in │   internal ALB          Fleet Server (1 pod)  │

     │ DaemonSet│─────────►│  eckfleet-a1b2...   ┌──────► :8220 (HTTPS)    │

     │ (5 pods, │  :443    │  :443 TLS  ─────────┘  NodePort 32202→8220    │

     │  3 AZs)  │  HTTPS   │  (internal cert)       kube-proxy → the 1 pod │

     └────┬─────┘          │       ▲  re-encrypt HTTPS to a node's :32202  │

          │                │       │                     │                 │

          │ ship data      └───────┼─────────────────────┼─────────────────┘

          │ (logs/metrics)         │ DNS                 │ writes .fleet-* / agent

          ▼                   eck-fleet-server.          ▼ metadata, reads policy

 elasticsearch.               internal-domain.local  Elasticsearch + Kibana (Fleet app)

 internal-domain.local:443 ◄──────────────────────  

 (NOT via the fleet ALB)

Traffic Flows: Breakdown

1. Agent --> Fleet Server (Control Plane: Enrollment & Policy Check-in)

The agent acts as the client, and Fleet Server listens on :8220. The agent dials https://eck-fleet-server.internal-domain.local:443:

Traffic Path:

agent --> DNS --> internal ALB :443 (TLS termination, internal cert) --> re-encrypt --> a node's NodePort :32202 -->  kube-proxy --> the single Fleet Server pod :8220

Cross-AZ Routing: 

Because the agent always targets the ALB DNS name (not the pod directly), an agent can land on any node's NodePort. It is then forwarded by kube-proxy to whichever node is hosting the single Fleet Server pod—making cross-AZ hops standard behavior.

The Long Poll Mechanism: The check-in is a long poll. The agent opens a connection, and Fleet Server holds it open for up to ~5 minutes until a policy change occurs or the poll times out.

The Latency "False Positive": This held-open duration is exactly what the ALB records as a TargetResponseTime of ~300s, which triggered the false-positive alert. The ALB's idle_timeout is intentionally set to 300s to support these long polls. Conversely, the health check (/health, expects 404, 300s interval) is a separate lightweight probe, explaining why targets report as perfectly healthy while "latency" metrics look highly skewed.

2. Agent --> Elasticsearch (Data Plane: Telemetry Shipping)

This data path does not traverse the Fleet ALB. The Fleet output configuration points agents directly at https://elasticsearch.internal-domain.local:443.

Consequently, the eckfleet ALB only handles control-plane check-in traffic. This explains why its request volume is incredibly low (~3–5 requests/min) while individual request durations hover around 300s. They are two entirely separate paths utilizing two different load balancers.

3. Fleet Server --> Elasticsearch + Kibana

Fleet Server (via elasticsearchRefs / kibanaRef) acts as the client here. It writes agent and policy metadata directly into the .fleet-* indices and coordinates with the Kibana Fleet application, where the eck-fleet-server and eck-agent policies are managed.

One-Line Summary

Agents (DaemonSet, 5 pods across 2a/2b/2c) long-poll the single Fleet Server pod for policy updates via the internal ALB on :443 --> NodePort 32202 --> :8220, but ship their actual telemetry data directly to Elasticsearch, completely bypassing that ALB. The ALB only processes the slow, held-open control traffic—which is the root cause of the  alarm.

Why does fleet server holds connection open for so long? Why it does not reply immediately?                                                                                                                                                                                   

It's a deliberate design choice called long polling, and it's the opposite of a bug. Here's the reasoning.

The problem Fleet is solving

Fleet Server's job is to push policy changes out to agents promptly — a new integration, a changed log path, an output credential rotation. Agents need to find out "has my policy changed?" with low latency.

There are three ways an agent could learn about changes:

  1. Short polling — agent asks "anything new?" every N seconds, server replies instantly "no", agent sleeps, repeats.

  2. Server push — server opens a connection to each agent when something changes.

  3. Long polling — agent asks "anything new?", and the server holds the request open until either something actually changes or a timeout fires.

Fleet uses #3, and the connection sitting open for ~5 minutes is that hold.

Why not reply immediately (short polling)?

If Fleet answered every check-in instantly with "nothing changed", then to get fast reaction to a policy change, agents would have to poll very frequently — say every few seconds. With your 5 agents that's tolerable, but Fleet is built to manage thousands to tens of thousands of agents from one server. At that scale:

  - Frequent short polls = a constant storm of requests, almost all of which return "no change". Huge wasted CPU/network on both ends.

  - To keep reaction time low you'd poll more often, which makes the storm worse. To cut the storm you'd poll less often, which makes policy changes take longer to land. There's no good setting.

Long polling breaks that trade-off: the agent gets a near-instant reaction to a real change (the server responds the moment policy updates) and there's almost no idle chatter (one held-open connection per agent instead of hundreds of empty round-trips).

Why not server push (#2)?

Pushing would mean the server initiating connections inward to every agent. Agents are all over the place — behind NAT, firewalls, in private subnets, on laptops, on hostNetwork pods like yours. The server usually can't reach them, and you'd need inbound rules everywhere. Long polling flips it: the agent always dials out to the server, the connection is already established and held open, and the server pushes the change down that existing agent-initiated connection the instant it happens. You get push-like latency with poll-like (outbound-only) connectivity. That's exactly why your agents work behind hostNetwork with no inbound exposure.

 So what's actually happening in those ~300s

A check-in is essentially: "Here's my current state; tell me the moment my policy differs." The server parks that request. Two things can end it:

  - A policy change occurs → server responds immediately with the new policy (could be 2 seconds in).

  - Nothing changes → the server lets the request time out at its poll ceiling (~5 min), responds "no change", and the agent immediately opens a fresh one.

Since your policies rarely change, almost every check-in runs the full clock and returns at ~300s. That's the held connection the ALB measures as TargetResponseTime.

Why this specifically fools the ALB

 From the ALB's point of view, "request received → response sent" took 300 seconds, so it reports TargetResponseTime ≈ 300s. The ALB can't tell the difference between "the backend was slow for 300s" (bad) and "the backend intentionally held an idle long-poll for 300s" (normal). That ambiguity is the whole reason your generic 0.8s threshold misfires — and why the right fix (DOP-833) is to exempt this endpoint rather than treat it as latency. It's also why the Ingress sets idle_timeout.timeout_seconds=300: the ALB has to be told to tolerate the held connection, otherwise it would sever the long poll before the agent's poll cycle completes.

In short: Fleet holds the connection open so it can deliver policy changes near-instantly without either hammering the server with empty polls or needing to reach inward to firewalled agents. The ~300s is just an idle long-poll waiting for a change that usually never comes during that window — efficient by design, and only a problem for a latency metric that doesn't know to expect it.

----

Grafana Observability Stack

 

Grafana uses these components together as an observability stack, but each has a clear role:

Loki – log database. It stores and indexes logs (especially from Kubernetes) in a cost‑efficient, label‑based way, similar to Prometheus but for logs.

Tempo – distributed tracing backend. It stores distributed traces (spans) from OpenTelemetry, Jaeger, Zipkin, etc., so you can see call flows across microservices and where latency comes from.

Mimir – Prometheus‑compatible metrics backend. It is a horizontally scalable, long‑term storage and query engine for Prometheus‑style metrics (time series).

Alloy – telemetry pipeline (collector). It is Grafana’s distribution of the OpenTelemetry Collector / Prometheus agent / Promtail ideas, used to collect, process, and forward metrics, logs, traces, profiles into Loki/Tempo/Mimir (or other backends).

How Grafana UI relates to them

Grafana UI itself is “just” the visualization and alerting layer:

• It connects to Loki, Tempo, Mimir (and many others) as data sources.
• For each backend you configure:
• A Loki data source for logs.
• A Tempo data source for traces.
• A Prometheus/Mimir data source for metrics (Mimir exposes a Prometheus‑compatible API).
• Grafana then lets you:
• Build dashboards and alerts from Mimir metrics.
• Explore logs from Loki.
• Explore traces from Tempo and cross‑link them with logs/metrics (e.g., click from a log line to a trace, or from a metrics graph into logs/traces).

A useful mental model: Loki/Tempo/Mimir are databases, Alloy is the collector/router, and Grafana is the UI on top.

Are they deployed in the same Kubernetes cluster?

Common patterns:

• Very common: deploy Loki, Tempo, Mimir, Alloy, and Grafana in the same Kubernetes cluster as your apps. This is the typical “in‑cluster LGTM” setup; all telemetry stays inside the cluster and traffic is simple.
• Also common: run them in a separate observability cluster (or use Grafana Cloud backends), while Alloy/agents run in each workload cluster and ship data over the network. This improves isolation and makes it easier to share one observability stack across many clusters.
• In smaller setups or dev environments, everything (apps + LGTM + Grafana) often lives in one cluster; in larger/regulated setups, people tend to separate “workload clusters” and an “observability cluster”.

So: they don’t have to be on the same cluster, but it’s perfectly normal (and often simplest) to run Grafana + Loki + Tempo + Mimir + Alloy together in a single Kubernetes cluster and point your apps’ telemetry to Alloy.

Mimir

• Open-source, horizontally scalable, and highly available Time Series Database (TSDB)
• Designed by Grafana Labs as an extension for Prometheus
• Solves the scaling and storage limits of standalone Prometheus servers, allowing enterprises to ingest, store, and query over a billion active metrics

Why is it used?

• Unlimited Scalability & Long-Term Storage: While Prometheus is typically a single-node application with limited local disk storage, Mimir decouples storage from ingestion. It offloads long-term metric data to cheap object storage like Amazon S3, Google Cloud Storage (GCS), or Azure Blob Storage.
• Prometheus Compatibility: It uses the same data format as Prometheus and is 100% compatible with PromQL (Prometheus Query Language), making it a drop-in replacement or upgrade for existing Prometheus setups.
• Multi-Tenancy: It allows different teams or departments to use a single, shared Mimir cluster while keeping their data and queries entirely isolated.
• High Availability (HA): Mimir uses a microservices-based architecture (distributors, ingesters, queriers, and compactors). This means if a node fails, your incoming metrics are safely replicated elsewhere without data loss

How it Works in Your Stack?

• Collection: Systems like Prometheus, Grafana Alloy, or the OpenTelemetry Collector scrape and collect system/application metrics.
• Ingestion: These metrics are sent to Grafana Mimir via the remote_write protocol. Mimir then shards, replicates, and safely stores the data.
• Visualization: You connect Grafana to Mimir as a data source to build interactive dashboards and set up alerting rules.

Why not using elasticsearch instead of loki, tempo and mimir?

Elasticsearch can replace part of what Loki, Tempo, and Mimir do, but not all of it, and usually with higher cost/complexity for cloud‑native observability.

1. Scope: logs vs full observability

Elasticsearch is a general search and analytics engine that’s great at full‑text search, aggregations, and analytics over documents (including logs).

The LGTM stack is explicitly split by signal:

• Loki → logs
• Tempo → traces
• Mimir → metrics

Each is optimized only for its signal type and integrates tightly with Grafana and modern telemetry standards.

You could plausibly replace Loki with Elasticsearch for logs, but Elasticsearch does not natively replace Tempo (distributed tracing backend) or Mimir (Prometheus‑compatible metrics backend).

2. Logs: Loki vs Elasticsearch

Elasticsearch strengths:

• Very powerful full‑text search, fuzzy matching, relevance scoring, complex aggregations.
• Good when you need deep forensic search and advanced analytics on log text.

Loki strengths:

• Stores logs as compressed chunks plus a small label index, so storage and compute are much cheaper than Elasticsearch for typical Kubernetes logs.
• Very tight integration with Grafana and the rest of LGTM, and simple, label‑based querying.

Trade‑off: Elasticsearch gives richer search at a high infra + ops cost, Loki gives “good enough” search for operational troubleshooting with much lower cost and operational burden.

3. Traces and metrics: Tempo & Mimir vs “just ES”

Tempo:

• Implements distributed tracing concepts (spans, traces, service graphs) and OpenTelemetry/Jaeger/Zipkin protocols; the data model and APIs are specialized for traces.
• Elasticsearch can store trace‑like JSON documents, but you’d have to build/maintain all the trace stitching, UI navigation, and integrations yourself.

Mimir:

• Is a horizontally scalable, Prometheus‑compatible time‑series database, with native remote‑write/read and PromQL semantics.
• Elasticsearch can store time‑stamped metrics, but you lose Prometheus compatibility, PromQL semantics, and the whole ecosystem that expects a Prometheus‑style API.

So using only Elasticsearch means you’re giving up the standard metrics and tracing ecosystems and rebuilding a lot of tooling on top of a generic search engine.

​

4. Cost, complexity, and operational burden

Elasticsearch clusters generally need:

• More RAM/CPU per node, careful shard and index management, and capacity planning.
• Storage overhead from full‑text indexes (often 1.5–3× raw log size plus replicas).

​

Loki/Tempo/Mimir:

• Are designed for object storage, compression, and label‑only indexing, which dramatically lowers storage and compute requirements for logs and metrics.
• Have simpler, well‑documented reference architectures specifically for observability.

For a modern Kubernetes‑centric environment, that usually makes LGTM cheaper and easier to run than a single big Elasticsearch cluster for everything.

5. When Elasticsearch still makes sense

You might still choose Elasticsearch (often with Kibana/APM) if:

• You already have a strong ELK stack and team expertise.
• Your primary need is deep, flexible text search and analytics over logs, with less emphasis on Prometheus/OTel ecosystems.
• You want Elasticsearch’s ML/anomaly‑detection features and are willing to pay the operational cost.

But if your goal is a Grafana‑centric, standards‑based (Prometheus + OpenTelemetry) observability platform, LGTM (Loki+Tempo+Mimir, plus Alloy as collector) is a better fit than trying to push everything into Elasticsearch.

---

Elasticsearch Data Streams

In Elasticsearch, a data stream is an abstraction layer designed to simplify the management of continuously generated time-series data, such as logs, metrics, and events. 

Key Characteristics

• Time-Series Focus: Every document indexed into a data stream must contain a @timestamp field, which is used to organize and query the data.
• Append-Only Design: Data streams are optimized for use cases where data is rarely updated or deleted. We cannot send standard update or delete requests directly to the stream; these must be performed via _update_by_query or directed at specific backing indices.
• Unified Interface: Users interact with a single named resource (the data stream name) for both indexing and searching, even though the data is physically spread across multiple underlying indices. 

Architecture: Backing Indices

A data stream consists of one or more hidden, auto-generated backing indices: 

• Write Index: The most recently created backing index. All new documents are automatically routed here.
• Rollover: When the write index reaches a specific size or age, Elasticsearch automatically creates a new backing index (rollover) and sets it as the new write index.
• Search: Search requests sent to the data stream are automatically routed to all of its backing indices to return a complete result set. 

Automated Management

Data streams rely on two primary automation tools: 

• Index Templates: These define the stream's structure, including field mappings and settings, and must include a data_stream object to enable the feature.
• Lifecycle Management (ILM/DSL): Tools like Index Lifecycle Management (ILM) or the newer Data Stream Lifecycle automate tasks like moving old indices to cheaper hardware (hot/warm/cold tiers) and eventually deleting them based on retention policies. 

When to Use

• Ideal for: Logs, events, performance metrics, and security traces.
• Avoid for: Use cases requiring frequent updates to existing records (like a product catalog) or data that lacks a timestamp.

How does data stream know when to rollover?

Data streams are typically managed by:

• Index Lifecycle Management (ILM)
• Data Stream Lifecycle (DSL) - newer concept

In cluster settings, data_streams.lifecycle.poll_interval defines how often shall Elasticsearch go over each data stream, check if it is eligible for a rollover and then perform it. 

To find this interval value, check the output of 

GET _cluster/settings

By default, the GET _cluster/settings command only returns settings that have been manually overridden so if we are using default values, we need to add ?include_defaults=true.

Default interval value is 5 minutes which can be verified by checking cluster's default settings:

GET _cluster/settings?include_defaults=true&filter_path=defaults.data_streams.lifecycle.poll_interval

Output:

{

  "defaults": {

    "data_streams": {

      "lifecycle": {

        "poll_interval": "5m"

      }

    }

  }

}

After this interval, Elasticsearch rolls over the write index of the data stream, if it fulfills the conditions defined by cluster.lifecycle.default.rollover. If we are using default cluster settings, we can check its default value:

GET _cluster/settings?include_defaults=true&filter_path=defaults.cluster.lifecycle

Output:

{

  "defaults": {

    "cluster": {

      "lifecycle": {

        "default": {

          "rollover": "max_age=auto,max_primary_shard_size=50gb,min_docs=1,max_primary_shard_docs=200000000"

        }

      }

    }

  }

}

max_age=7d: This is why our indices are rolling over every week.

max_primary_shard_size=50gb: Prevents shards from becoming too large and slow.

max_primary_shard_docs=200000000: A built-in limit to maintain search performance, even if the 50GB size hasn't been reached yet

In our case max_age=auto which means Elasticsearch is using a dynamic rollover strategy based on our retention period. If we look at https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/action/admin/indices/rollover/RolloverConfiguration.java#L174-L195 we can see the comment:

    /**

     * When max_age is auto we’ll use the following retention dependent heuristics to compute the value of max_age:

     * - If retention is null aka infinite (default), max_age will be 30 days

     * - If retention is less than or equal to 1 day, max_age will be 1 hour

     * - If retention is less than or equal to 14 days, max_age will be 1 day

     * - If retention is less than or equal to 90 days, max_age will be 7 days

     * - If retention is greater than 90 days, max_age will be 30 days

     */

So, max age of backing index before rollover depends on how long we want to keep data overall in our data stream. For example, if it's 90 days, Elasticsearch will perform rollover and create a new backing index every 7 days.

Instead of a single fixed value for every data stream, auto adjusts the rollover age to ensure that indices aren't kept too long or rolled over too frequently for their specific retention settings.

max_age=auto is a "smart" setting designed to prevent "small index bloat" while ensuring data is deleted on time. It ensures our max_age is always a fraction of our total retention so that we have several backing indices to delete sequentially as they expire.

Data Stream Lifecycle (DSL)

This is a streamlined, automated alternative to the older Index Lifecycle Management (ILM). 

While ILM focuses on "how" data is stored (tiers, hardware, merging), the lifecycle block focuses on "what" happens to the data based on business needs, primarily focusing on retention and automated optimization.

How to find out if data stream is managed by Index Lifecycle Management (ILM) or Data Stream Lifecycle (DSL)?

Get the data stream's details and look at template, lifecycle, next_generation_managed_by and prefer_ilm attributes. Example:

GET _data_stream/ilm-history-7

Output snippet:

      "template": "ilm-history-7",

      "lifecycle": {

        "enabled": true,

        "data_retention": "90d",

        "effective_retention": "90d",

        "retention_determined_by": "data_stream_configuration"

      },

      "next_generation_managed_by": "Data stream lifecycle",

      "prefer_ilm": true,

lifecycle block in our data stream's index template refers to the Data Stream Lifecycle (DSL). 

Inside that lifecycle block, we typically see these children:

• enabled (Boolean):
• Interpretation: Determines if Elasticsearch should actively manage this data stream using DSL.
• Behavior: When set to true, Elasticsearch automatically handles rollover (based on cluster defaults) and deletion (based on our retention settings). If this is missing but other attributes are present, it often defaults to true.
• data_retention (String):
• Interpretation: The minimum amount of time Elasticsearch is guaranteed to store our data.
• Format: Uses time units like 90d (90 days), 30m (30 minutes), or 1h (1 hour).
• Behavior: This period is calculated starting from the moment a backing index is rolled over (it becomes "read-only"), not from its creation date.
• effective_retention
• This is the final calculated value that Elasticsearch actually uses to delete data. 
• What it represents: It is the minimum amount of time our data is guaranteed to stay in the cluster after an index has rolled over.
• Why it might differ from our setting: We might set data_retention: "90d", but the cluster might have a global "max retention" or "default retention" policy that overrides our specific request
• retention_determined_by
• This attribute identifies the source of the effective_retention value. Common values include: 
• data_stream_configuration: The retention is coming directly from the data_retention we set in our index template or data stream.
• default_retention: We didn't specify a retention period, so Elasticsearch is using the cluster-wide default (e.g., data_streams.lifecycle.retention.default).
• max_retention: We tried to set a very long retention (e.g., 1 year), but a cluster admin has capped all streams at a lower value (e.g., 90 days) using data_streams.lifecycle.retention.max
• downsampling (Object/Array):
• Interpretation: Configures the automatic reduction of time-series data resolution over time.
• Behavior: It defines when (e.g., after 7 days) and how (e.g., aggregate 1-minute metrics into 1-hour blocks) data should be condensed to save storage space while keeping historical trends searchable.

Elasticsearch determines the final retention value using this priority: 

• If a Max Retention is set on the cluster and our setting exceeds it, Max Retention wins.
• If we have configured Data Retention on the stream, it is used (as long as it's under the max).
• If we have not configured anything, the Default Retention for the cluster is used.
• If no defaults or maxes exist and we haven't set a value, retention is Infinite.

prefer_ilm setting is a transition flag used when a data stream has both an Index Lifecycle Management (ILM) policy and a Data Stream Lifecycle (DSL) configuration. It tells Elasticsearch which of the two management systems should take control of the data stream. Value options are:

• true: Elasticsearch will use the ILM policy defined in index template setting index.lifecycle.name. It will ignore the lifecycle block (DSL). Use this if you need granular control over shard allocation, force merging, or specific rollover ages that DSL doesn't offer.
• false (or unset): Elasticsearch will prioritize the Data Stream Lifecycle (DSL) block. It will ignore the ILM policy for rollover and retention. This is the default behavior in modern 2026 clusters to encourage the use of the simpler DSL. 

What if data retention is set both in lifecycle (DSL) and in ILM associated to the index template used for data stream?

If we see retention-related settings in both the lifecycle block and the settings block of an index template, the lifecycle block takes precedence because it is the native configuration for the Data Stream Lifecycle (DSL). This is the modern way to manage data streams. When the lifecycle block is present and enabled: true, Elasticsearch ignores any traditional ILM "Delete" phase settings. It manages the retention of the data stream indices exclusively through the DSL background process.

If a data stream has both a lifecycle block and an ILM policy in data stream index template like:

"settings": {

  "index.lifecycle.name": "my-ilm-policy"

}

...then:

• The lifecycle block wins: Elasticsearch will prioritize the Data Stream Lifecycle (DSL) for retention and rollover.
• The ILM policy is ignored: We will often see a warning in the logs or the "Explain" API indicating that the ILM policy is being bypassed because DSL is active.

If we have a custom setting in the settings block (like a metadata field or a legacy retention setting) index.lifecycle.retention: It is ignored for logic: DSL only looks at the lifecycle object. Any other setting is treated as a static index setting and will not trigger the deletion of indices.

How do we associate ILM policy with data stream?

Associating an ILM policy with a data stream requires configuring the data stream's backing index template. Because your current template uses the newer Data Stream Lifecycle (DSL), you must also explicitly tell Elasticsearch to favor ILM.

1. Update the Index Template

To associate a policy, you must add the policy name to the index settings within the template that matches your data stream.

Using the API:

PUT _index_template/<your-template-name>

{

  "index_patterns": ["ilm-history-7*"],

  "template": {

    "settings": {

      "index.lifecycle.name": "your-ilm-policy-name",

      "index.lifecycle.prefer_ilm": true

    }

  },

  "data_stream": { }

}

index.lifecycle.name: Specifies which ILM policy to use.

index.lifecycle.prefer_ilm: true: This is critical if your template still has a lifecycle block. It forces Elasticsearch to use the ILM policy instead of DSL.

2. Apply to Existing Backing Indices

Updating a template only affects future indices created by the data stream. To apply the policy to the 14 indices already in your stream, you must update their settings directly: 

PUT .ds-ilm-history-7-*/_settings

{

  "index": {

    "lifecycle": {

      "name": "your-ilm-policy-name",

      "prefer_ilm": true

    }

  }

}

Note: Use a wildcard like .ds-ilm-history-7-* to target all existing backing indices at once.

If you are moving back to ILM because you need a specific max_age (e.g., rollover every 1 day instead of 7), ensure your new ILM policy has the rollover action defined in its Hot phase. Once applied, the ilm-history-7 stream will immediately begin following the custom timings in your ILM policy instead of the cluster-wide DSL defaults.

What if index template has lifecycle attribute but no index.lifecycle.name?

If an index template contains a lifecycle block, it is configured to use Data Stream Lifecycle (DSL).

If you want to associate a specific ILM policy (to gain granular control over rollover max_age, for example) while this block exists, you must handle the conflict as follows:

1. DSL vs. ILM Precedence

The presence of "lifecycle": { "enabled": true } tells Elasticsearch to ignore traditional ILM. To force the use of an ILM policy instead, you must add index.lifecycle.prefer_ilm: true to the settings block.

Without that setting, the lifecycle block will "win," and your ILM policy will be ignored.

2. How to associate the ILM Policy

To properly link an ILM policy to this specific template, you should update it to look like this:

{

  "template": {

    "settings": {

      "index": {

        "lifecycle": {

          "name": "your-ilm-policy-name",  // 1. Point to your ILM policy

          "prefer_ilm": true              // 2. Tell ES to favor ILM over the DSL block

        },

        "number_of_shards": "1",

        "auto_expand_replicas": "0-1",

        "routing": {

          "allocation": {

            "include": {

              "_tier_preference": "data_hot"

            }

          }

        }

      }

    },

    "mappings": { ... },

    "lifecycle": {              // This block can remain, but will be ignored 

      "enabled": true,          // because prefer_ilm is true above.

      "data_retention": "90d"

    }

  }

}

Key Interpretations

• The lifecycle block at the root: This governs retention (90 days) and rollover (defaulted to auto, which usually means 7 days or 30 days depending on retention).
• The settings.index block: This is where you define the ILM link.
• Conflict Resolution: If you don't add prefer_ilm: true, Elasticsearch 2026 defaults to using the lifecycle block. Your data stream will continue rolling over every 7–30 days based on the auto logic, even if you put an ILM policy name in the settings.

Recommendation

If you want to use an ILM policy, the cleanest approach is to remove the lifecycle block from the template entirely and only use index.lifecycle.name in the settings. This eliminates any ambiguity for the orchestration engine.

How to fix a Shard Explosion?

Shard Explsion happens when there are more than ~20 shards per GB of heap (Elasticsearch node heap  - the JVM heap allocated to the Elasticsearch process, not the Kubernetes node's total memory.)

Force merge does not reduce the number of backing indices in a data stream. 

A force merge (_forcemerge) acts on the segments within the shards of the backing indices, not on the backing indices themselves. 

Here is the breakdown of what force merge does and how to reduce backing indices:

What Force Merge Does

• Merges Segments: It reduces the number of Lucene segments in each shard, ideally to one, which improves search performance.
• Cleans Deleted Docs: It permanently removes (expunges) documents that were soft-deleted, freeing up disk space.
• Targets Shards: The operation is performed on the shards of one or more indices (or data stream backing indices). 

What Reduces Backing Indices

To reduce the number of backing indices for a data stream, you must use other strategies:

• Rollover (ILM): Index Lifecycle Management (ILM) creates new backing indices and can automatically delete old ones based on age or size.
• Data Stream Lifecycle: This automates the deletion of backing indices that are older than the defined retention period.
• Shrink API: While not typically used to combine multiple daily indices into one, it can be used to reduce the primary shard count of a specific, read-only backing index.
• Delete Index API: You can manually delete older backing indices (except the current write index). 

Summary:

• Force Merge = Reduces segments inside shards (better search, less disk space).
• Rollover/Delete = Reduces the total number of backing indices (fewer indices overall).

---

Kubernetes DaemonSet

 

Kubernetes DaemonSet is a workload resource that ensures a specific pod runs on all (or selected) nodes in a cluster. It's commonly used for deploying node-level services like log collectors, monitoring agents, or network plugins.

Example:

Elasticsearch Agents are Elastic’s unified data shippers typically used in k8s cluster to collect container logs, Kubernetes metrics, node-level metrics and ship all of that data to Elasticsearch. They are deployed in the cluster as a DaemonSet.

We can use a DaemonSet to run a copy of a pod on every node, or we can use node affinity or selector rules to run it on only certain nodes.

What is the difference between ReplicaSet and DaemonSet?

ReplicaSets ensure a specific number of identical pods run for scaling stateless apps (e.g., web servers), while DaemonSets guarantee one pod runs on every (or a subset of) node(s) for node-specific tasks like logging or monitoring. The key difference is quantity versus location: ReplicaSets focus on maintaining pod count for availability, whereas DaemonSets ensure pod presence on each node for system-level services. 

ReplicaSet

• Purpose: Maintain a stable set of replica pods for stateless applications, ensuring high availability and scalability.
• Scaling: Scales pods up or down based on the replicas field you define in the manifest.
• Use Case: Running web frontends, APIs, or any application needing multiple identical instances.
• Behavior: If a pod dies, it creates a new one to meet the replica count; if a node fails, it tries to reschedule elsewhere. 

DaemonSet

• Purpose: Run a single copy of a pod on every (or specific) node in the cluster for node-specific tasks.
• Scaling: Automatically adds a pod when a new node joins the cluster and removes it when a node leaves.
• Use Case: Logging agents (Fluentd, Elastic Agent), monitoring agents (Prometheus node-exporter), or storage daemons.
• Behavior: Ensures that a particular service runs locally on each machine for local data collection or management. 

References:

DaemonSet | Kubernetes

DevOps Interview: Replica sets vs Daemon sets - DEV Community

Elasticsearch Nodes

Elasticsearch nodes are individual instances of Elasticsearch servers that are part of a cluster. Each node stores data and participates in the cluster’s indexing and search capabilities, playing a critical role in the distributed architecture of Elasticsearch.​

Key Points about Elasticsearch Nodes:

A node is a single server or instance running Elasticsearch, identified by a unique name.

Nodes collectively form a cluster, which is a group of Elasticsearch nodes working together.

Nodes can have different roles:

• Master Node: Manages the cluster state and handles cluster-wide actions like adding/removing nodes and creating/deleting indices.
• Data Node: Stores data and executes data-related operations such as searches and aggregations.
• Client (Coordinating) Node: Routes requests to the appropriate nodes but does not hold data.
• Other special roles include ingestion and machine learning nodes.

Nodes communicate through TCP ports (commonly 9200 for REST API and 9300 for node-to-node communication).

Elasticsearch distributes data across nodes using shards, enabling horizontal scalability, fault tolerance, and high availability.​

In essence, nodes are the building blocks of an Elasticsearch cluster, with each node running on a server (physical or virtual) and working in coordination to provide fast search and analytics on distributed data.

To list all nodes with their attributes we can run this command in Kibana DevTools:

GET /_cat/nodes?v

Output example:

ip            heap.percent ram.percent cpu load_1m load_5m load_15m    node.role       master name

10.199.43.136           44          61   5    1.69    1.71     1.51 cdfhilmrstw -      default-2

10.199.6.164            38          55   4    0.96    1.40     1.33 cdfhilmrstw -      default-1

10.199.30.70            25          51   9    1.61    1.57     1.06 cdfhilmrstw -      data-0

10.199.38.215           46         100  13    1.69    1.71     1.51 cdfhilmrstw -      data-1

10.199.1.249            81          76  30    0.96    1.40     1.33 cdfhilmrstw *      monitoring-1

10.199.32.134           75         100  27    1.69    1.71     1.51 cdfhilmrstw -      monitoring-0

10.199.23.94            77         100  26    1.61    1.57     1.06 cdfhilmrstw -      monitoring-2

10.199.18.75            23          91  19    1.61    1.57     1.06 cdfhilmrstw -      default-0

10.199.15.193           59          56   5    0.96    1.40     1.33 cdfhilmrstw -      data-2

Node Attributes

node_id 

• e.g. aZB9fwOuRWCpINHh6IrOJg",

node_name

• monitoring-2

transport_address

• e.g. 10.2.31.148:9300

node_attributes

• ml.allocated_processors_double
• e.g. 1.0
• ml.config_version
• e.g. 12.0.0
• transform.config_version
• e.g. 10.0.0
• xpack.installed
• true
• ml.allocated_processors
• e.g. 1
• k8s_node_name
• ip-10-2-30-209.us-east-2.compute.internal
• ml.machine_memory
• 6442450944
• ml.max_jvm_size
• e.g. 2147483648
• type
• TODO: how it's set?
• e.g. monitoring
• used by filter decider which decides whether index shard should be allocated on the node by checking if node has filter required by index. If it doesn't we'll see a messages like this

"deciders": [

  ...

  {

    "decider": "filter",

    "decision": "NO",

    "explanation": """node does not match index setting [index.routing.allocation.require] filters [type:"monitoring"]"""

  },

  ...

]

 

roles

• data
• data_cold
• data_content
• data_frozen
• data_hot
• data_warm
• ingest
• master
• ml
• remote_cluster_client
• transform

---

Elasticsearch Indices

An Elasticsearch index is a logical namespace that stores and organizes a collection of related JSON documents, similar to a database table in relational databases but designed for full-text search and analytics. 

Each index is uniquely named and can contain any number of documents, where each document is a set of key-value pairs (fields) representing your data.​

Key Features of an Elasticsearch Index

• Structure: An index is comprised of one or more shards, which are distributed across nodes in the Elasticsearch cluster for scalability and resilience.​
• Mapping and Search: Indexes define mappings that control how document fields are stored and searched.
• Indexing Process: Data is ingested and stored as JSON documents in the index, and Elasticsearch builds an inverted index to allow for fast searches.​
• Use Case: Indices are used to organize datasets in log analysis, search applications, analytics, or any scenario where rapid search/retrieval is needed.​

In summary, an Elasticsearch index is the foundational storage and retrieval structure enabling efficient search and analytics on large datasets.

When analysing an arbitrary index, we want to know:

• its size
• shards
• their number
• allocation - on which nodes they are allocated (and allocation criteria: which node types should these shards be allocated to)
• does it have any data retention defined (Index Lifecycle Policy)
• historical rate/growth of storage usage for data

Index Lifecycle Policy (ILM)

An Index Lifecycle Management (ILM) policy defines what happens to an index as it ages — automatically. It’s a set of rules for retention, rollover, shrink, freeze, and delete.

Example:

PUT _ilm/policy/functionbeat

{

  "policy": {

    "phases": {

      "hot": {

        "actions": {

          "rollover": { "max_age": "30d", "max_size": "50GB" }

        }

      },

      "delete": {

        "min_age": "90d",

        "actions": { "delete": {} }

      }

    }

  }

}

This says:

• Keep the index hot (actively written to) until it’s 30 days old or 50 GB big.
• Then roll over (create a new index and switch writes to it).
• After 90 days, delete the old index.

ILM can be applied to a standard (non–data stream) index. We can attach an ILM policy to any index, not just data streams. However, there’s a big difference:

• Rollover alias required:
• Standard Index:Yes. We must manually set up an alias to make rollover work!
• Data Stream: No (handled automatically - Elastic manages the alias and the backing indices)
• Multiple backing indices
• Standard Index: Optional (via rollover)
• Data Stream: Always (that’s how data streams work)
• Simplified management
• Standard Index: Manual setup
• Data Stream: Built-in

Index vs Data Stream

If we have a continuous stream of documents (e.g. logs) being written to Elasticsearch, we should not write them to a regular index as its size will grow over time and we'll need to keep increasing a node storage. Instead, we should consider one of the following options:

1. Data Stream
2. Index with ILM policy which defines a rollover conditions

What does rollover mean for a standard index?

When a rollover is triggered (by size, age, or doc count):

• Elasticsearch creates a new index with the same alias.
• The alias used for writes (e.g. functionbeat-write) is moved from the old index to the new one.
• Functionbeat or Logstash continues writing to the same alias, unaware that rollover happened.

Example:

# Initially

functionbeat-000001  (write alias: functionbeat-write)

# After rollover

functionbeat-000001  (read-only)

functionbeat-000002  (write alias: functionbeat-write)

This keeps the write flow continuous and allows you to:

• Manage old data (delete, freeze, move to cold tier)
• Limit index size for performance

How to apply ILM to a standard index?

Here’s a minimal configuration:

PUT _ilm/policy/functionbeat

{

  "policy": {

    "phases": {

      "hot": {

        "actions": {

          "rollover": { "max_age": "30d", "max_size": "50GB" }

        }

      },

      "delete": {

        "min_age": "30d",

        "actions": { "delete": {} }

      }

    }

  }

}

PUT _template/functionbeat

{

  "index_patterns": ["functionbeat-*"],

  "settings": {

    "index.lifecycle.name": "functionbeat",

    "index.lifecycle.rollover_alias": "functionbeat-write"

  }

}

The following command creates a new index called functionbeat-000001 (if it doesn’t already exist). If the index does exist, it updates the aliases section. It creates an alias named functionbeat-write that points to this index. (Aliases are like virtual index names — you can send reads or writes to the alias instead of a specific index. They’re lightweight and flexible.). is_write_index: true tells Elasticsearch: “When someone writes to this alias, route the write operations to this index.” If you later have: functionbeat-000001, functionbeat-000002 and both share the alias functionbeat-write, then only the one with "is_write_index": true will receive new documents.

PUT functionbeat-000001

{

  "aliases": {

    "functionbeat-write": { "is_write_index": true }

  }

}

ILM rollover works by:

• Watching the alias (functionbeat-write), not a specific index.
• When rollover conditions are met (e.g. 50 GB or 30 days), Elasticsearch:
• Creates a new index (functionbeat-000002)
• Moves "is_write_index": true from 000001 to 000002. From that moment, all new Functionbeat writes go to the new index — automatically.

After rollover:

• functionbeat-000001 becomes read-only, but still searchable.
• ILM will later delete it when it ages out (based on your policy).

So that last command effectively bootstraps the first generation of an ILM-managed index family.

• ILM policy: Automates rollover, delete, etc.
• Rollover action: Creates a new index and shifts the alias
• Alias requirement: Required, used for write continuity
• Data stream alternative: Better option, handles rollover and aliasing for you

Index Template

Index templates do not retroactively apply to existing indices. They only apply automatically to new indices created after the template exists.

When we define an index template like:

PUT _index_template/functionbeat

{

  "index_patterns": ["functionbeat-*"],

  "template": {

    "settings": {

      "index.lifecycle.name": "functionbeat"

    }

  }

}

That template becomes part of the index creation logic.

So:

When a new index is created (manually or via rollover),

→ Elasticsearch checks all templates matching the name.

→ The matching template(s) are merged into the new index settings.

Existing indices are not touched or updated.

If we already have an index — e.g. functionbeat-8.7.1 — that matches the template pattern, it won’t automatically get the template settings.

We need to apply those manually, for example:

PUT functionbeat-8.7.1/_settings

{

  "index.lifecycle.name": "functionbeat",

  "index.lifecycle.rollover_alias": "functionbeat-write"

}

Now the existing index is under ILM control (using the same settings the template would have applied if it were created fresh).

Elasticsearch treats index templates as blueprints for new indices, not as live configurations.

This is intentional — applying settings automatically to existing indices could cause:

• unintended allocation moves,
• mapping conflicts,
• or lifecycle phase resets.

We want to keep as least as possible data in Elasticsearch. If data stored are logs, we want to:

• make sure apps are sending only meaningful logs
• make sure we capture repetitive error messages so the app can be fixed and stop emitting them

Component templates are building blocks for constructing index templates that specify index mappings, settings, and aliases.

If index template consists of several component templates and some of these templates impose ILM policies, ILM policy of the last component template wins and is chosen to be the one applied to index or data stream based on index template. 

Data retention is applied at the document level while ILM policy is applied at the index level, ILM policies take precedence. Data retention is defined in Component Templates.

Shards and Replicas

We can set the number of shards and replicas per index in Elasticsearch when we create the index, and we can dynamically update the number of replicas (but not the number of primary shards) for existing indices.​

Setting Shards and Replicas on Index Creation

Specify the desired number in the index settings payload:

PUT /indexName

{

  "settings": {

    "index": {

      "number_of_shards": 6,

      "number_of_replicas": 2

    }

  }

}

This creates the index with 6 primary shards and 2 replicas per primary shard.​

Adjusting Replicas After Creation

You can adjust the number of replicas for an existing index using the settings API:

PUT /indexName/_settings

{

  "index": {

    "number_of_replicas": 3

  }

}

Replicas can be changed at any time, but the number of primary shards is fixed for the lifetime of the index.​

Shard and Replica Principles

Each index has a configurable number of primary shards.

Each primary shard can have multiple replica shards (copies).

Replicas improve fault tolerance and can spread search load.​

We should choose shard and replica counts based on data size, node count, and performance needs. Adjusting these settings impacts resource usage and indexing/search performance.

Index Size

To find out the size of each index (shards) we can use the following Kibana DevTools query:

GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason,node,store&s=store:desc

The output contains the following columns:

• index - index name
• shard - order number of a (primary) shard. If we have 2 shards and 2 replicas, we'd have 4 rows, with shard=0 for first two rows (first primary and replica) and shard=1 for next two rows (second primary and replica)
• prirep - is shard a primary (p) or replica (r)
• state - e.g. STARTED
• unassigned
• reason
• node - name of the node
• store - used storage (in gb, mb or kb)

Each shard should not be larger than 50GB. We can impose this via Index Lifecycle Policy where we can set rollover criteria.

Document Routing 

Routing in an Elasticsearch cluster determines which shard a document is sent to and stored in. The default process uses a hash of the document's _id to find the shard, ensuring an even distribution. However, you can implement custom routing to a specific field value to ensure related documents are on the same shard, which improves search performance by reducing the scope of queries. 

How routing works

Default routing: When a document is indexed, Elasticsearch calculates the target shard by hashing the document's ID and using the formula shard = hash(_routing) % number_of_primary_shards. The default _routing value is the _id of the document.

Custom routing: You can specify a different routing value, like a user ID or a country code, by providing it during indexing. This directs all documents with the same routing value to the same shard, which can significantly speed up queries that filter by that value.

Querying with custom routing: When you perform a query, you can provide the same routing value. 

Elasticsearch will then only search the specific shard containing documents with that value, rather than searching all shards in the cluster. 

Benefits of custom routing

Improved search speed: By narrowing the search to a specific shard, you reduce the amount of data that needs to be searched, leading to faster results.

Efficient resource use: Routing minimizes the computational overhead on the cluster because nodes don't have to process queries that are irrelevant to their data.

Scalability for multitenant applications: Routing is crucial for horizontal scaling in applications with multiple tenants, as it can isolate each tenant's data to specific shards. 

Considerations

Data distribution: If you use a custom routing value, ensure the data is relatively evenly distributed across all shards. If one shard accumulates a disproportionate amount of data, it can create performance bottlenecks.

Security: For multitenant applications, the application layer must handle security and access checks to prevent users from querying data from another user's shard, as Elasticsearch does not enforce this isolation automatically. 

Shard Allocation 

Just as there are rules which determine in which shard should a document be written, there are rules which determine onto which node a shard should be pushed to (allocated).

 

Routing allocation watermarks in an Elasticsearch cluster are thresholds that control shard allocation based on a node's disk usage to prevent the cluster from running out of space. The three main watermarks are low (stops allocating new shards), high (relocates existing shards), and flood stage (makes indices read-only). These settings help maintain stability by proactively managing disk space, but should be adjusted based on cluster needs and capacity.

 

Watermark thresholds

Low watermark: The default is 85%. When a node's disk usage exceeds this limit, Elasticsearch stops allocating new shards to that node.

High watermark: The default is 90%. If a node's disk usage goes above this threshold, Elasticsearch starts relocating existing shards away from that node to other nodes with more available space.

Flood stage watermark: The default is 95%. When this threshold is reached, Elasticsearch makes all indices on that node read-only to prevent further data from being written, though reads are still possible. 

Configuration and use cases

Monitoring and prevention: These settings are crucial for preventing nodes from running out of storage, which can cause shard failures and instability.

Proactive scaling: You can set the thresholds based on your infrastructure's growth. For instance, if you anticipate nodes filling up quickly, you might set lower thresholds to proactively distribute the load.

Dynamic systems: Using percentage values is best for dynamic systems where disk sizes can vary. You can also use absolute byte values for fixed-size storage environments.

Cluster settings: These settings are cluster-wide and are managed through the Elasticsearch configuration file (elasticsearch.yml) or by using the cluster update settings API.

Node role-specific settings: For advanced setups, you can configure different flood stage watermarks for nodes with different roles, such as hot, warm, and cold nodes, allowing for more tailored allocation strategies. 

How to manage them

Review current settings: You can check the current watermark settings using the 

cluster.routing.allocation.disk.watermark.low, high, and flood_stage settings in the cluster settings API.

Change settings: To modify the watermarks, update the cluster settings via the API or by editing the elasticsearch.yml configuration file on the master nodes and restarting.

Recommendations: It's recommended to have enough buffer space (e.g., 3x the size of your largest shard) to handle shard relocation and potential growth.

To check the current allocation watermarks (for each node) in cluster:

GET _cluster/settings?include_defaults=true

Output snippet:

"routing": {

  "use_adaptive_replica_selection": "true",

  "rebalance": {

    "enable": "all"

  },

  "allocation": {

    "enforce_default_tier_preference": "true",

    "node_concurrent_incoming_recoveries": "2",

    "node_initial_primaries_recoveries": "4",

    "desired_balance": {

      "max_balance_computation_time_during_index_creation": "1s",

      "progress_log_interval": "1m",

      "undesired_allocations": {

        "log_interval": "1h",

        "threshold": "0.1"

      }

    },

    "same_shard": {

      "host": "false"

    },

    "total_shards_per_node": "-1",

    "type": "desired_balance",

    "disk": {

      "threshold_enabled": "true",

      "reroute_interval": "60s",

      "watermark": {

        "flood_stage.frozen.max_headroom": "20GB",

        "flood_stage": "95%",

        "high": "90%",

        "low": "85%",

        "flood_stage.frozen": "95%",

        "flood_stage.max_headroom": "100GB",

        "low.max_headroom": "200GB",

        "high.max_headroom": "150GB"

      }

    },

    "awareness": {

      "attributes": [

        "k8s_node_name"

      ]

    },

    "balance": {

      "disk_usage": "2.0E-11",

      "index": "0.55",

      "threshold": "1.0",

      "shard": "0.45",

      "write_load": "10.0"

    },

    "enable": "all",

    "node_concurrent_outgoing_recoveries": "2",

    "allow_rebalance": "always",

    "cluster_concurrent_rebalance": "2",

    "node_concurrent_recoveries": "2"

  }

},

We can see routing.allocation.disk.watermark settings.

If allocation of some shard onto node of target type fails, we can check the reason:

GET /_cluster/allocation/explain

...might have the output which unveils that the root cause for shard being unassigned is no enough storage on that node:

  "node_allocation_decisions": [

    {

      "node_id": "8r4E9pZL........wwAw",

      "node_name": "data-0",

      "transport_address": "10.22.31.122:9300",

      "node_attributes": {

        "k8s_node_name": "ip-10-22-18-240.us-east-1.compute.internal",

        "ml.machine_memory": "8589934592",

        "ml.max_jvm_size": "3221225472",

        "type": "data",

        "ml.allocated_processors_double": "1.0",

        "ml.config_version": "12.0.0",

        "transform.config_version": "10.0.0",

        "xpack.installed": "true",

        "ml.allocated_processors": "1"

      },

      "roles": [

        "data",

        "data_cold",

        "data_content",

        "data_frozen",

        "data_hot",

        "data_warm",

        "ingest",

        "master",

        "ml",

        "remote_cluster_client",

        "transform"

      ],

      "node_decision": "no",

      "deciders": [

        {

          "decider": "disk_threshold",

          "decision": "NO",

          "explanation": "the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85%], having less than the minimum required [95.8gb] free space, actual free: [67.5gb], actual used: [89.4%]"

        }

      ]

    },

---