Showing posts with label ELK. Show all posts
Showing posts with label ELK. Show all posts

Thursday, 25 June 2026

Elastic Fleet


Elastic Agents do not strictly require Fleet and Fleet Server. You can deploy them in standalone mode, which allows you to manually configure and manage them using local configuration files.

However, running Elastic Agents in Fleet-managed mode with a Fleet Server is the recommended best practice for most enterprise environments.

Here is how the two approaches compare:

1. Fleet-Managed Mode (Recommended)


In this setup, you use the Fleet UI in Kibana to centrally manage agent policies, roll out upgrades, and apply integrations.
  • How it works: Agents connect to a Fleet Server (which is just a specialized Elastic Agent process), which then receives policies from Elasticsearch and pushes them to your endpoints.
  • Best for: Large-scale deployments, continuous monitoring, and Elastic Security/Endpoint integrations.
  • Action: To set this up, refer to the official Elastic Agent Installation Guide.


2. Standalone Mode


In this setup, you manually install the agent and define its inputs, outputs, and integrations directly in a local YAML configuration file.

  • How it works: The agent connects directly to outputs like Elasticsearch or Logstash without a Fleet Server intermediary.
  • Limitations: Central management, automated upgrades, and certain advanced Endpoint Security features are disabled.
  • Best for: Edge cases, highly air-gapped networks, or evaluating specific integrations.
  • Action: To configure this, use the steps outlined in the Standalone Elastic Agent Tutorial.


Example of how an Elastic Fleet can actually be wired


Components & where they live


Everything is in cluster company-prod-elastic-eks (EKS, us-east-2, now on v1.36.2), namespace elastic-system, across 5 nodes in 3 AZs:

Node            AZ  Node Group             Runs
====            ==  ==========             ====
ip-10-99-44-1   2a  es MNG (m7g.2xlarge)   agent
ip-10-99-55-34  2b  default MNG (m5.large) agent + Fleet Server pod
ip-10-99-55-89  2b  es MNG (m7g.2xlarge)   agent
ip-10-99-66-51  2c  es MNG (m7g.2xlarge)   agent
ip-10-99-66-78  2c  default MNG (m5.large) agent


  • Fleet Server — Deployment fleet-server-prod, replicas: 1 (single pod, currently on the 2b node). Listens on :8220 HTTPS. Exposed by a NodePort service fleet-server-prod-agent-http (8220 --> 32202 on every node).
  • Elastic Agents — DaemonSet agent-prod-eck-agent, one pod per node (5 total), spread across all 3 AZs. hostNetwork: true, mode: fleet, FLEET_INSECURE=true. They collect node/pod logs + metrics (hostPath mounts of /var/log/...).
  • ALB k8s-elastics-eckfleet-a1b2c3d4e5 — internal (not internet-facing), spanning subnets in 2a / 2b / 2c. Created by the AWS LB controller from Ingress eck-fleet-server-prod-ingress. Targets = all 5 nodes' NodePort 32202 (instance mode), backend-protocol: HTTPS, idle_timeout: 300s.
  • DNS — eck-fleet-server.internal-domain.local (Route53 via external-dns) -->  the internal ALB. Elasticsearch is a separate endpoint, elasticsearch.internal-domain.local:443.

Who listens, who initiates


The key thing: every connection is initiated by the agent (outbound). Nothing is ever pushed to the agents — that's why this works with agents behind hostNetwork and no inbound rules of their own.

                          

                           ┌───────────────────────────────────────────────┐
                           │ cluster company-prod-elastic-eks / elastic-sys│
     ┌──────────┐          │                                               │
     │ Agent    │ check-in │   internal ALB          Fleet Server (1 pod)  │
     │ DaemonSet│─────────►│  eckfleet-a1b2...   ┌──────► :8220 (HTTPS)    │
     │ (5 pods, │  :443    │  :443 TLS  ─────────┘  NodePort 32202→8220    │
     │  3 AZs)  │  HTTPS   │  (internal cert)       kube-proxy → the 1 pod │
     └────┬─────┘          │       ▲  re-encrypt HTTPS to a node's :32202  │
          │                │       │                     │                 │
          │ ship data      └───────┼─────────────────────┼─────────────────┘
          │ (logs/metrics)         │ DNS                 │ writes .fleet-* / agent
          ▼                   eck-fleet-server.          ▼ metadata, reads policy
 elasticsearch.               internal-domain.local  Elasticsearch + Kibana (Fleet app)
 internal-domain.local:443 ◄──────────────────────  
 (NOT via the fleet ALB)



Traffic Flows: Breakdown


1. Agent --> Fleet Server (Control Plane: Enrollment & Policy Check-in)

The agent acts as the client, and Fleet Server listens on :8220. The agent dials https://eck-fleet-server.internal-domain.local:443:

Traffic Path:

agent --> DNS --> internal ALB :443 (TLS termination, internal cert) --> re-encrypt --> a node's NodePort :32202 -->  kube-proxy --> the single Fleet Server pod :8220

Cross-AZ Routing: 

Because the agent always targets the ALB DNS name (not the pod directly), an agent can land on any node's NodePort. It is then forwarded by kube-proxy to whichever node is hosting the single Fleet Server pod—making cross-AZ hops standard behavior.

The Long Poll Mechanism: The check-in is a long poll. The agent opens a connection, and Fleet Server holds it open for up to ~5 minutes until a policy change occurs or the poll times out.

The Latency "False Positive": This held-open duration is exactly what the ALB records as a TargetResponseTime of ~300s, which triggered the false-positive alert. The ALB's idle_timeout is intentionally set to 300s to support these long polls. Conversely, the health check (/health, expects 404, 300s interval) is a separate lightweight probe, explaining why targets report as perfectly healthy while "latency" metrics look highly skewed.


2. Agent --> Elasticsearch (Data Plane: Telemetry Shipping)

This data path does not traverse the Fleet ALB. The Fleet output configuration points agents directly at https://elasticsearch.internal-domain.local:443.

Consequently, the eckfleet ALB only handles control-plane check-in traffic. This explains why its request volume is incredibly low (~3–5 requests/min) while individual request durations hover around 300s. They are two entirely separate paths utilizing two different load balancers.


3. Fleet Server --> Elasticsearch + Kibana

Fleet Server (via elasticsearchRefs / kibanaRef) acts as the client here. It writes agent and policy metadata directly into the .fleet-* indices and coordinates with the Kibana Fleet application, where the eck-fleet-server and eck-agent policies are managed.


One-Line Summary


Agents (DaemonSet, 5 pods across 2a/2b/2c) long-poll the single Fleet Server pod for policy updates via the internal ALB on :443 --> NodePort 32202 --> :8220, but ship their actual telemetry data directly to Elasticsearch, completely bypassing that ALB. The ALB only processes the slow, held-open control traffic—which is the root cause of the  alarm.



Why does fleet server holds connection open for so long? Why it does not reply immediately?                                                                                                                                                                                   


It's a deliberate design choice called long polling, and it's the opposite of a bug. Here's the reasoning.

The problem Fleet is solving

Fleet Server's job is to push policy changes out to agents promptly — a new integration, a changed log path, an output credential rotation. Agents need to find out "has my policy changed?" with low latency.

There are three ways an agent could learn about changes:

  1. Short polling — agent asks "anything new?" every N seconds, server replies instantly "no", agent sleeps, repeats.
  2. Server push — server opens a connection to each agent when something changes.
  3. Long polling — agent asks "anything new?", and the server holds the request open until either something actually changes or a timeout fires.

Fleet uses #3, and the connection sitting open for ~5 minutes is that hold.

Why not reply immediately (short polling)?


If Fleet answered every check-in instantly with "nothing changed", then to get fast reaction to a policy change, agents would have to poll very frequently — say every few seconds. With your 5 agents that's tolerable, but Fleet is built to manage thousands to tens of thousands of agents from one server. At that scale:

  - Frequent short polls = a constant storm of requests, almost all of which return "no change". Huge wasted CPU/network on both ends.
  - To keep reaction time low you'd poll more often, which makes the storm worse. To cut the storm you'd poll less often, which makes policy changes take longer to land. There's no good setting.

Long polling breaks that trade-off: the agent gets a near-instant reaction to a real change (the server responds the moment policy updates) and there's almost no idle chatter (one held-open connection per agent instead of hundreds of empty round-trips).

Why not server push (#2)?


Pushing would mean the server initiating connections inward to every agent. Agents are all over the place — behind NAT, firewalls, in private subnets, on laptops, on hostNetwork pods like yours. The server usually can't reach them, and you'd need inbound rules everywhere. Long polling flips it: the agent always dials out to the server, the connection is already established and held open, and the server pushes the change down that existing agent-initiated connection the instant it happens. You get push-like latency with poll-like (outbound-only) connectivity. That's exactly why your agents work behind hostNetwork with no inbound exposure.

 So what's actually happening in those ~300s


A check-in is essentially: "Here's my current state; tell me the moment my policy differs." The server parks that request. Two things can end it:

  - A policy change occurs → server responds immediately with the new policy (could be 2 seconds in).
  - Nothing changes → the server lets the request time out at its poll ceiling (~5 min), responds "no change", and the agent immediately opens a fresh one.

Since your policies rarely change, almost every check-in runs the full clock and returns at ~300s. That's the held connection the ALB measures as TargetResponseTime.

Why this specifically fools the ALB


 From the ALB's point of view, "request received → response sent" took 300 seconds, so it reports TargetResponseTime ≈ 300s. The ALB can't tell the difference between "the backend was slow for 300s" (bad) and "the backend intentionally held an idle long-poll for 300s" (normal). That ambiguity is the whole reason your generic 0.8s threshold misfires — and why the right fix (DOP-833) is to exempt this endpoint rather than treat it as latency. It's also why the Ingress sets idle_timeout.timeout_seconds=300: the ALB has to be told to tolerate the held connection, otherwise it would sever the long poll before the agent's poll cycle completes.

In short: Fleet holds the connection open so it can deliver policy changes near-instantly without either hammering the server with empty polls or needing to reach inward to firewalled agents. The ~300s is just an idle long-poll waiting for a change that usually never comes during that window — efficient by design, and only a problem for a latency metric that doesn't know to expect it.


----

Posted by at No comments:
Labels: , ,

Thursday, 12 June 2025

Useful Kibana DevTools Queries






Elasticsearch’s query DSL is structured by query types.

Every top-level query must be one of the defined types:
  • match
  • term
  • range
  • bool
  • wildcard
  • query_string
  • function_score
  • etc.

Main APIs:

  • _cat - for a human-readable summary
  • _stats - for a detailed JSON response

Failed Queries


Example:

{
  "statusCode": 502,
  "error": "Bad Gateway",
  "message": "Client request timeout for: https://my.elastic-system.svc:9200 with request GET /my_index/_search?pretty=true"
}


A 502 Bad Gateway combined with a timeout usually means the Elasticsearch engine is struggling to process the request, or there is a networking bottleneck between Kibana and the database.



Cluster

To check cluster health:

GET /_cluster/health

GET /_cluster/health?level=shards

The output contains status which can be green, yellow or red.

To check status of each shard:

GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason,node

The output shows if shard is primary (p) or replica (r). It also shows the status which can be e.g. STARTED, UNASSIGNED  and reason which can be e.g. ALLOCATION_FAILED.

To sort the output by some column we can use s parameter:

GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason,node,store&s=state
GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason,node,store&s=node
GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason,node,store&s=index 

To sort in descending order, append :desc to the name of the sorted column:

GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason,node,store&s=store:desc



To get memory allocation and consumption per node:

GET /_cat/allocation?v&s=node

The output contains the following columns:
  • shards (number)
  • shards.undesired
  • write_load.forecast
  • disk.indices.forecast (in Gb or Tb)
  • disk.indices (in Gb or Tb)
  • disk.used (in Gb or Tb)
  • disk.avail (in Gb or Tb)
  • disk.total (in Gb or Tb)
  • disk.percent (number, %)
  • host (IP address)
  • ip (IP address)
  • node (node name or UNASSIGNED)
  • node.role (combination of cdfhilmrstw)

If some shard is not allocated, we can check the reason:

GET /_cluster/allocation/explain

To manually trigger retry of all previously failed shard allocations:

POST /_cluster/reroute?retry_failed=true

To check the progress, check the health of the cluster and:

GET /_cat/recovery/my_index?v



Index



In Elasticsearch, every index has a Mapping. Think of it like a database table definition. If a field isn't explicitly defined or hasn't been automatically detected from an uploaded document, Elasticsearch acts like that field doesn't exist. You can't sort by a column that the database doesn't know about!

To find out which fields exist in index:

GET /my_index/_mapping


To perform a search operation on a specific index:

GET /my_index/_search 

By itself (without a request body), it returns the first 10 documents by default. This request is the same as the above one:

GET /my_index/_search
{
  "query": {
    "match_all": {}
  }
}

In Kibana's Dev Tools, the query parameter in a GET request refers to the search query that defines which documents we want to retrieve from Elasticsearch. It's part of the request body and specifies the search criteria. The query parameter essentially tells Elasticsearch "find me documents that match these conditions." It's the core part of any search request and determines which documents from our index will be returned in the response.

The query object can contain various types of queries. Common query types:

match_all - Returns all documents:

{
  "query": {
    "match_all": {}
  }
}

match - Full-text search on a specific field:

{
  "query": {
    "match": {
      "field_name": "search_term"
    }
  }
}

term - Exact term matching:

{
  "query": {
    "term": {
      "status": "active"
    }
  }
}



To find all documents written in past 1 minute:

GET my_index/_search
{
  "query": {
    "range": {
        "timestamp": {
            "gte": "now-1m",
            "lte": "now"
        }
    }
  }
}


For X days use: Xd
For X hours use: Xh

bool


bool - Combine multiple queries with logical operators:

{
  "query": {
    "bool": {
      "must": [
        {"match": {"title": "elasticsearch"}},
        {"range": {"date": {"gte": "2023-01-01"}}}
      ]
    }
  }
}


bool is the workhorse of combinational querying in Elasticsearch.
It's like the logical brain of the query DSL.

A bool query lets you combine multiple conditions (AND, OR, NOT) into one query. Think of it like this:

(bool)
 ├── must → AND conditions
 ├── filter → AND but cheap
 ├── should → OR conditions
 └── must_not → NOT conditions


Without a bool query, Elasticsearch can only run one query condition at a time. But real searches need multiple conditions.

For example:
  • logs in the last 7 days
  • log_group is X
  • success = false
  • AND NOT status=200
  • AND (message contains A OR B)

You need logical operators → that's what bool provides.

Within bool we can use filter instead of must. 

filter is not a standalone query. It is only one part of the bool query type.

Filters are:
  • cached
  • more efficient
  • ideal for exact and boolean conditions

Everything inside filter is combined with AND.

"filter": [
  { cond1 },
  { cond2 },
  { cond3 }
]


This means:

cond1 AND cond2 AND cond3



filter is not a query type — it is an instruction telling the bool query how to treat subqueries (as cached, non-scoring, mandatory matches). 

So this:

"filter": [...]

doesn’t make sense by itself — filter what?

Whereas this:

"bool": {
  "filter": [...]
}

...means: "Run these filter queries together inside a boolean query."

Example:

GET logs-*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "now-7d",
              "lte": "now"
            }
          }
        },
        {
          "term": {
            "log_group": "/aws/lambda/my-lambda"
          }
        },
        {
          "term": {
            "mycorp.message.my-lambda.success": false
          }
        }
      ]
    }
  }
}


All must clauses must also match.

"must": [
  { cond1 },
  { cond2 }
]

Meaning:

cond1 AND cond2


Inside should, clauses are OR unless minimum_should_match makes them mandatory.

"should": [
  { cond1 },
  { cond2 }
]

Meaning:

cond1 OR cond2

must_not: All must NOT match.

"must_not": [
  { cond1 }
]

Meaning:

NOT cond1



range - Query for values within a range:

{
  "query": {
    "range": {
      "age": {
        "gte": 18,
        "lte": 65
      }
    }
  }
}


To get the number of documents in an Elasticsearch index, you can use the _count API or the _stats API.

GET /my_index/_count

This will return a response like:

{
  "count": 12345,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  }
}


To get a certain number of documents, use size argument:

GET my_index/_search?size=900

We can also use _cat API:

GET /_cat/count/my_index?v

This will return output like:

epoch      timestamp count
1718012345 10:32:25  12345


GET /my_index/_stats

"indices": {
  "my_index": {
    "primaries": {
      "docs": {
        "count": 12345,
        "deleted": 12
      }
    }
  }
}


To get the union of all values of some field e.g. channel_type field across all documents in the my_index index, we can use an Elasticsearch terms aggregation:


GET my_index/_search
{
  "size": 0, 
  "aggs": {
    "unique_channel_types": {
      "terms": {
        "field": "channel_type.keyword",
        "size": 10000  // increase if you expect many unique values
      }
    }
  }
}


Explanation:
  • "size": 0: No documents returned, just aggregation results.
  • "terms": Collects unique values.
  • "channel_type.keyword": Use .keyword to aggregate on the raw value (not analyzed text).
  • "size": 10000: Max number of buckets (unique values) to return. Adjust as needed.

Response example:

{
  "aggregations": {
    "unique_channel_types": {
      "buckets": [
        { "key": "email", "doc_count": 456 },
        { "key": "push", "doc_count": 321 },
        { "key": "sms", "doc_count": 123 }
      ]
    }
  }
}

The "key" values in the buckets array are your union of channel_type values.


Let's assume that my_index has the timestamp field (as the root field...but it can be at any path in which case we'd need to adjust the query) is correctly mapped as a date type.


To get the oldest document:

GET my_index/_search
{
  "size": 1,
  "sort": [
    { "@timestamp": "asc" }
  ]
}


To get the newest document:

GET my_index/_search
{
  "size": 1,
  "sort": [
    { "@timestamp": "desc" }
  ]
}


Sorting by a field that isn't indexed or doc_values-enabled (especially on a large or unoptimized index) can cause the memory usage to spike and the request to hang until it times out.


How to get all possible values of some field in all documents added to index in last 24 hours?

We can use Terms Aggregation with Range Query:


GET /my_index/_search
{
  "size": 0,
  "query": {
    "range": {
      "@timestamp": {
        "gte": "now-24h/h",
        "lte": "now"
      }
    }
  },
  "aggs": {
    "unique_values": {
      "terms": {
        "field": "my_field.keyword",
        "size": 10000
      }
    }
  }
}


Check number of documents which are older than N days:


POST my_index/_count
{
  "query": {
    "range": {
      "@timestamp": {
        "lt": "now-Nd/d"
      }
    }
  }
}

Delete all documents older than N days:

POST my_index/_delete_by_query?conflicts=proceed&wait_for_completion=false
{
  "query": {
    "range": {
      "@timestamp": {
        "lt": "now-Nd/d"
      }
    }
  }
}

The output of the above command is task ID:

{
  "task": "BeJWGidDTtWkRL9aQEkjhg:26425516"
}

To check all tasks, grouped by node on which they are running:

GET _tasks

The output shows task's id, action name (e.g. "indices:data/write/bulk[s][p]", "cluster:monitor/tasks/lists[n]"), type (e.g. transport, monitoring, ...).

To check the task status:

GET _tasks/<task_id>

To check if any delete_by_query task is running and number of docs deleted so far:

GET _tasks?actions=*delete/byquery&detailed=true


Once delete_by_query task is completed: deletion is done, but disk space might not yet be reclaimed. To free disk space, run a forcemerge:

POST my_index/_forcemerge?only_expunge_deletes=true

For a huge shard, consider doing this after several weekly chunks, not after every single one, to reduce I/O spikes.

Check if any forcemerge tasks are running:

GET _tasks?actions=*forcemerge
GET _tasks?actions=*forcemerge&detailed=true
GET _tasks?actions=*forcemerge&detailed=true&group_by=parents

Check number of merges:

GET my_index/_stats?level=shards



To get number of shards in index:

GET my-index/_settings/?filter_path=**.number_of_shards


To get number of replicas:

GET my-index/_settings/?filter_path=**.number_of_replicas

Output:

{
  "my-index": {
    "settings": {
      "index": {
        "number_of_replicas": "1"
      }
    }
  }
}


How to find out disk size used by some index?

GET /_cat/indices/your-index-name*?v&h=index,docs.count,store.size,pri.store.size&s=store.size:desc
  • store.size: Total size on disk (includes all primary shards and replica shards).
  • pri.store.size: Size of only the primary shards (useful for knowing the "true" data size without redundancy).
  • s=store.size:desc: Sorts the list by size (useful if you are using a wildcard *).

If you need a precise number for an automated script or a deeper dive into memory usage, use the _stats endpoint.

GET /your-index-name*/_stats/store

In the JSON response, look for:
  • total.store.size_in_bytes: The exact byte count for the whole index (primaries + replicas).
  • primaries.store.size_in_bytes: The exact byte count for just the primary data.

To see which fields take up the most space:

POST /my-index/_disk_usage

To get the frequency of writes (document ingestion) over the last 24 hours, broken down by the hour, you should use a Date Histogram aggregation. In Elasticsearch, "writing into an index" typically equates to the creation of new documents with a @timestamp.

GET /your-index-name*/_search
{
  "size": 0, 
  "query": {
    "range": {
      "@timestamp": {
        "gte": "now-24h",
        "lte": "now"
      }
    }
  },
  "aggs": {
    "writes_per_hour": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "1h",
        "extended_bounds": {
          "min": "now-24h",
          "max": "now"
        }
      }
    }
  }
}


Breakdown of the Query:
  • "size": 0: Tells Elasticsearch we don't want to see the actual documents (the "hits"), just the statistical summary.
  • range: Filters the data to only include documents from the last 24 hours.
  • date_histogram: This is the magic part. It buckets your data by time.
  • fixed_interval: "1h": Groups the results into 1-hour chunks.
  • extended_bounds: Ensures that even if an hour had zero writes, it still shows up in your list as a "0" count rather than being skipped entirely.



----
Posted by at No comments:
Labels: , , , ,

Thursday, 9 January 2025

ELK Stack Interview Questions




Elasticsearch (ES)





Kibana


  • How to install Kibana on bare metal?
    • How to install Kibana in k8s cluster?
  • What are Dashboards?
  • What are Alerts?
  • How to back up and Elastic objects like dashboards and alerts? How to restore them in another Elastic instance?
  • TBC


Posted by at No comments:
Labels: , ,

Friday, 16 February 2024

Introduction to ELK Stack





What is ELK stack?

The ELK stack is a set of tools used for searching, analyzing, and visualizing large volumes of data in real-time. It is composed of three main components:

What is it used for?
  • aggregates logs from all systems and applications
  • logs analytics
  • visualizations for application and infrastructure monitoring, faster troubleshooting, security analytics etc.
image source: https://www.guru99.com/



Logstash


  • Server-side data processing pipeline that ingests (takes in) data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch. 
  • Supports a variety of input sources, such as:
    • log files (log shipper)
    • databases
    • message queues
  • Allows for complex data transformations and filtering
  • Helps easily transform source data and load it into Elasticsearch cluster

Logstash configuration examples:


# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
        file {
                path => "/home/my-app/.pm2/logs/my-app-out.log"
                start_position => "beginning"
                sincedb_path => "/opt/logstash/sincedb-access"
        }
}

filter {
        grok {
                match => { "message" => "%{DATA:timestamp} - info: processRequestMain: my-product: (input|output) sessionid = \{%{GREEDYDATA:session_id}\} (reqXml|resXml) = %{GREEDYDATA:content_xml}" 
       }

        if "_grokparsefailure" in [tags] {
                drop { }
        }  

        xml {
                source => "content_xml"
                target => "content"
        }

        split {
                field => "content[app]"
        }

        mutate {
                add_field => {
                        "env" => "${MYAPP_ENV}"
                        "instance_id" => "${MYAPP_INSTANCE_ID}"                
                }
        }
}

output {
    amazon_es {
        hosts => [ "search-myapp-dev-af6m6cidasgqsnmskxup2fh57y.us-east-1.es.amazonaws.com" ]
        region => "us-east-1"
        index => "logstash-myapp-%{+YYYY.MM.dd}"
    }
}


Elasticsearch


  • Distributed, RESTful search and analytics engine
  • Built on Apache Lucene
  • Used for storing (it is basically a Database), searching, and analyzing large volumes of data (e.g. logs) quickly and in near real-time
  • Scalable, fast, and able to handle complex queries
  • Licensed, not open source
    • OpenSearch is open-sourced alternative (supported by AWS)
    • FluentD is another open-source data collection alternative
  • Data in the form of JSON documents is sent to Elasticsearch using:
    • API
    • Ingestion tools
      • Logstash - e.g. it's pushing logs to ElasticSearch
      • Amazon Kinesis Data Firehose
  • The original document is automatically stored and a searchable reference is added to the document in the cluster’s index
  • Elasticsearch REST-based API is used to manipulate with documents:
    • send
    • search
    • retrieve 
  • Uses schema-free JSON documents
  • Distributed system
    • Enables it to process large volumes of data in parallel, quickly finding the best matches for your queries
  • Operations such as reading or writing data usually take less than a second to complete => Elasticsearch can be used for near real-time use cases such as application monitoring and anomaly detection
  • Has support for various languages: Java, Python, PHP, JavaScript, Node.js, Ruby etc...

Kibana


  • Visualisation and reporting tool
  • Used with Elasticsearch to:
    • visualize the data
    • build interactive dashboards

Filebeat


  • https://www.elastic.co/beats/filebeat
  • log shipper
  • both Filebeat and Logstash can be used to send logs from a file-based data source to a supported output destination
  • Filebeat is a lightweight option, ideal for environments with limited resources and basic log parsing needs. Conversely, Logstash is tailored for scenarios that demand advanced log processing
  • both FB and LS can be used in tandem when building a logging pipeline with the ELK Stack because both have a different function


image source: https://www.guru99.com/




References:



Posted by at No comments:
Labels: , , , ,
Subscribe to: Posts (Atom)