Friday 16 February 2024

Introduction to ELK Stack

What is ELK stack?

What is it used for?
  • aggregates logs from all systems and applications
  • logs analytics
  • visualizations for application and infrastructure monitoring, faster troubleshooting, security analytics etc.


  • distributed search and analytics engine built on Apache Lucene
  • licensed, not open source
    • OpenSearch is open-sourced alternative (supported by AWS)
    • FluentD is another open-source data collection alternative
  • data in the form of JSON documents is sent to Elasticsearch using:
    • API
    • ingestion tools
      • Logstash
      • Amazon Kinesis Data Firehose
  • the original document automatically stored and a searchable reference is added to the document in the cluster’s index
  • Elasticsearch REST-based API is used to manipulate with documents:
    • send
    • search
    • retrieve 
  • uses schema-free JSON documents
  • distributed system
    • enables it to process large volumes of data in parallel, quickly finding the best matches for your queries
  • operations such as reading or writing data usually take less than a second to complete => Elasticsearch can be used for near real-time use cases such as application monitoring and anomaly detection
  • has support for various languages: Java, Python, PHP, JavaScript, Node.js, Ruby etc...
  • .


  • log shipper
  • helps easily transform source data and load it into Elasticsearch cluster
  • .


  • log shipper
  • both Filebeat and Logstash can be used to send logs from a file-based data source to a supported output destination
  • Filebeat is a lightweight option, ideal for environments with limited resources and basic log parsing needs. Conversely, Logstash is tailored for scenarios that demand advanced log processing
  • both FB and LS can be used in tandem when building a logging pipeline with the ELK Stack because both have a different function
  • .


  • visualisation and reporting tool
  • used with Elasticsearch to:
    • visualize the data
    • build interactive dashboards
  • .

No comments: