Introduction to ELK Stack

What is ELK stack?

• Elasticsearch [https://www.elastic.co/elasticsearch]
• Logstash [https://www.elastic.co/logstash]
• Kibana [https://www.elastic.co/kibana]

What is it used for?

• aggregates logs from all systems and applications
• logs analytics
• visualizations for application and infrastructure monitoring, faster troubleshooting, security analytics etc.

Elasticsearch

• distributed search and analytics engine built on Apache Lucene
• licensed, not open source
• OpenSearch is open-sourced alternative (supported by AWS)
• FluentD is another open-source data collection alternative
• data in the form of JSON documents is sent to Elasticsearch using:
• API
• ingestion tools
• Logstash
• Amazon Kinesis Data Firehose
• the original document automatically stored and a searchable reference is added to the document in the cluster’s index
• Elasticsearch REST-based API is used to manipulate with documents:
• send
• search
• retrieve 
• uses schema-free JSON documents
• distributed system
• enables it to process large volumes of data in parallel, quickly finding the best matches for your queries
• operations such as reading or writing data usually take less than a second to complete => Elasticsearch can be used for near real-time use cases such as application monitoring and anomaly detection
• has support for various languages: Java, Python, PHP, JavaScript, Node.js, Ruby etc...
• .

Logstash

• log shipper
• helps easily transform source data and load it into Elasticsearch cluster
• .

Filebeat

• https://www.elastic.co/beats/filebeat
• log shipper
• both Filebeat and Logstash can be used to send logs from a file-based data source to a supported output destination
• Filebeat is a lightweight option, ideal for environments with limited resources and basic log parsing needs. Conversely, Logstash is tailored for scenarios that demand advanced log processing
• both FB and LS can be used in tandem when building a logging pipeline with the ELK Stack because both have a different function
.

Kibana

• visualisation and reporting tool
• used with Elasticsearch to:
• visualize the data
• build interactive dashboards
• .