Friday 16 February 2024

Introduction to ELK Stack





What is ELK stack?

What is it used for?
  • aggregates logs from all systems and applications
  • logs analytics
  • visualizations for application and infrastructure monitoring, faster troubleshooting, security analytics etc.

Elasticsearch


  • distributed search and analytics engine built on Apache Lucene
  • licensed, not open source
    • OpenSearch is open-sourced alternative (supported by AWS)
    • FluentD is another open-source data collection alternative
  • data in the form of JSON documents is sent to Elasticsearch using:
    • API
    • ingestion tools
      • Logstash
      • Amazon Kinesis Data Firehose
  • the original document automatically stored and a searchable reference is added to the document in the cluster’s index
  • Elasticsearch REST-based API is used to manipulate with documents:
    • send
    • search
    • retrieve 
  • uses schema-free JSON documents
  • distributed system
    • enables it to process large volumes of data in parallel, quickly finding the best matches for your queries
  • operations such as reading or writing data usually take less than a second to complete => Elasticsearch can be used for near real-time use cases such as application monitoring and anomaly detection
  • has support for various languages: Java, Python, PHP, JavaScript, Node.js, Ruby etc...
  • .


Logstash


  • log shipper
  • helps easily transform source data and load it into Elasticsearch cluster
  • .

Filebeat


  • https://www.elastic.co/beats/filebeat
  • log shipper
  • both Filebeat and Logstash can be used to send logs from a file-based data source to a supported output destination
  • Filebeat is a lightweight option, ideal for environments with limited resources and basic log parsing needs. Conversely, Logstash is tailored for scenarios that demand advanced log processing
  • both FB and LS can be used in tandem when building a logging pipeline with the ELK Stack because both have a different function
  • .

Kibana


  • visualisation and reporting tool
  • used with Elasticsearch to:
    • visualize the data
    • build interactive dashboards
  • .

Friday 2 February 2024

Installing GraphViz on MacOS

I wanted to test Command: graph | Terraform | HashiCorp Developer by cd-ing to an arbitrary Terraform module and executing:

% terraform graph -type=plan | dot -Tpng >graph.png

But this issued an error:

zsh: command not found: dot

Solution:

% brew install graphviz  
...
==> Installing graphviz
==> Pouring graphviz--9.0.0.arm64_ventura.bottle.tar.gz
🍺  /opt/homebrew/Cellar/graphviz/9.0.0: 287 files, 7.1MB
==> Running `brew cleanup graphviz`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).


To verify installation:

% dot --version 
dot - graphviz version 9.0.0 (20230911.1827)