Introduction to Grafana Loki

Grafana Loki:

• Log aggregation system. Like Prometheus, but for logs
• Repo: https://github.com/grafana/loki

These are the notes from Loki Helm chart: 

***********************************************************************

  Welcome to Grafana Loki

  Chart version: 6.31.0

  Chart Name: loki

  Loki version: 3.5.0

***********************************************************************

Tip:

Watch the deployment status using the command: kubectl get pods -w --namespace grafana-loki

If pods are taking too long to schedule make sure pod affinity can be fulfilled in the current cluster.

***********************************************************************

Installed components:

***********************************************************************

* gateway

* read

* write

* backend

***********************************************************************

Sending logs to Loki

***********************************************************************

Loki has been configured with a gateway (nginx) to support reads and writes from a single component.

You can send logs from inside the cluster using the cluster DNS:

http://loki-gateway.grafana-loki.svc.cluster.local/loki/api/v1/push

You can test to send data from outside the cluster by port-forwarding the gateway to your local machine:

  kubectl port-forward --namespace grafana-loki svc/loki-gateway 3100:80 &

And then using http://127.0.0.1:3100/loki/api/v1/push URL as shown below:

curl \

-H "Content-Type: application/json" \

-XPOST \

-s "http://127.0.0.1:3100/loki/api/v1/push"  \

--data-raw "{\"streams\": [{\"stream\": {\"job\": \"test\"}, \"values\": [[\"$(date +%s)000000000\", \"fizzbuzz\"]]}]}" \

-H X-Scope-OrgId:foo

Then verify that Loki did receive the data using the following command:

curl "http://127.0.0.1:3100/loki/api/v1/query_range" \

--data-urlencode 'query={job="test"}' \

-H X-Scope-OrgId:foo | jq .data.result

***********************************************************************

Connecting Grafana to Loki

***********************************************************************

If Grafana operates within the cluster, you'll set up a new Loki datasource by utilizing the following URL:

http://loki-gateway.grafana-loki.svc.cluster.local/

***********************************************************************

Multi-tenancy

***********************************************************************

Loki is configured with auth enabled (multi-tenancy) and expects tenant headers (`X-Scope-OrgID`) to be set for all API calls.

You must configure Grafana's Loki datasource using the `HTTP Headers` section with the `X-Scope-OrgID` to target a specific tenant.

For each tenant, you can create a different datasource.

The agent of your choice must also be configured to propagate this header.

For example, when using Promtail you can use the `tenant` stage. https://grafana.com/docs/loki/latest/send-data/promtail/stages/tenant/

When not provided with the `X-Scope-OrgID` while auth is enabled, Loki will reject reads and writes with a 404 status code `no org id`.

You can also use a reverse proxy, to automatically add the `X-Scope-OrgID` header as suggested by https://grafana.com/docs/loki/latest/operations/authentication/

For more information, read our documentation about multi-tenancy: https://grafana.com/docs/loki/latest/operations/multi-tenancy/

> When using curl you can pass `X-Scope-OrgId` header using `-H X-Scope-OrgId:foo` option, where foo can be replaced with the tenant of your choice.

EOT -> (known after apply)

 

---

Grafana Observability Stack

 

Grafana uses these components together as an observability stack, but each has a clear role:

Loki – log database. It stores and indexes logs (especially from Kubernetes) in a cost‑efficient, label‑based way, similar to Prometheus but for logs.

Tempo – distributed tracing backend. It stores distributed traces (spans) from OpenTelemetry, Jaeger, Zipkin, etc., so you can see call flows across microservices and where latency comes from.

Mimir – Prometheus‑compatible metrics backend. It is a horizontally scalable, long‑term storage and query engine for Prometheus‑style metrics (time series).

Alloy – telemetry pipeline (collector). It is Grafana’s distribution of the OpenTelemetry Collector / Prometheus agent / Promtail ideas, used to collect, process, and forward metrics, logs, traces, profiles into Loki/Tempo/Mimir (or other backends).

How Grafana UI relates to them

Grafana UI itself is “just” the visualization and alerting layer:

• It connects to Loki, Tempo, Mimir (and many others) as data sources.
• For each backend you configure:
• A Loki data source for logs.
• A Tempo data source for traces.
• A Prometheus/Mimir data source for metrics (Mimir exposes a Prometheus‑compatible API).
• Grafana then lets you:
• Build dashboards and alerts from Mimir metrics.
• Explore logs from Loki.
• Explore traces from Tempo and cross‑link them with logs/metrics (e.g., click from a log line to a trace, or from a metrics graph into logs/traces).

A useful mental model: Loki/Tempo/Mimir are databases, Alloy is the collector/router, and Grafana is the UI on top.

Are they deployed in the same Kubernetes cluster?

Common patterns:

• Very common: deploy Loki, Tempo, Mimir, Alloy, and Grafana in the same Kubernetes cluster as your apps. This is the typical “in‑cluster LGTM” setup; all telemetry stays inside the cluster and traffic is simple.
• Also common: run them in a separate observability cluster (or use Grafana Cloud backends), while Alloy/agents run in each workload cluster and ship data over the network. This improves isolation and makes it easier to share one observability stack across many clusters.
• In smaller setups or dev environments, everything (apps + LGTM + Grafana) often lives in one cluster; in larger/regulated setups, people tend to separate “workload clusters” and an “observability cluster”.

So: they don’t have to be on the same cluster, but it’s perfectly normal (and often simplest) to run Grafana + Loki + Tempo + Mimir + Alloy together in a single Kubernetes cluster and point your apps’ telemetry to Alloy.

Why not using elasticsearch instead of loki, tempo and mimir?

Elasticsearch can replace part of what Loki, Tempo, and Mimir do, but not all of it, and usually with higher cost/complexity for cloud‑native observability.

1. Scope: logs vs full observability

Elasticsearch is a general search and analytics engine that’s great at full‑text search, aggregations, and analytics over documents (including logs).

The LGTM stack is explicitly split by signal:

• Loki → logs
• Tempo → traces
• Mimir → metrics

Each is optimized only for its signal type and integrates tightly with Grafana and modern telemetry standards.

You could plausibly replace Loki with Elasticsearch for logs, but Elasticsearch does not natively replace Tempo (distributed tracing backend) or Mimir (Prometheus‑compatible metrics backend).

2. Logs: Loki vs Elasticsearch

Elasticsearch strengths:

• Very powerful full‑text search, fuzzy matching, relevance scoring, complex aggregations.
• Good when you need deep forensic search and advanced analytics on log text.

Loki strengths:

• Stores logs as compressed chunks plus a small label index, so storage and compute are much cheaper than Elasticsearch for typical Kubernetes logs.
• Very tight integration with Grafana and the rest of LGTM, and simple, label‑based querying.

Trade‑off: Elasticsearch gives richer search at a high infra + ops cost, Loki gives “good enough” search for operational troubleshooting with much lower cost and operational burden.

3. Traces and metrics: Tempo & Mimir vs “just ES”

Tempo:

• Implements distributed tracing concepts (spans, traces, service graphs) and OpenTelemetry/Jaeger/Zipkin protocols; the data model and APIs are specialized for traces.
• Elasticsearch can store trace‑like JSON documents, but you’d have to build/maintain all the trace stitching, UI navigation, and integrations yourself.

Mimir:

• Is a horizontally scalable, Prometheus‑compatible time‑series database, with native remote‑write/read and PromQL semantics.
• Elasticsearch can store time‑stamped metrics, but you lose Prometheus compatibility, PromQL semantics, and the whole ecosystem that expects a Prometheus‑style API.

So using only Elasticsearch means you’re giving up the standard metrics and tracing ecosystems and rebuilding a lot of tooling on top of a generic search engine.

​

4. Cost, complexity, and operational burden

Elasticsearch clusters generally need:

• More RAM/CPU per node, careful shard and index management, and capacity planning.
• Storage overhead from full‑text indexes (often 1.5–3× raw log size plus replicas).

​

Loki/Tempo/Mimir:

• Are designed for object storage, compression, and label‑only indexing, which dramatically lowers storage and compute requirements for logs and metrics.
• Have simpler, well‑documented reference architectures specifically for observability.

For a modern Kubernetes‑centric environment, that usually makes LGTM cheaper and easier to run than a single big Elasticsearch cluster for everything.

5. When Elasticsearch still makes sense

You might still choose Elasticsearch (often with Kibana/APM) if:

• You already have a strong ELK stack and team expertise.
• Your primary need is deep, flexible text search and analytics over logs, with less emphasis on Prometheus/OTel ecosystems.
• You want Elasticsearch’s ML/anomaly‑detection features and are willing to pay the operational cost.

But if your goal is a Grafana‑centric, standards‑based (Prometheus + OpenTelemetry) observability platform, LGTM (Loki+Tempo+Mimir, plus Alloy as collector) is a better fit than trying to push everything into Elasticsearch.

---

Here document (heredoc)

Here document (heredoc) redirects a multiline string literal to the preceding command while preserving line breaks. Unix syntax for it is:

[command] <<DELIMITER

    First line.

    Second line.

    Third line.

    Fourth line.

DELIMITER

<< is Redirection Operator

- is optional Tab Suppression

DELIMITER - an arbitrary string, Delimiter Token; must be the same at the beginning and at the end

Appending a minus sign to the redirection operator <<- causes all leading tab characters to be ignored. This allows you to use indentation when writing heredocs in shell scripts. We can then indent both the here-doc and the delimiter with tabs (not spaces!):

#! /bin/bash

cat <<-EOF

    indented

    EOF

echo Done

---

References:

Here document - Wikipedia

How to fix pods in Not Ready state?

kubectl get pods might show that some of the pods have 0/N value in READY column.

What is the meaning of READY column value?

In the context of kubectl get pods, the READY column shows the number of containers in the pod that have passed their health checks and are ready to serve traffic.

The anatomy of R/T:

• R (Left side): This is the number of containers currently Ready. A 0 means the application inside the container is not responding to its "Readiness Probe" or has not finished starting up.
• / (Separator): Separates ready containers from the total.
• T (Right side): This is the Total number of user containers defined in that pod.

When we have a pod in 0/1 state with a status of Running, it means the container has started, but Kubernetes does not consider it "healthy" enough to handle requests. Our pods are technically "alive" (Running) but "unusable" (Not Ready).

Common Reasons for 0/1 Running:

• Failed Readiness Probe: The application is running, but the health check URL (e.g., /ready) is returning an error or timing out.
• Slow Startup: The application takes a long time to initialize, and the "Initial Delay" isn't long enough.
• Dependency Issues: The pod is waiting for a database, a config file, or another service that isn't available.
• CrashLoopBackOff (Transitions): Sometimes pods flicker between 0/1 Running and 0/1 CrashLoopBackOff as they try to start and immediately fail.

How to find out exactly what's wrong

To diagnose why our my-app pods are stuck, run these two commands:

(1) Check the Events (Why it's not ready):

kubectl describe pod my-app-0 -n my-app-namespace

Look at the "Events" section at the bottom for messages like "Readiness probe failed."

Output example:

Events:

  Type     Reason     Age                   From     Message

  ----     ------     ----                  ----     -------

  Warning  Unhealthy  47s (x81205 over 8d)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 503

(2) Check the Logs (What the app is saying):

kubectl logs my-app-0 -n my-app-namespace

Look for "Error," "Exception," or "Connection Refused" messages.

---

How to use terraform-docs automatically generate Terraform code documentation

 

terraform-docs is a tool used to automatically generate Terraform code documentation.

To install it on Mac:

% brew install terraform-docs 

To verify installation:

% terraform-docs --version                                        

terraform-docs version v0.21.0 darwin/arm64

To generate a documentation for a module in the current directory and append it to the README file (which is in the same directory):

% terraform-docs markdown table --output-file README.md --output-mode inject ./

How to install Terraform on Mac

First add Hashicorp's package repository:

% brew tap hashicorp/tap

Then install the Terraform:

% brew install hashicorp/tap/terraform

If Terraform was already installed, the command above will update it.

To verify installation, we can check its version:

% terraform --version                                                                                    

Terraform v1.14.5

on darwin_arm64

Amazon EKS Autoscaling with Karpenter

Kubernetes autoscaling is a function that scales resources in and out depending on the current workload. AWS supports two autoscaling implementations:

• Cluster Autoscaler
• autoscaler/cluster-autoscaler/cloudprovider/aws/README.md at master · kubernetes/autoscaler
• automatically adjusts the number of nodes in the cluster when pods fail or are rescheduled onto other nodes
• uses Auto Scaling groups
• Karpenter 
• Karpenter
• flexible, high-performance Kubernetes cluster autoscaler
• helps improve application availability and cluster efficiency
• launches right-sized compute resources (for example, Amazon EC2 instances) in response to changing application load in under a minute
• can provision just-in-time compute resources that precisely meet the requirements of your workload
• automatically provisions new compute resources based on the specific requirements of cluster workloads. These include compute, storage, acceleration, and scheduling requirements. 
• creates Kubernetes nodes directly from EC2 instances
• improves the efficiency and cost of running workloads on the cluster
• open-source

Pod Scheduler

• Kubernetes cluster component responsible for determining which node Pods get assigned to
• default Pod scheduler for Kubernetes is kube-scheduler
• logs the reasons Pods can't be scheduled

Unschedulable Pods

From How to troubleshoot unschedulable Pods in Kubernetes:

A Pod is unschedulable when it's been put into Kubernetes' scheduling queue, but can't be deployed to a node. This can be for a number of reasons, including:

• The cluster not having enough CPU or RAM available to meet the Pod's requirements.
• Pod affinity or anti-affinity rules preventing it from being deployed to available nodes.
• Nodes being cordoned due to updates or restarts.
• The Pod requiring a persistent volume that's unavailable, or bound to an unavailable node.

How to detect unschedulable Pods?

Pods waiting to be scheduled are held in the "Pending" status, but if the Pod can't be scheduled, it will remain in this state. However, Pods that are being deployed normally are also marked as "Pending." The difference comes down to how long a Pod remains in "Pending." 

How to  fix unschedulable Pods? 

There is no single solution for unschedulable Pods as they have many different causes. However, there are a few things you can try depending on the cause. 

• Enable cluster autoscaling
• If you're using a managed Kubernetes service like Amazon EKS or Google Kubernetes Engine (GKE), you can very easily take advantage of autoscaling to increase and decrease cluster capacity on-demand. With autoscaling enabled, Kubernetes' Cluster Autoscaler will trigger your provider to add nodes when needed. As long as you've configured your cluster node pool and it hasn't reached its max node limit, your provider will automatically provision a new node and add it to the pool, making it available to the cluster and to your Pods.
• Increase your node capacity
• Check your Pod requests
• Check your affinity and anti-affinity rules 

 

In this article we'll show how to enable cluster autoscaling with Karpenter.

How does the regular Kubernetes Autoscaler work in AWS?

When we create a regular Kubernetes cluster in AWS, each node group is managed by the AWS Auto-scaling group [Auto Scaling groups - Amazon EC2 Auto Scaling]. Cluster native autoscaler adjusts the desired size based on the load in the cluster to fit all unscheduled pods.

HorizontalPodAutoscaler (HPA) [Horizontal Pod Autoscaling | Kubernetes] is built into Kubernetes and it uses metrics like CPU usage, memory usage or custom metrics we can write to decide when to spin up or down additional pods in the node of the cluster. If our app is receiving more traffic, HPA will kick in and provision additional pods. 

VerticalPodAutoscaler (VPA) can also be installed in cluster where it manages the resource (like CPU and memory) allocation to pods that are already running.

What about when there's not enough capacity to schedule any more pods in the node? That's when we'll need an additional node. So we have a pod that needs to be scheduled but we don't know where to put it. We could call AWS API, spin up an additional EC2 node, get added it to our cluster or if we're using managed groups we can use Managed Node Group API, bump up the desired size but easier approach is to use cluster auto-scaler. There is a mature open-source solution called Cluster Auto-Scaler (CAS).

CAS was built to handle hundreds of different comninations of nodes types, zones, purchase options available in AWS. CAS works directly with managed node groups or self-managed managed nodes and auto-scaling groups which are AWS constructs to help us manage nodes. 

What are the issues with the regular Kubernetes Autoscaler?

Let's say CAS is installed on node, in cluster and manages one managed node group (MNG). It's filling up and we have an additional pod that needs to be provisioned so CAS tells MNG to bump up the number of nodes so it spins up another one so pod can now be scheduled. But this is not ideal. We have a single pod in a node, we don't need such a big node. 

This can be solved by creating a different MNG with a smaller instance type and now CAS recognizes that instance and provisions pod on a more appropriately-sized node.

Unfortunately, we might end up with many MNGs, based on requirements which might be a challenge to manage especially when looking best practices in terms of cost efficiency and high availability. 

How does Karpenter work?

Karpenter works differently, It doesn't use MNG or ASGs and manages each node directly. Let's say we have different pods, of different sizes. Let's say that HPA says that we need more of the smaller pods. Karpenter will intelligently pick the right instance type for that workload. If we need to spin up a larger pod it will again pick the right instance type. 

Karpenter picks exactly the right type of node for our workload. 

If we're using spot instances and spot capacity is not available, Karpenter does retries more quickly. Karpenter offers, faster, dynamic, more intelligent compute, using best practices without operational overhead of managing nodes ourselves. 

How to control how Karpenter operates?

There are many dimensions here. We can set constraints on Karpenter to limit the instances type, we can set up taints to isolate workloads to specific types of nodes. Different teams can have isolated access to different pods, one team can access billing pods, another GPU-based instances. 

Workload Consolidation feature: Pods are consolidated into fewer nodes.. let's say we have 3 nodes, two at 70% and one at 20% utilization. Karpenter detects this and will move pods from underutilized node to those two and shut down this now empty node (instances are terminated). This leads to lower costs.

Karpenter is making it easier to use spot and graviton instances which can also lead to lower costs. 

A feature to keep our nodes up to date. ttlSecondsUntilExpired parameter tells Karpenter to terminate nodes after a set amount of time. These nodes will automatically be replaced with new nodes, running the latest AMIs.

Karpenter:

1) lower costs

2) higher application availability 

3) lower operation overhead

Karpenter needs permissions to create EC2 instances in AWS. 

If we use a self-hosted (on bare metal boxes or EC2 instances), self-managed (you have full control over all aspects of Kubernetes) Kubernetes cluster, for example by using kOps (see also Is k8s Kops preferable than eks? : r/kubernetes), we can add additional IAM policies to the existing IAM role attached to Kubernetes nodes. 

If using EKS, the best way to grant access to internal service is with IAM roles for service accounts (IRSA).

How to configure Karpenter?

We can configure specific Karpenter NodePools or Provisioners.

How to know if node was provisioned by Karpenter?

Karpenter applies labels on nodes it provisions so let's check labels:

% kubectl get nodes --show-labels

If labels like karpenter.sh/nodepool or karpenter.sh/provisioner-name exist, Karpenter launched the node.

References:

Autoscaling - Amazon EKS

Karpenter for Kubernetes | Karpenter vs Cluster Autoscaler - YouTube

Cluster-Autoscaler - EKS Best Practices Guides

Karpenter Vs Cluster Autoscaler: The Essential Guide