Introduction to KubePug (Kubernetes tool)

What is KubePug?

KubePug (Kubernetes PreUpGrade Checker) is an open-source kubectl plugin and CLI tool designed to identify deprecated or deleted APIs in your Kubernetes cluster or manifest files before you perform an upgrade. 

Key Features

• Deprecation Detection: Scans your live cluster or static YAML manifests to find resources using APIs that are slated for removal in future Kubernetes versions.
• Replacement Guidance: Not only flags outdated APIs but also suggests the recommended replacement API and specifies the exact version where the deprecation or deletion occurs.
• Version Targeting: Allows you to specify a target Kubernetes version (e.g., v1.31) to validate your current resources against that specific future release.
• Flexible Data Source: It automatically downloads a frequently updated API definition file (every 30 minutes) to stay current with the latest Kubernetes releases. 

Why Use It?

As Kubernetes evolves, APIs are moved from alpha/beta to stable (GA), and older versions are eventually removed. If you upgrade your cluster without updating your manifests, those resources will fail to deploy or operate. KubePug provides a "pre-flight" check to prevent these breaking changes from reaching production.

Installation & Usage

You can install KubePug via Krew (where it is listed under the name deprecations) or as a standalone binary. 

Method                  Command

---------                    -------------

Install via Krew kubectl krew install deprecations

Scan Live Cluster kubectl deprecations --k8s-version=v1.30

Scan Manifest File kubepug --input-file=./my-manifest.yaml

Installation via Krew

% kubectl krew install deprecations

Updated the local copy of plugin index.

Installing plugin: deprecations

Installed plugin: deprecations

\

 | Use this plugin:

 | kubectl deprecations

 | Documentation:

 | https://github.com/rikatz/kubepug

 | Caveats:

 | \

 |  | * By default, deprecations finds deprecated object relative to the current kubernetes

 |  | master branch. To target a different kubernetes release, use the --k8s-version

 |  | argument.

 |  | 

 |  | * Deprecations needs permission to GET all objects in the Cluster

 | /

/

WARNING: You installed plugin "deprecations" from the krew-index plugin repository.

   These plugins are not audited for security by the Krew maintainers.

   Run them at your own risk.

Execution

Once installed, the plugin is invoked using kubectl deprecations.

Scan Current Cluster

Check your live cluster for deprecated APIs against a specific target Kubernetes version:

% kubectl deprecations --k8s-version=v1.33

                    

Error: failed to get apiservices: apiservices.apiregistration.k8s.io is forbidden: User "sso:user" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope

time="2026-02-26T14:20:19Z" level=error msg="An error has occurred: failed to get apiservices: apiservices.apiregistration.k8s.io is forbidden: User \"sso:user\" cannot list resource \"apiservices\" in API group \"apiregistration.k8s.io\" at the cluster scope"

KubePug requires running user to have "list" permissions on resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope as otherwise the above error will appear.

If required permissions are in place:

% kubectl deprecations --k8s-version=v1.33

No deprecated or deleted APIs found

Kubepug validates the APIs using Kubernetes markers. To know what are the deprecated and deleted APIS it checks, please go to https://kubepug.xyz/status/

Scan Local Manifest Files

Validate static YAML files before applying them to a cluster:

kubectl deprecations --input-file=./my-manifests/

View Results in Different Formats

Output the findings in json or yaml for automated processing:

kubectl deprecations --format=json

Check for Help and Flags

See all available configuration options, such as using a custom database file or setting error codes:

kubectl deprecations --help

Key Parameters

--k8s-version: The Kubernetes release you intend to upgrade to (defaults to the latest stable).

--error-on-deprecated: Forces the command to exit with an error code if deprecated APIs are found, which is useful for CI/CD pipelines. 

---

Introduction to Krew (Kubernetes tool)

Krew is the official plugin manager for the kubectl command-line tool. Much like apt for Debian or Homebrew for macOS, it allows users to easily discover, install, and manage custom extensions that add new subcommands to Kubernetes. 

Core Functionality

• Discovery: Users can search a community-curated index of over 200 plugins designed for tasks like security auditing, resource visualization, and cluster management.
• Lifecycle Management: It automates the process of installing, updating, and removing plugins across different operating systems (Linux, macOS, and Windows).
• Unified Interface: Once a plugin is installed via Krew, it is invoked directly through kubectl (e.g., kubectl <plugin-name>). 

Installation

MacOS example output:

% (

  set -x; cd "$(mktemp -d)" &&

  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&

  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&

  KREW="krew-${OS}_${ARCH}" &&

  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&

  tar zxvf "${KREW}.tar.gz" &&

  ./"${KREW}" install krew

)

+-zsh:142> mktemp -d

+-zsh:142> cd /var/folders/8j/60m6_18j359_39ls0sr9ccvm0000gp/T/tmp.LKDSyQcO10

+-zsh:143> OS=+-zsh:143> uname

+-zsh:143> OS=+-zsh:143> tr '[:upper:]' '[:lower:]'

+-zsh:143> OS=darwin 

+-zsh:144> ARCH=+-zsh:144> uname -m

+-zsh:144> ARCH=+-zsh:144> sed -e s/x86_64/amd64/ -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/'

+-zsh:144> ARCH=arm64 

+-zsh:145> KREW=krew-darwin_arm64 

+-zsh:146> curl -fsSLO https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-darwin_arm64.tar.gz

+-zsh:147> tar zxvf krew-darwin_arm64.tar.gz

x ./LICENSE

x ./krew-darwin_arm64

+-zsh:148> ./krew-darwin_arm64 install krew

Adding "default" plugin index from https://github.com/kubernetes-sigs/krew-index.git.

Updated the local copy of plugin index.

Installing plugin: krew

Installed plugin: krew

\

 | Use this plugin:

 | kubectl krew

 | Documentation:

 | https://krew.sigs.k8s.io/

 | Caveats:

 | \

 |  | krew is now installed! To start using kubectl plugins, you need to add

 |  | krew's installation directory to your PATH:

 |  | 

 |  |   * macOS/Linux:

 |  |     - Add the following to your ~/.bashrc or ~/.zshrc:

 |  |         export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

 |  |     - Restart your shell.

 |  | 

 |  |   * Windows: Add %USERPROFILE%\.krew\bin to your PATH environment variable

 |  | 

 |  | To list krew commands and to get help, run:

 |  |   $ kubectl krew

 |  | For a full list of available plugins, run:

 |  |   $ kubectl krew search

 |  | 

 |  | You can find documentation at

 |  |   https://krew.sigs.k8s.io/docs/user-guide/quickstart/.

 | /

/

Add Krew path to PATHs:

% vi ~/.zshrc 

% cat ~/.zshrc   

...

export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH" <-- added manually

Reload .zshrc configuration file within the currently running Zsh terminal (or just restart the terminal):

% source ~/.zshrc

Installation verification:

% kubectl krew

krew is the kubectl plugin manager.

You can invoke krew through kubectl: "kubectl krew [command]..."

Usage:

  kubectl krew [command]

Available Commands:

  help        Help about any command

  index       Manage custom plugin indexes

  info        Show information about an available plugin

  install     Install kubectl plugins

  list        List installed kubectl plugins

  search      Discover kubectl plugins

  uninstall   Uninstall plugins

  update      Update the local copy of the plugin index

  upgrade     Upgrade installed plugins to newer versions

  version     Show krew version and diagnostics

Flags:

  -h, --help      help for krew

  -v, --v Level   number for the log level verbosity

Use "kubectl krew [command] --help" for more information about a command.

Common Commands

To use Krew, you must first install it as a kubectl plugin itself. Key commands include: 

kubectl krew update: Updates the local list of available plugins.

kubectl krew search: Finds plugins in the official Krew index.

kubectl krew install <plugin>: Installs a specific plugin.

kubectl krew list: Displays all plugins currently installed through Krew.

kubectl krew upgrade: Updates all installed plugins to their latest versions. 

Example:

% kubectl krew install deprecations

Updated the local copy of plugin index.

Installing plugin: deprecations

Installed plugin: deprecations

\

 | Use this plugin:

 | kubectl deprecations

 | Documentation:

 | https://github.com/rikatz/kubepug

 | Caveats:

 | \

 |  | * By default, deprecations finds deprecated object relative to the current kubernetes

 |  | master branch. To target a different kubernetes release, use the --k8s-version

 |  | argument.

 |  | 

 |  | * Deprecations needs permission to GET all objects in the Cluster

 | /

/

WARNING: You installed plugin "deprecations" from the krew-index plugin repository.

   These plugins are not audited for security by the Krew maintainers.

   Run them at your own risk.

Popular Plugins Managed by Krew 

• ctx / ns: Rapidly switch between Kubernetes contexts and namespaces.
• tree: Visualizes the hierarchy of Kubernetes resources in a tree view.
• access-matrix: Displays an RBAC (Role-Based Access Control) matrix for server resources.
• get-all: Lists all resources in a namespace, including those often missed by kubectl get all. 

Note on Security (!)

Plugins in the Krew index are community-contributed and are not audited for security by the Kubernetes maintainers; you should only install plugins from sources you trust. 

---

Introduction to Pluto (Kubernetes tool)

 

Pluto is:

• CLI tool that helps users find deprecated Kubernetes API versions in your code repositories and Helm releases. 
• It's especially useful when upgrading Kubernetes clusters, as it identifies resources that need updating before the upgrade.
• It works against:
• Live clusters
• Helm charts
• Raw YAML
• Pluto will show which APIs are deprecated or removed, what version they were deprecated in, and what the replacement API should be.

Installation on Mac (https://pluto.docs.fairwinds.com/installation/#homebrew-tap):

% brew install FairwindsOps/tap/pluto

Let's see its CLI arguments:

% pluto                    

You must specify a sub-command.

A tool to detect Kubernetes apiVersions

Usage:

  pluto [flags]

  pluto [command]

Available Commands:

  completion            Generate the autocompletion script for the specified shell

  detect                Checks a single file or stdin for deprecated apiVersions.

  detect-all-in-cluster run all in-cluster detections

  detect-api-resources  detect-api-resources

  detect-files          detect-files

  detect-helm           detect-helm

  help                  Help about any command

  list-versions         Outputs a JSON object of the versions that Pluto knows about.

  version               Prints the current version of the tool.

Flags:

  -f, --additional-versions string        Additional deprecated versions file to add to the list. Cannot contain any existing versions

      --columns strings                   A list of columns to print. Mandatory when using --output custom, optional with --output markdown

      --components strings                A list of components to run checks for. If nil, will check for all found in versions.

  -h, --help                              help for pluto

      --ignore-deprecations               Ignore the default behavior to exit 2 if deprecated apiVersions are found. (Only show removed APIs, not just deprecated ones)

      --ignore-removals                   Ignore the default behavior to exit 3 if removed apiVersions are found. (Only show deprecated APIs, not removed ones)

      --ignore-unavailable-replacements   Ignore the default behavior to exit 4 if deprecated but unavailable apiVersions are found.

  -H, --no-headers                        When using the default or custom-column output format, don't print headers (default print headers).

  -r, --only-show-removed                 Only display the apiVersions that have been removed in the target version.

  -o, --output string                     The output format to use. (normal|wide|custom|json|yaml|markdown|csv) (default "normal")

  -t, --target-versions stringToString    A map of targetVersions to use. This flag supersedes all defaults in version files. (default [])

  -v, --v Level                           number for the log level verbosity

Use "pluto [command] --help" for more information about a command.

detect-files

To scan and detect deprecated APIs in manifest files in a directory:

% pluto detect-files -d /path/to/your/manifests

detect-files is for checking YAML files in our repositories/filesystem before deploying them - that's separate from detect-helm and detect-all-in-cluster cluster commands.

To target particular k8s version:

% pluto detect-files -d . --target-versions k8s=v1.33.0

If we use Terraform to deploy Helm charts, we might want to keep chart values in separate files (.yaml or .yaml.tpl) as otherwise we won't be able to use Pluto directly (we'd need to extract values into files first). For more details, see Where to keep Helm chart values in Terraform projects | My Public Notepad. 

detect-helm

To check Helm releases in the cluster (already deployed):

% pluto detect-helm -owide

To target particular k8s version:

% pluto detect-helm -owide --target-versions k8s=v1.33.0   

There were no resources found with known deprecated apiVersions.

detect-helm specifically checks Helm release metadata stored in our cluster (in secrets or configmaps) after Helm chart have been deployed. It looks at the manifests that Helm used to install releases, which might contain deprecated APIs even if they haven't been applied yet or are stored in Helm's history.

detect-all-in-cluster

To check all resources in the cluster:

% pluto detect-all-in-cluster -o wide

I0226 12:01:02.279788   47100 warnings.go:110] "Warning: v1 ComponentStatus is deprecated in v1.19+"

There were no resources found with known deprecated apiVersions.

detect-all-in-cluster scans all live resources currently running in our cluster by querying the Kubernetes API directly. It checks deployments, services, pods, etc. that are actively deployed.

detect-all-in-cluster does NOT include detect-helm or detect-files. Here's why they're separate:

• detect-all-in-cluster sees the current state of resources
• detect-helm sees Helm's stored templates and history, which may include:
• Templated manifests that haven't been rendered yet
• Old release revisions
• Chart templates with deprecated APIs
• Run both to get complete coverage!

Target a specific Kubernetes version:

% pluto detect-all-in-cluster --target-versions k8s=v1.33.0

I0226 12:02:26.551401   47113 warnings.go:110] "Warning: v1 ComponentStatus is deprecated in v1.19+"

There were no resources found with known deprecated apiVersions.

The warning message:

Warning: v1 ComponentStatus is deprecated in v1.19+

This is just Pluto itself triggering a Kubernetes API warning while scanning - it's not something wrong with our cluster resources.

The main result:

There were no resources found with known deprecated apiVersions.

This means all our cluster resources are using API versions that are still valid in Kubernetes v1.33.0 (our target version). This means:

• Our cluster resources are already compatible with k8s v1.33.0
• No manifests need updating before upgrading
• No deprecated APIs that would be removed in v1.33.0

To be thorough before a k8s upgrade, we need to run all three commands:

• detect-files
• detect-helm
• detect-all-in-cluster

---

Where to keep Helm chart values in Terraform projects

If we use Terraform to deploy Helm charts, we might be using one of these strategies to keep chart values:

1. Values are in inline YAML string
2. Values in separate .yaml file
3. Values in separate YAML Template files (.yaml.tpl)
4. Use Helm's set for Dynamic Values
5. Multiple Values Files

(1) Values in inline YAML string

This is not ideal as problems with Inline YAML in Terraform include:

• No syntax highlighting or validation - Easy to break YAML formatting
• Hard to review in diffs - Changes are messy in PRs
• Can't use standard tooling - No yamllint, Pluto, or other YAML tools
• Mixing concerns - Infrastructure code mixed with application config
• Escaping nightmares - Terraform string interpolation conflicts with Helm templating

Example:

resource "helm_release" "app" {

  values = [<<-EOT

    replicaCount: ${var.replicas}

    image:

      repository: myapp

      tag: ${var.tag}

    service:

      type: LoadBalancer

  EOT

  ]

}

(2) Separate Values Files 

Keep values in YAML files, reference them in Terraform.

This is a better approach because:

• Clean separation
• Easy to validate with standard tools
• Better diffs
• Can use Pluto directly: pluto detect-files -d .

Example:

main.tf:

resource "helm_release" "my_app" {

  name       = "my-app"

  chart      = "my-chart"

  repository = "https://charts.example.com"

  

  values = [

    file("${path.module}/helm-values.yaml")

  ]

}

(3) Templated Values Files

Use Terraform's templatefile() to inject dynamic values:

helm-values.yaml.tpl:

replicaCount: ${replica_count}

image:

  repository: ${image_repo}

  tag: ${image_tag}

ingress:

  enabled: ${enable_ingress}

  host: ${hostname}

main.tf:

resource "helm_release" "my_app" {

  name  = "my-app"

  chart = "my-chart"

  

  values = [

    templatefile("${path.module}/helm-values.yaml.tpl", {

      replica_count  = var.replica_count

      image_repo     = var.image_repository

      image_tag      = var.image_tag

      enable_ingress = var.enable_ingress

      hostname       = var.hostname

    })

  ]

}

Pros:

• Still gets variable injection
• Can be validated as YAML (with placeholders)
• Clean and readable

(4) Use Helm's set for Dynamic Values

Keep static config in files, override specific values:

resource "helm_release" "my_app" {

  name       = "my-app"

  chart      = "my-chart"

  

  # Base values from file

  values = [

    file("${path.module}/helm-values.yaml")

  ]

  

  # Override specific values dynamically

  set {

    name  = "image.tag"

    value = var.image_tag

  }

  

  set {

    name  = "replicaCount"

    value = var.replica_count

  }

  

  set_sensitive {

    name  = "secret.password"

    value = var.db_password

  }

}

Pros:

• Clear what's dynamic vs static
• Base values file can be validated
• Sensitive values handled properly

Here is the example how we can migrate inline YAML from the above to templated file:

helm-values.yaml:

image:

  repository: myapp

service:

  type: LoadBalancer

main.tf:

resource "helm_release" "app" {

  values = [

    file("${path.module}/helm-values.yaml")

  ]

  

  set {

    name  = "replicaCount"

    value = var.replicas

  }

  

  set {

    name  = "image.tag"

    value = var.tag

  }

}

Now we can run: 

% pluto detect-files -f helm-values.yaml

(5) Multiple Values Files

We can layer our configuration:

resource "helm_release" "my_app" {

  name  = "my-app"

  chart = "my-chart"

  

  values = [

    file("${path.module}/helm-values-base.yaml"),

    file("${path.module}/helm-values-${var.environment}.yaml")

  ]

}

---

Introduction to Kubent (Kube No Trouble)

 

Kubent (Kube No Trouble)  [this link was the original repo, see comments below] is a tool which scans k8s cluster and reports resources that use deprecated or removed Kubernetes APIs, based on the target Kubernetes version. It’s especially useful before upgrading (e.g., EKS 1.32 → 1.33)

WARNING: Development at project's original repo (https://github.com/doitintl/kube-no-trouble) is not active anymore as the last commit was in January 2025. The original author announced here that they would be moving development to https://github.com/dark0dave/kube-no-trouble and that repo is ssemingly active as of today (last change was ) BUT https://github.com/dark0dave/kube-no-trouble/tree/301e5783904de5966f79b217a956651146630f50/pkg/rules/rego shows that rulesets only up to v1.32 were added (!).

Kube No Trouble relies on static Rego rule files in the repo. If new Kubernetes versions (e.g., >1.32) don’t have updated rules, then:

• It won’t know about newly deprecated APIs
• It won’t know about newly removed APIs
• --target-version becomes unreliable for newer releases

For modern upgrades (especially 1.32 → 1.33+), kubent is no longer the safest tool.

To install it:

% sh -c "$(curl -sSL https://git.io/install-kubent)"

>>> kubent installation script <<<

> Detecting latest version

> Downloading version 0.7.3

Target directory (/usr/local/bin) is not writable, trying to use sudo

Password:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 12.4M  100 12.4M    0     0  14.7M      0 --:--:-- --:--:-- --:--:-- 13.2M

> Done. kubent was installed to /usr/local/bin/.

To verify installation:

% kubent --version

7:48AM INF version 0.7.3 (git sha 57480c07b3f91238f12a35d0ec88d9368aae99aa)

To check CLI arguments:

% kubent --help   

Usage of kubent:

  -A, --additional-annotation strings   additional annotations that should be checked to determine the last applied config

  -a, --additional-kind strings         additional kinds of resources to report in Kind.version.group.com format

  -c, --cluster                         enable Cluster collector (default true)

  -x, --context string                  kubeconfig context

  -e, --exit-error                      exit with non-zero code when issues are found

  -f, --filename strings                manifests to check, use - for stdin

      --helm3                           enable Helm v3 collector (default true)

  -k, --kubeconfig string               path to the kubeconfig file

  -l, --log-level string                set log level (trace, debug, info, warn, error, fatal, panic, disabled) (default "info")

  -o, --output string                   output format - [text|json|csv] (default "text")

  -O, --output-file string              output file, use - for stdout (default "-")

  -t, --target-version string           target K8s version in SemVer format (autodetected by default)

  -v, --version                         prints the version of kubent and exits

pflag: help requested

It looks at default ~/.kube/config file in order to find the current context, otherwise use -k to specify kubeconfig at non-default location.

% kubent                       

7:59AM INF >>> Kube No Trouble `kubent` <<<

7:59AM INF version 0.7.3 (git sha 57480c07b3f91238f12a35d0ec88d9368aae99aa)

7:59AM INF Initializing collectors and retrieving data

7:59AM INF Target K8s version is 1.32.11-eks-ac2d5a0

7:59AM INF Retrieved 12 resources from collector name=Cluster

8:00AM INF Retrieved 361 resources from collector name="Helm v3"

8:00AM INF Loaded ruleset name=custom.rego.tmpl

8:00AM INF Loaded ruleset name=deprecated-1-16.rego

8:00AM INF Loaded ruleset name=deprecated-1-22.rego

8:00AM INF Loaded ruleset name=deprecated-1-25.rego

8:00AM INF Loaded ruleset name=deprecated-1-26.rego

8:00AM INF Loaded ruleset name=deprecated-1-27.rego

8:00AM INF Loaded ruleset name=deprecated-1-29.rego

8:00AM INF Loaded ruleset name=deprecated-1-32.rego

8:00AM INF Loaded ruleset name=deprecated-future.rego

Running kubent with no other arguments:

• Connects to your current kube-context
• Detects your cluster version automatically
• Scans all namespaces
• Compares resources against deprecations for that version

Before the upgrade to v1.33, we want kubent to scan the resources against that next k8s version so we need to specify it with --target-version:

% kubent --target-version=1.33

8:02AM INF >>> Kube No Trouble `kubent` <<<

8:02AM INF version 0.7.3 (git sha 57480c07b3f91238f12a35d0ec88d9368aae99aa)

8:02AM INF Initializing collectors and retrieving data

8:02AM INF Target K8s version is 1.33.0

8:02AM INF Retrieved 12 resources from collector name=Cluster

8:03AM INF Retrieved 361 resources from collector name="Helm v3"

8:03AM INF Loaded ruleset name=custom.rego.tmpl

8:03AM INF Loaded ruleset name=deprecated-1-16.rego

8:03AM INF Loaded ruleset name=deprecated-1-22.rego

8:03AM INF Loaded ruleset name=deprecated-1-25.rego

8:03AM INF Loaded ruleset name=deprecated-1-26.rego

8:03AM INF Loaded ruleset name=deprecated-1-27.rego

8:03AM INF Loaded ruleset name=deprecated-1-29.rego

8:03AM INF Loaded ruleset name=deprecated-1-32.rego

8:03AM INF Loaded ruleset name=deprecated-future.rego

---

Introduction to Grafana Loki

Grafana Loki:

• Log aggregation system. Like Prometheus, but for logs
• Repo: https://github.com/grafana/loki

These are the notes from Loki Helm chart: 

***********************************************************************

  Welcome to Grafana Loki

  Chart version: 6.31.0

  Chart Name: loki

  Loki version: 3.5.0

***********************************************************************

Tip:

Watch the deployment status using the command: kubectl get pods -w --namespace grafana-loki

If pods are taking too long to schedule make sure pod affinity can be fulfilled in the current cluster.

***********************************************************************

Installed components:

***********************************************************************

* gateway

* read

* write

* backend

***********************************************************************

Sending logs to Loki

***********************************************************************

Loki has been configured with a gateway (nginx) to support reads and writes from a single component.

You can send logs from inside the cluster using the cluster DNS:

http://loki-gateway.grafana-loki.svc.cluster.local/loki/api/v1/push

You can test to send data from outside the cluster by port-forwarding the gateway to your local machine:

  kubectl port-forward --namespace grafana-loki svc/loki-gateway 3100:80 &

And then using http://127.0.0.1:3100/loki/api/v1/push URL as shown below:

curl \

-H "Content-Type: application/json" \

-XPOST \

-s "http://127.0.0.1:3100/loki/api/v1/push"  \

--data-raw "{\"streams\": [{\"stream\": {\"job\": \"test\"}, \"values\": [[\"$(date +%s)000000000\", \"fizzbuzz\"]]}]}" \

-H X-Scope-OrgId:foo

Then verify that Loki did receive the data using the following command:

curl "http://127.0.0.1:3100/loki/api/v1/query_range" \

--data-urlencode 'query={job="test"}' \

-H X-Scope-OrgId:foo | jq .data.result

***********************************************************************

Connecting Grafana to Loki

***********************************************************************

If Grafana operates within the cluster, you'll set up a new Loki datasource by utilizing the following URL:

http://loki-gateway.grafana-loki.svc.cluster.local/

***********************************************************************

Multi-tenancy

***********************************************************************

Loki is configured with auth enabled (multi-tenancy) and expects tenant headers (`X-Scope-OrgID`) to be set for all API calls.

You must configure Grafana's Loki datasource using the `HTTP Headers` section with the `X-Scope-OrgID` to target a specific tenant.

For each tenant, you can create a different datasource.

The agent of your choice must also be configured to propagate this header.

For example, when using Promtail you can use the `tenant` stage. https://grafana.com/docs/loki/latest/send-data/promtail/stages/tenant/

When not provided with the `X-Scope-OrgID` while auth is enabled, Loki will reject reads and writes with a 404 status code `no org id`.

You can also use a reverse proxy, to automatically add the `X-Scope-OrgID` header as suggested by https://grafana.com/docs/loki/latest/operations/authentication/

For more information, read our documentation about multi-tenancy: https://grafana.com/docs/loki/latest/operations/multi-tenancy/

> When using curl you can pass `X-Scope-OrgId` header using `-H X-Scope-OrgId:foo` option, where foo can be replaced with the tenant of your choice.

EOT -> (known after apply)

 

---

Grafana Observability Stack

 

Grafana uses these components together as an observability stack, but each has a clear role:

Loki – log database. It stores and indexes logs (especially from Kubernetes) in a cost‑efficient, label‑based way, similar to Prometheus but for logs.

Tempo – distributed tracing backend. It stores distributed traces (spans) from OpenTelemetry, Jaeger, Zipkin, etc., so you can see call flows across microservices and where latency comes from.

Mimir – Prometheus‑compatible metrics backend. It is a horizontally scalable, long‑term storage and query engine for Prometheus‑style metrics (time series).

Alloy – telemetry pipeline (collector). It is Grafana’s distribution of the OpenTelemetry Collector / Prometheus agent / Promtail ideas, used to collect, process, and forward metrics, logs, traces, profiles into Loki/Tempo/Mimir (or other backends).

How Grafana UI relates to them

Grafana UI itself is “just” the visualization and alerting layer:

• It connects to Loki, Tempo, Mimir (and many others) as data sources.
• For each backend you configure:
• A Loki data source for logs.
• A Tempo data source for traces.
• A Prometheus/Mimir data source for metrics (Mimir exposes a Prometheus‑compatible API).
• Grafana then lets you:
• Build dashboards and alerts from Mimir metrics.
• Explore logs from Loki.
• Explore traces from Tempo and cross‑link them with logs/metrics (e.g., click from a log line to a trace, or from a metrics graph into logs/traces).

A useful mental model: Loki/Tempo/Mimir are databases, Alloy is the collector/router, and Grafana is the UI on top.

Are they deployed in the same Kubernetes cluster?

Common patterns:

• Very common: deploy Loki, Tempo, Mimir, Alloy, and Grafana in the same Kubernetes cluster as your apps. This is the typical “in‑cluster LGTM” setup; all telemetry stays inside the cluster and traffic is simple.
• Also common: run them in a separate observability cluster (or use Grafana Cloud backends), while Alloy/agents run in each workload cluster and ship data over the network. This improves isolation and makes it easier to share one observability stack across many clusters.
• In smaller setups or dev environments, everything (apps + LGTM + Grafana) often lives in one cluster; in larger/regulated setups, people tend to separate “workload clusters” and an “observability cluster”.

So: they don’t have to be on the same cluster, but it’s perfectly normal (and often simplest) to run Grafana + Loki + Tempo + Mimir + Alloy together in a single Kubernetes cluster and point your apps’ telemetry to Alloy.

Why not using elasticsearch instead of loki, tempo and mimir?

Elasticsearch can replace part of what Loki, Tempo, and Mimir do, but not all of it, and usually with higher cost/complexity for cloud‑native observability.

1. Scope: logs vs full observability

Elasticsearch is a general search and analytics engine that’s great at full‑text search, aggregations, and analytics over documents (including logs).

The LGTM stack is explicitly split by signal:

• Loki → logs
• Tempo → traces
• Mimir → metrics

Each is optimized only for its signal type and integrates tightly with Grafana and modern telemetry standards.

You could plausibly replace Loki with Elasticsearch for logs, but Elasticsearch does not natively replace Tempo (distributed tracing backend) or Mimir (Prometheus‑compatible metrics backend).

2. Logs: Loki vs Elasticsearch

Elasticsearch strengths:

• Very powerful full‑text search, fuzzy matching, relevance scoring, complex aggregations.
• Good when you need deep forensic search and advanced analytics on log text.

Loki strengths:

• Stores logs as compressed chunks plus a small label index, so storage and compute are much cheaper than Elasticsearch for typical Kubernetes logs.
• Very tight integration with Grafana and the rest of LGTM, and simple, label‑based querying.

Trade‑off: Elasticsearch gives richer search at a high infra + ops cost, Loki gives “good enough” search for operational troubleshooting with much lower cost and operational burden.

3. Traces and metrics: Tempo & Mimir vs “just ES”

Tempo:

• Implements distributed tracing concepts (spans, traces, service graphs) and OpenTelemetry/Jaeger/Zipkin protocols; the data model and APIs are specialized for traces.
• Elasticsearch can store trace‑like JSON documents, but you’d have to build/maintain all the trace stitching, UI navigation, and integrations yourself.

Mimir:

• Is a horizontally scalable, Prometheus‑compatible time‑series database, with native remote‑write/read and PromQL semantics.
• Elasticsearch can store time‑stamped metrics, but you lose Prometheus compatibility, PromQL semantics, and the whole ecosystem that expects a Prometheus‑style API.

So using only Elasticsearch means you’re giving up the standard metrics and tracing ecosystems and rebuilding a lot of tooling on top of a generic search engine.

​

4. Cost, complexity, and operational burden

Elasticsearch clusters generally need:

• More RAM/CPU per node, careful shard and index management, and capacity planning.
• Storage overhead from full‑text indexes (often 1.5–3× raw log size plus replicas).

​

Loki/Tempo/Mimir:

• Are designed for object storage, compression, and label‑only indexing, which dramatically lowers storage and compute requirements for logs and metrics.
• Have simpler, well‑documented reference architectures specifically for observability.

For a modern Kubernetes‑centric environment, that usually makes LGTM cheaper and easier to run than a single big Elasticsearch cluster for everything.

5. When Elasticsearch still makes sense

You might still choose Elasticsearch (often with Kibana/APM) if:

• You already have a strong ELK stack and team expertise.
• Your primary need is deep, flexible text search and analytics over logs, with less emphasis on Prometheus/OTel ecosystems.
• You want Elasticsearch’s ML/anomaly‑detection features and are willing to pay the operational cost.

But if your goal is a Grafana‑centric, standards‑based (Prometheus + OpenTelemetry) observability platform, LGTM (Loki+Tempo+Mimir, plus Alloy as collector) is a better fit than trying to push everything into Elasticsearch.

---