Introduction to KubePug (Kubernetes tool)

What is KubePug?

KubePug (Kubernetes PreUpGrade Checker) is an open-source kubectl plugin and CLI tool designed to identify deprecated or deleted APIs in your Kubernetes cluster or manifest files before you perform an upgrade. 

Key Features

• Deprecation Detection: Scans your live cluster or static YAML manifests to find resources using APIs that are slated for removal in future Kubernetes versions.
• Replacement Guidance: Not only flags outdated APIs but also suggests the recommended replacement API and specifies the exact version where the deprecation or deletion occurs.
• Version Targeting: Allows you to specify a target Kubernetes version (e.g., v1.31) to validate your current resources against that specific future release.
• Flexible Data Source: It automatically downloads a frequently updated API definition file (every 30 minutes) to stay current with the latest Kubernetes releases. 

Why Use It?

As Kubernetes evolves, APIs are moved from alpha/beta to stable (GA), and older versions are eventually removed. If you upgrade your cluster without updating your manifests, those resources will fail to deploy or operate. KubePug provides a "pre-flight" check to prevent these breaking changes from reaching production.

Installation & Usage

You can install KubePug via Krew (where it is listed under the name deprecations) or as a standalone binary. 

Method                  Command

---------                    -------------

Install via Krew kubectl krew install deprecations

Scan Live Cluster kubectl deprecations --k8s-version=v1.30

Scan Manifest File kubepug --input-file=./my-manifest.yaml

Installation via Krew

% kubectl krew install deprecations

Updated the local copy of plugin index.

Installing plugin: deprecations

Installed plugin: deprecations

\

 | Use this plugin:

 | kubectl deprecations

 | Documentation:

 | https://github.com/rikatz/kubepug

 | Caveats:

 | \

 |  | * By default, deprecations finds deprecated object relative to the current kubernetes

 |  | master branch. To target a different kubernetes release, use the --k8s-version

 |  | argument.

 |  | 

 |  | * Deprecations needs permission to GET all objects in the Cluster

 | /

/

WARNING: You installed plugin "deprecations" from the krew-index plugin repository.

   These plugins are not audited for security by the Krew maintainers.

   Run them at your own risk.

Execution

Once installed, the plugin is invoked using kubectl deprecations.

Scan Current Cluster

Check your live cluster for deprecated APIs against a specific target Kubernetes version:

% kubectl deprecations --k8s-version=v1.33

                    

Error: failed to get apiservices: apiservices.apiregistration.k8s.io is forbidden: User "sso:user" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope

time="2026-02-26T14:20:19Z" level=error msg="An error has occurred: failed to get apiservices: apiservices.apiregistration.k8s.io is forbidden: User \"sso:user\" cannot list resource \"apiservices\" in API group \"apiregistration.k8s.io\" at the cluster scope"

KubePug requires running user to have "list" permissions on resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope as otherwise the above error will appear.

If required permissions are in place:

% kubectl deprecations --k8s-version=v1.33

No deprecated or deleted APIs found

Kubepug validates the APIs using Kubernetes markers. To know what are the deprecated and deleted APIS it checks, please go to https://kubepug.xyz/status/

Scan Local Manifest Files

Validate static YAML files before applying them to a cluster:

kubectl deprecations --input-file=./my-manifests/

View Results in Different Formats

Output the findings in json or yaml for automated processing:

kubectl deprecations --format=json

Check for Help and Flags

See all available configuration options, such as using a custom database file or setting error codes:

kubectl deprecations --help

Key Parameters

--k8s-version: The Kubernetes release you intend to upgrade to (defaults to the latest stable).

--error-on-deprecated: Forces the command to exit with an error code if deprecated APIs are found, which is useful for CI/CD pipelines. 

---

Introduction to Krew (Kubernetes tool)

Krew is the official plugin manager for the kubectl command-line tool. Much like apt for Debian or Homebrew for macOS, it allows users to easily discover, install, and manage custom extensions that add new subcommands to Kubernetes. 

Core Functionality

• Discovery: Users can search a community-curated index of over 200 plugins designed for tasks like security auditing, resource visualization, and cluster management.
• Lifecycle Management: It automates the process of installing, updating, and removing plugins across different operating systems (Linux, macOS, and Windows).
• Unified Interface: Once a plugin is installed via Krew, it is invoked directly through kubectl (e.g., kubectl <plugin-name>). 

Installation

MacOS example output:

% (

  set -x; cd "$(mktemp -d)" &&

  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&

  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&

  KREW="krew-${OS}_${ARCH}" &&

  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&

  tar zxvf "${KREW}.tar.gz" &&

  ./"${KREW}" install krew

)

+-zsh:142> mktemp -d

+-zsh:142> cd /var/folders/8j/60m6_18j359_39ls0sr9ccvm0000gp/T/tmp.LKDSyQcO10

+-zsh:143> OS=+-zsh:143> uname

+-zsh:143> OS=+-zsh:143> tr '[:upper:]' '[:lower:]'

+-zsh:143> OS=darwin 

+-zsh:144> ARCH=+-zsh:144> uname -m

+-zsh:144> ARCH=+-zsh:144> sed -e s/x86_64/amd64/ -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/'

+-zsh:144> ARCH=arm64 

+-zsh:145> KREW=krew-darwin_arm64 

+-zsh:146> curl -fsSLO https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-darwin_arm64.tar.gz

+-zsh:147> tar zxvf krew-darwin_arm64.tar.gz

x ./LICENSE

x ./krew-darwin_arm64

+-zsh:148> ./krew-darwin_arm64 install krew

Adding "default" plugin index from https://github.com/kubernetes-sigs/krew-index.git.

Updated the local copy of plugin index.

Installing plugin: krew

Installed plugin: krew

\

 | Use this plugin:

 | kubectl krew

 | Documentation:

 | https://krew.sigs.k8s.io/

 | Caveats:

 | \

 |  | krew is now installed! To start using kubectl plugins, you need to add

 |  | krew's installation directory to your PATH:

 |  | 

 |  |   * macOS/Linux:

 |  |     - Add the following to your ~/.bashrc or ~/.zshrc:

 |  |         export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

 |  |     - Restart your shell.

 |  | 

 |  |   * Windows: Add %USERPROFILE%\.krew\bin to your PATH environment variable

 |  | 

 |  | To list krew commands and to get help, run:

 |  |   $ kubectl krew

 |  | For a full list of available plugins, run:

 |  |   $ kubectl krew search

 |  | 

 |  | You can find documentation at

 |  |   https://krew.sigs.k8s.io/docs/user-guide/quickstart/.

 | /

/

Add Krew path to PATHs:

% vi ~/.zshrc 

% cat ~/.zshrc   

...

export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH" <-- added manually

Reload .zshrc configuration file within the currently running Zsh terminal (or just restart the terminal):

% source ~/.zshrc

Installation verification:

% kubectl krew

krew is the kubectl plugin manager.

You can invoke krew through kubectl: "kubectl krew [command]..."

Usage:

  kubectl krew [command]

Available Commands:

  help        Help about any command

  index       Manage custom plugin indexes

  info        Show information about an available plugin

  install     Install kubectl plugins

  list        List installed kubectl plugins

  search      Discover kubectl plugins

  uninstall   Uninstall plugins

  update      Update the local copy of the plugin index

  upgrade     Upgrade installed plugins to newer versions

  version     Show krew version and diagnostics

Flags:

  -h, --help      help for krew

  -v, --v Level   number for the log level verbosity

Use "kubectl krew [command] --help" for more information about a command.

Common Commands

To use Krew, you must first install it as a kubectl plugin itself. Key commands include: 

kubectl krew update: Updates the local list of available plugins.

kubectl krew search: Finds plugins in the official Krew index.

kubectl krew install <plugin>: Installs a specific plugin.

kubectl krew list: Displays all plugins currently installed through Krew.

kubectl krew upgrade: Updates all installed plugins to their latest versions. 

Example:

% kubectl krew install deprecations

Updated the local copy of plugin index.

Installing plugin: deprecations

Installed plugin: deprecations

\

 | Use this plugin:

 | kubectl deprecations

 | Documentation:

 | https://github.com/rikatz/kubepug

 | Caveats:

 | \

 |  | * By default, deprecations finds deprecated object relative to the current kubernetes

 |  | master branch. To target a different kubernetes release, use the --k8s-version

 |  | argument.

 |  | 

 |  | * Deprecations needs permission to GET all objects in the Cluster

 | /

/

WARNING: You installed plugin "deprecations" from the krew-index plugin repository.

   These plugins are not audited for security by the Krew maintainers.

   Run them at your own risk.

Popular Plugins Managed by Krew 

• ctx / ns: Rapidly switch between Kubernetes contexts and namespaces.
• tree: Visualizes the hierarchy of Kubernetes resources in a tree view.
• access-matrix: Displays an RBAC (Role-Based Access Control) matrix for server resources.
• get-all: Lists all resources in a namespace, including those often missed by kubectl get all. 

Note on Security (!)

Plugins in the Krew index are community-contributed and are not audited for security by the Kubernetes maintainers; you should only install plugins from sources you trust. 

---

Introduction to Pluto (Kubernetes tool)

 

Pluto is:

• CLI tool that helps users find deprecated Kubernetes API versions in your code repositories and Helm releases. 
• It's especially useful when upgrading Kubernetes clusters, as it identifies resources that need updating before the upgrade.
• It works against:
• Live clusters
• Helm charts
• Raw YAML
• Pluto will show which APIs are deprecated or removed, what version they were deprecated in, and what the replacement API should be.

Installation on Mac (https://pluto.docs.fairwinds.com/installation/#homebrew-tap):

% brew install FairwindsOps/tap/pluto

Let's see its CLI arguments:

% pluto                    

You must specify a sub-command.

A tool to detect Kubernetes apiVersions

Usage:

  pluto [flags]

  pluto [command]

Available Commands:

  completion            Generate the autocompletion script for the specified shell

  detect                Checks a single file or stdin for deprecated apiVersions.

  detect-all-in-cluster run all in-cluster detections

  detect-api-resources  detect-api-resources

  detect-files          detect-files

  detect-helm           detect-helm

  help                  Help about any command

  list-versions         Outputs a JSON object of the versions that Pluto knows about.

  version               Prints the current version of the tool.

Flags:

  -f, --additional-versions string        Additional deprecated versions file to add to the list. Cannot contain any existing versions

      --columns strings                   A list of columns to print. Mandatory when using --output custom, optional with --output markdown

      --components strings                A list of components to run checks for. If nil, will check for all found in versions.

  -h, --help                              help for pluto

      --ignore-deprecations               Ignore the default behavior to exit 2 if deprecated apiVersions are found. (Only show removed APIs, not just deprecated ones)

      --ignore-removals                   Ignore the default behavior to exit 3 if removed apiVersions are found. (Only show deprecated APIs, not removed ones)

      --ignore-unavailable-replacements   Ignore the default behavior to exit 4 if deprecated but unavailable apiVersions are found.

  -H, --no-headers                        When using the default or custom-column output format, don't print headers (default print headers).

  -r, --only-show-removed                 Only display the apiVersions that have been removed in the target version.

  -o, --output string                     The output format to use. (normal|wide|custom|json|yaml|markdown|csv) (default "normal")

  -t, --target-versions stringToString    A map of targetVersions to use. This flag supersedes all defaults in version files. (default [])

  -v, --v Level                           number for the log level verbosity

Use "pluto [command] --help" for more information about a command.

detect-files

To scan and detect deprecated APIs in manifest files in a directory:

% pluto detect-files -d /path/to/your/manifests

detect-files is for checking YAML files in our repositories/filesystem before deploying them - that's separate from detect-helm and detect-all-in-cluster cluster commands.

To target particular k8s version:

% pluto detect-files -d . --target-versions k8s=v1.33.0

If we use Terraform to deploy Helm charts, we might want to keep chart values in separate files (.yaml or .yaml.tpl) as otherwise we won't be able to use Pluto directly (we'd need to extract values into files first). For more details, see Where to keep Helm chart values in Terraform projects | My Public Notepad. 

detect-helm

To check Helm releases in the cluster (already deployed):

% pluto detect-helm -owide

To target particular k8s version:

% pluto detect-helm -owide --target-versions k8s=v1.33.0   

There were no resources found with known deprecated apiVersions.

detect-helm specifically checks Helm release metadata stored in our cluster (in secrets or configmaps) after Helm chart have been deployed. It looks at the manifests that Helm used to install releases, which might contain deprecated APIs even if they haven't been applied yet or are stored in Helm's history.

detect-all-in-cluster

To check all resources in the cluster:

% pluto detect-all-in-cluster -o wide

I0226 12:01:02.279788   47100 warnings.go:110] "Warning: v1 ComponentStatus is deprecated in v1.19+"

There were no resources found with known deprecated apiVersions.

detect-all-in-cluster scans all live resources currently running in our cluster by querying the Kubernetes API directly. It checks deployments, services, pods, etc. that are actively deployed.

detect-all-in-cluster does NOT include detect-helm or detect-files. Here's why they're separate:

• detect-all-in-cluster sees the current state of resources
• detect-helm sees Helm's stored templates and history, which may include:
• Templated manifests that haven't been rendered yet
• Old release revisions
• Chart templates with deprecated APIs
• Run both to get complete coverage!

Target a specific Kubernetes version:

% pluto detect-all-in-cluster --target-versions k8s=v1.33.0

I0226 12:02:26.551401   47113 warnings.go:110] "Warning: v1 ComponentStatus is deprecated in v1.19+"

There were no resources found with known deprecated apiVersions.

The warning message:

Warning: v1 ComponentStatus is deprecated in v1.19+

This is just Pluto itself triggering a Kubernetes API warning while scanning - it's not something wrong with our cluster resources.

The main result:

There were no resources found with known deprecated apiVersions.

This means all our cluster resources are using API versions that are still valid in Kubernetes v1.33.0 (our target version). This means:

• Our cluster resources are already compatible with k8s v1.33.0
• No manifests need updating before upgrading
• No deprecated APIs that would be removed in v1.33.0

To be thorough before a k8s upgrade, we need to run all three commands:

• detect-files
• detect-helm
• detect-all-in-cluster

---

Where to keep Helm chart values in Terraform projects

If we use Terraform to deploy Helm charts, we might be using one of these strategies to keep chart values:

1. Values are in inline YAML string
2. Values in separate .yaml file
3. Values in separate YAML Template files (.yaml.tpl)
4. Use Helm's set for Dynamic Values
5. Multiple Values Files

(1) Values in inline YAML string

This is not ideal as problems with Inline YAML in Terraform include:

• No syntax highlighting or validation - Easy to break YAML formatting
• Hard to review in diffs - Changes are messy in PRs
• Can't use standard tooling - No yamllint, Pluto, or other YAML tools
• Mixing concerns - Infrastructure code mixed with application config
• Escaping nightmares - Terraform string interpolation conflicts with Helm templating

Example:

resource "helm_release" "app" {

  values = [<<-EOT

    replicaCount: ${var.replicas}

    image:

      repository: myapp

      tag: ${var.tag}

    service:

      type: LoadBalancer

  EOT

  ]

}

(2) Separate Values Files 

Keep values in YAML files, reference them in Terraform.

This is a better approach because:

• Clean separation
• Easy to validate with standard tools
• Better diffs
• Can use Pluto directly: pluto detect-files -d .

Example:

main.tf:

resource "helm_release" "my_app" {

  name       = "my-app"

  chart      = "my-chart"

  repository = "https://charts.example.com"

  

  values = [

    file("${path.module}/helm-values.yaml")

  ]

}

(3) Templated Values Files

Use Terraform's templatefile() to inject dynamic values:

helm-values.yaml.tpl:

replicaCount: ${replica_count}

image:

  repository: ${image_repo}

  tag: ${image_tag}

ingress:

  enabled: ${enable_ingress}

  host: ${hostname}

main.tf:

resource "helm_release" "my_app" {

  name  = "my-app"

  chart = "my-chart"

  

  values = [

    templatefile("${path.module}/helm-values.yaml.tpl", {

      replica_count  = var.replica_count

      image_repo     = var.image_repository

      image_tag      = var.image_tag

      enable_ingress = var.enable_ingress

      hostname       = var.hostname

    })

  ]

}

Pros:

• Still gets variable injection
• Can be validated as YAML (with placeholders)
• Clean and readable

(4) Use Helm's set for Dynamic Values

Keep static config in files, override specific values:

resource "helm_release" "my_app" {

  name       = "my-app"

  chart      = "my-chart"

  

  # Base values from file

  values = [

    file("${path.module}/helm-values.yaml")

  ]

  

  # Override specific values dynamically

  set {

    name  = "image.tag"

    value = var.image_tag

  }

  

  set {

    name  = "replicaCount"

    value = var.replica_count

  }

  

  set_sensitive {

    name  = "secret.password"

    value = var.db_password

  }

}

Pros:

• Clear what's dynamic vs static
• Base values file can be validated
• Sensitive values handled properly

Here is the example how we can migrate inline YAML from the above to templated file:

helm-values.yaml:

image:

  repository: myapp

service:

  type: LoadBalancer

main.tf:

resource "helm_release" "app" {

  values = [

    file("${path.module}/helm-values.yaml")

  ]

  

  set {

    name  = "replicaCount"

    value = var.replicas

  }

  

  set {

    name  = "image.tag"

    value = var.tag

  }

}

Now we can run: 

% pluto detect-files -f helm-values.yaml

(5) Multiple Values Files

We can layer our configuration:

resource "helm_release" "my_app" {

  name  = "my-app"

  chart = "my-chart"

  

  values = [

    file("${path.module}/helm-values-base.yaml"),

    file("${path.module}/helm-values-${var.environment}.yaml")

  ]

}

---

Introduction to Kubent (Kube No Trouble)

 

Kubent (Kube No Trouble)  [this link was the original repo, see comments below] is a tool which scans k8s cluster and reports resources that use deprecated or removed Kubernetes APIs, based on the target Kubernetes version. It’s especially useful before upgrading (e.g., EKS 1.32 → 1.33)

WARNING: Development at project's original repo (https://github.com/doitintl/kube-no-trouble) is not active anymore as the last commit was in January 2025. The original author announced here that they would be moving development to https://github.com/dark0dave/kube-no-trouble and that repo is ssemingly active as of today (last change was ) BUT https://github.com/dark0dave/kube-no-trouble/tree/301e5783904de5966f79b217a956651146630f50/pkg/rules/rego shows that rulesets only up to v1.32 were added (!).

Kube No Trouble relies on static Rego rule files in the repo. If new Kubernetes versions (e.g., >1.32) don’t have updated rules, then:

• It won’t know about newly deprecated APIs
• It won’t know about newly removed APIs
• --target-version becomes unreliable for newer releases

For modern upgrades (especially 1.32 → 1.33+), kubent is no longer the safest tool.

To install it:

% sh -c "$(curl -sSL https://git.io/install-kubent)"

>>> kubent installation script <<<

> Detecting latest version

> Downloading version 0.7.3

Target directory (/usr/local/bin) is not writable, trying to use sudo

Password:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 12.4M  100 12.4M    0     0  14.7M      0 --:--:-- --:--:-- --:--:-- 13.2M

> Done. kubent was installed to /usr/local/bin/.

To verify installation:

% kubent --version

7:48AM INF version 0.7.3 (git sha 57480c07b3f91238f12a35d0ec88d9368aae99aa)

To check CLI arguments:

% kubent --help   

Usage of kubent:

  -A, --additional-annotation strings   additional annotations that should be checked to determine the last applied config

  -a, --additional-kind strings         additional kinds of resources to report in Kind.version.group.com format

  -c, --cluster                         enable Cluster collector (default true)

  -x, --context string                  kubeconfig context

  -e, --exit-error                      exit with non-zero code when issues are found

  -f, --filename strings                manifests to check, use - for stdin

      --helm3                           enable Helm v3 collector (default true)

  -k, --kubeconfig string               path to the kubeconfig file

  -l, --log-level string                set log level (trace, debug, info, warn, error, fatal, panic, disabled) (default "info")

  -o, --output string                   output format - [text|json|csv] (default "text")

  -O, --output-file string              output file, use - for stdout (default "-")

  -t, --target-version string           target K8s version in SemVer format (autodetected by default)

  -v, --version                         prints the version of kubent and exits

pflag: help requested

It looks at default ~/.kube/config file in order to find the current context, otherwise use -k to specify kubeconfig at non-default location.

% kubent                       

7:59AM INF >>> Kube No Trouble `kubent` <<<

7:59AM INF version 0.7.3 (git sha 57480c07b3f91238f12a35d0ec88d9368aae99aa)

7:59AM INF Initializing collectors and retrieving data

7:59AM INF Target K8s version is 1.32.11-eks-ac2d5a0

7:59AM INF Retrieved 12 resources from collector name=Cluster

8:00AM INF Retrieved 361 resources from collector name="Helm v3"

8:00AM INF Loaded ruleset name=custom.rego.tmpl

8:00AM INF Loaded ruleset name=deprecated-1-16.rego

8:00AM INF Loaded ruleset name=deprecated-1-22.rego

8:00AM INF Loaded ruleset name=deprecated-1-25.rego

8:00AM INF Loaded ruleset name=deprecated-1-26.rego

8:00AM INF Loaded ruleset name=deprecated-1-27.rego

8:00AM INF Loaded ruleset name=deprecated-1-29.rego

8:00AM INF Loaded ruleset name=deprecated-1-32.rego

8:00AM INF Loaded ruleset name=deprecated-future.rego

Running kubent with no other arguments:

• Connects to your current kube-context
• Detects your cluster version automatically
• Scans all namespaces
• Compares resources against deprecations for that version

Before the upgrade to v1.33, we want kubent to scan the resources against that next k8s version so we need to specify it with --target-version:

% kubent --target-version=1.33

8:02AM INF >>> Kube No Trouble `kubent` <<<

8:02AM INF version 0.7.3 (git sha 57480c07b3f91238f12a35d0ec88d9368aae99aa)

8:02AM INF Initializing collectors and retrieving data

8:02AM INF Target K8s version is 1.33.0

8:02AM INF Retrieved 12 resources from collector name=Cluster

8:03AM INF Retrieved 361 resources from collector name="Helm v3"

8:03AM INF Loaded ruleset name=custom.rego.tmpl

8:03AM INF Loaded ruleset name=deprecated-1-16.rego

8:03AM INF Loaded ruleset name=deprecated-1-22.rego

8:03AM INF Loaded ruleset name=deprecated-1-25.rego

8:03AM INF Loaded ruleset name=deprecated-1-26.rego

8:03AM INF Loaded ruleset name=deprecated-1-27.rego

8:03AM INF Loaded ruleset name=deprecated-1-29.rego

8:03AM INF Loaded ruleset name=deprecated-1-32.rego

8:03AM INF Loaded ruleset name=deprecated-future.rego

---

Introduction to Grafana Loki

Grafana Loki:

• Log aggregation system. Like Prometheus, but for logs
• Repo: https://github.com/grafana/loki

These are the notes from Loki Helm chart: 

***********************************************************************

  Welcome to Grafana Loki

  Chart version: 6.31.0

  Chart Name: loki

  Loki version: 3.5.0

***********************************************************************

Tip:

Watch the deployment status using the command: kubectl get pods -w --namespace grafana-loki

If pods are taking too long to schedule make sure pod affinity can be fulfilled in the current cluster.

***********************************************************************

Installed components:

***********************************************************************

* gateway

* read

* write

* backend

***********************************************************************

Sending logs to Loki

***********************************************************************

Loki has been configured with a gateway (nginx) to support reads and writes from a single component.

You can send logs from inside the cluster using the cluster DNS:

http://loki-gateway.grafana-loki.svc.cluster.local/loki/api/v1/push

You can test to send data from outside the cluster by port-forwarding the gateway to your local machine:

  kubectl port-forward --namespace grafana-loki svc/loki-gateway 3100:80 &

And then using http://127.0.0.1:3100/loki/api/v1/push URL as shown below:

curl \

-H "Content-Type: application/json" \

-XPOST \

-s "http://127.0.0.1:3100/loki/api/v1/push"  \

--data-raw "{\"streams\": [{\"stream\": {\"job\": \"test\"}, \"values\": [[\"$(date +%s)000000000\", \"fizzbuzz\"]]}]}" \

-H X-Scope-OrgId:foo

Then verify that Loki did receive the data using the following command:

curl "http://127.0.0.1:3100/loki/api/v1/query_range" \

--data-urlencode 'query={job="test"}' \

-H X-Scope-OrgId:foo | jq .data.result

***********************************************************************

Connecting Grafana to Loki

***********************************************************************

If Grafana operates within the cluster, you'll set up a new Loki datasource by utilizing the following URL:

http://loki-gateway.grafana-loki.svc.cluster.local/

***********************************************************************

Multi-tenancy

***********************************************************************

Loki is configured with auth enabled (multi-tenancy) and expects tenant headers (`X-Scope-OrgID`) to be set for all API calls.

You must configure Grafana's Loki datasource using the `HTTP Headers` section with the `X-Scope-OrgID` to target a specific tenant.

For each tenant, you can create a different datasource.

The agent of your choice must also be configured to propagate this header.

For example, when using Promtail you can use the `tenant` stage. https://grafana.com/docs/loki/latest/send-data/promtail/stages/tenant/

When not provided with the `X-Scope-OrgID` while auth is enabled, Loki will reject reads and writes with a 404 status code `no org id`.

You can also use a reverse proxy, to automatically add the `X-Scope-OrgID` header as suggested by https://grafana.com/docs/loki/latest/operations/authentication/

For more information, read our documentation about multi-tenancy: https://grafana.com/docs/loki/latest/operations/multi-tenancy/

> When using curl you can pass `X-Scope-OrgId` header using `-H X-Scope-OrgId:foo` option, where foo can be replaced with the tenant of your choice.

EOT -> (known after apply)

 

---

Grafana Observability Stack

 

Grafana uses these components together as an observability stack, but each has a clear role:

Loki – log database. It stores and indexes logs (especially from Kubernetes) in a cost‑efficient, label‑based way, similar to Prometheus but for logs.

Tempo – distributed tracing backend. It stores distributed traces (spans) from OpenTelemetry, Jaeger, Zipkin, etc., so you can see call flows across microservices and where latency comes from.

Mimir – Prometheus‑compatible metrics backend. It is a horizontally scalable, long‑term storage and query engine for Prometheus‑style metrics (time series).

Alloy – telemetry pipeline (collector). It is Grafana’s distribution of the OpenTelemetry Collector / Prometheus agent / Promtail ideas, used to collect, process, and forward metrics, logs, traces, profiles into Loki/Tempo/Mimir (or other backends).

How Grafana UI relates to them

Grafana UI itself is “just” the visualization and alerting layer:

• It connects to Loki, Tempo, Mimir (and many others) as data sources.
• For each backend you configure:
• A Loki data source for logs.
• A Tempo data source for traces.
• A Prometheus/Mimir data source for metrics (Mimir exposes a Prometheus‑compatible API).
• Grafana then lets you:
• Build dashboards and alerts from Mimir metrics.
• Explore logs from Loki.
• Explore traces from Tempo and cross‑link them with logs/metrics (e.g., click from a log line to a trace, or from a metrics graph into logs/traces).

A useful mental model: Loki/Tempo/Mimir are databases, Alloy is the collector/router, and Grafana is the UI on top.

Are they deployed in the same Kubernetes cluster?

Common patterns:

• Very common: deploy Loki, Tempo, Mimir, Alloy, and Grafana in the same Kubernetes cluster as your apps. This is the typical “in‑cluster LGTM” setup; all telemetry stays inside the cluster and traffic is simple.
• Also common: run them in a separate observability cluster (or use Grafana Cloud backends), while Alloy/agents run in each workload cluster and ship data over the network. This improves isolation and makes it easier to share one observability stack across many clusters.
• In smaller setups or dev environments, everything (apps + LGTM + Grafana) often lives in one cluster; in larger/regulated setups, people tend to separate “workload clusters” and an “observability cluster”.

So: they don’t have to be on the same cluster, but it’s perfectly normal (and often simplest) to run Grafana + Loki + Tempo + Mimir + Alloy together in a single Kubernetes cluster and point your apps’ telemetry to Alloy.

Why not using elasticsearch instead of loki, tempo and mimir?

Elasticsearch can replace part of what Loki, Tempo, and Mimir do, but not all of it, and usually with higher cost/complexity for cloud‑native observability.

1. Scope: logs vs full observability

Elasticsearch is a general search and analytics engine that’s great at full‑text search, aggregations, and analytics over documents (including logs).

The LGTM stack is explicitly split by signal:

• Loki → logs
• Tempo → traces
• Mimir → metrics

Each is optimized only for its signal type and integrates tightly with Grafana and modern telemetry standards.

You could plausibly replace Loki with Elasticsearch for logs, but Elasticsearch does not natively replace Tempo (distributed tracing backend) or Mimir (Prometheus‑compatible metrics backend).

2. Logs: Loki vs Elasticsearch

Elasticsearch strengths:

• Very powerful full‑text search, fuzzy matching, relevance scoring, complex aggregations.
• Good when you need deep forensic search and advanced analytics on log text.

Loki strengths:

• Stores logs as compressed chunks plus a small label index, so storage and compute are much cheaper than Elasticsearch for typical Kubernetes logs.
• Very tight integration with Grafana and the rest of LGTM, and simple, label‑based querying.

Trade‑off: Elasticsearch gives richer search at a high infra + ops cost, Loki gives “good enough” search for operational troubleshooting with much lower cost and operational burden.

3. Traces and metrics: Tempo & Mimir vs “just ES”

Tempo:

• Implements distributed tracing concepts (spans, traces, service graphs) and OpenTelemetry/Jaeger/Zipkin protocols; the data model and APIs are specialized for traces.
• Elasticsearch can store trace‑like JSON documents, but you’d have to build/maintain all the trace stitching, UI navigation, and integrations yourself.

Mimir:

• Is a horizontally scalable, Prometheus‑compatible time‑series database, with native remote‑write/read and PromQL semantics.
• Elasticsearch can store time‑stamped metrics, but you lose Prometheus compatibility, PromQL semantics, and the whole ecosystem that expects a Prometheus‑style API.

So using only Elasticsearch means you’re giving up the standard metrics and tracing ecosystems and rebuilding a lot of tooling on top of a generic search engine.

​

4. Cost, complexity, and operational burden

Elasticsearch clusters generally need:

• More RAM/CPU per node, careful shard and index management, and capacity planning.
• Storage overhead from full‑text indexes (often 1.5–3× raw log size plus replicas).

​

Loki/Tempo/Mimir:

• Are designed for object storage, compression, and label‑only indexing, which dramatically lowers storage and compute requirements for logs and metrics.
• Have simpler, well‑documented reference architectures specifically for observability.

For a modern Kubernetes‑centric environment, that usually makes LGTM cheaper and easier to run than a single big Elasticsearch cluster for everything.

5. When Elasticsearch still makes sense

You might still choose Elasticsearch (often with Kibana/APM) if:

• You already have a strong ELK stack and team expertise.
• Your primary need is deep, flexible text search and analytics over logs, with less emphasis on Prometheus/OTel ecosystems.
• You want Elasticsearch’s ML/anomaly‑detection features and are willing to pay the operational cost.

But if your goal is a Grafana‑centric, standards‑based (Prometheus + OpenTelemetry) observability platform, LGTM (Loki+Tempo+Mimir, plus Alloy as collector) is a better fit than trying to push everything into Elasticsearch.

---

Here document (heredoc)

Here document (heredoc) redirects a multiline string literal to the preceding command while preserving line breaks. Unix syntax for it is:

[command] <<DELIMITER

    First line.

    Second line.

    Third line.

    Fourth line.

DELIMITER

<< is Redirection Operator

- is optional Tab Suppression

DELIMITER - an arbitrary string, Delimiter Token; must be the same at the beginning and at the end

Appending a minus sign to the redirection operator <<- causes all leading tab characters to be ignored. This allows you to use indentation when writing heredocs in shell scripts. We can then indent both the here-doc and the delimiter with tabs (not spaces!):

#! /bin/bash

cat <<-EOF

    indented

    EOF

echo Done

---

References:

Here document - Wikipedia

How to fix pods in Not Ready state?

kubectl get pods might show that some of the pods have 0/N value in READY column.

What is the meaning of READY column value?

In the context of kubectl get pods, the READY column shows the number of containers in the pod that have passed their health checks and are ready to serve traffic.

The anatomy of R/T:

• R (Left side): This is the number of containers currently Ready. A 0 means the application inside the container is not responding to its "Readiness Probe" or has not finished starting up.
• / (Separator): Separates ready containers from the total.
• T (Right side): This is the Total number of user containers defined in that pod.

When we have a pod in 0/1 state with a status of Running, it means the container has started, but Kubernetes does not consider it "healthy" enough to handle requests. Our pods are technically "alive" (Running) but "unusable" (Not Ready).

Common Reasons for 0/1 Running:

• Failed Readiness Probe: The application is running, but the health check URL (e.g., /ready) is returning an error or timing out.
• Slow Startup: The application takes a long time to initialize, and the "Initial Delay" isn't long enough.
• Dependency Issues: The pod is waiting for a database, a config file, or another service that isn't available.
• CrashLoopBackOff (Transitions): Sometimes pods flicker between 0/1 Running and 0/1 CrashLoopBackOff as they try to start and immediately fail.

How to find out exactly what's wrong

To diagnose why our my-app pods are stuck, run these two commands:

(1) Check the Events (Why it's not ready):

kubectl describe pod my-app-0 -n my-app-namespace

Look at the "Events" section at the bottom for messages like "Readiness probe failed."

Output example:

Events:

  Type     Reason     Age                   From     Message

  ----     ------     ----                  ----     -------

  Warning  Unhealthy  47s (x81205 over 8d)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 503

(2) Check the Logs (What the app is saying):

kubectl logs my-app-0 -n my-app-namespace

Look for "Error," "Exception," or "Connection Refused" messages.

---

How to use terraform-docs automatically generate Terraform code documentation

 

terraform-docs is a tool used to automatically generate Terraform code documentation.

To install it on Mac:

% brew install terraform-docs 

To verify installation:

% terraform-docs --version                                        

terraform-docs version v0.21.0 darwin/arm64

To generate a documentation for a module in the current directory and append it to the README file (which is in the same directory):

% terraform-docs markdown table --output-file README.md --output-mode inject ./

How to install Terraform on Mac

First add Hashicorp's package repository:

% brew tap hashicorp/tap

Then install the Terraform:

% brew install hashicorp/tap/terraform

If Terraform was already installed, the command above will update it.

To verify installation, we can check its version:

% terraform --version                                                                                    

Terraform v1.14.5

on darwin_arm64

Amazon EKS Autoscaling with Karpenter

Kubernetes autoscaling is a function that scales resources in and out depending on the current workload. AWS supports two autoscaling implementations:

• Cluster Autoscaler
• autoscaler/cluster-autoscaler/cloudprovider/aws/README.md at master · kubernetes/autoscaler
• automatically adjusts the number of nodes in the cluster when pods fail or are rescheduled onto other nodes
• uses Auto Scaling groups
• Karpenter 
• Karpenter
• flexible, high-performance Kubernetes cluster autoscaler
• helps improve application availability and cluster efficiency
• launches right-sized compute resources (for example, Amazon EC2 instances) in response to changing application load in under a minute
• can provision just-in-time compute resources that precisely meet the requirements of your workload
• automatically provisions new compute resources based on the specific requirements of cluster workloads. These include compute, storage, acceleration, and scheduling requirements. 
• creates Kubernetes nodes directly from EC2 instances
• improves the efficiency and cost of running workloads on the cluster
• open-source

Pod Scheduler

• Kubernetes cluster component responsible for determining which node Pods get assigned to
• default Pod scheduler for Kubernetes is kube-scheduler
• logs the reasons Pods can't be scheduled

Unschedulable Pods

From How to troubleshoot unschedulable Pods in Kubernetes:

A Pod is unschedulable when it's been put into Kubernetes' scheduling queue, but can't be deployed to a node. This can be for a number of reasons, including:

• The cluster not having enough CPU or RAM available to meet the Pod's requirements.
• Pod affinity or anti-affinity rules preventing it from being deployed to available nodes.
• Nodes being cordoned due to updates or restarts.
• The Pod requiring a persistent volume that's unavailable, or bound to an unavailable node.

How to detect unschedulable Pods?

Pods waiting to be scheduled are held in the "Pending" status, but if the Pod can't be scheduled, it will remain in this state. However, Pods that are being deployed normally are also marked as "Pending." The difference comes down to how long a Pod remains in "Pending." 

How to  fix unschedulable Pods? 

There is no single solution for unschedulable Pods as they have many different causes. However, there are a few things you can try depending on the cause. 

• Enable cluster autoscaling
• If you're using a managed Kubernetes service like Amazon EKS or Google Kubernetes Engine (GKE), you can very easily take advantage of autoscaling to increase and decrease cluster capacity on-demand. With autoscaling enabled, Kubernetes' Cluster Autoscaler will trigger your provider to add nodes when needed. As long as you've configured your cluster node pool and it hasn't reached its max node limit, your provider will automatically provision a new node and add it to the pool, making it available to the cluster and to your Pods.
• Increase your node capacity
• Check your Pod requests
• Check your affinity and anti-affinity rules 

 

In this article we'll show how to enable cluster autoscaling with Karpenter.

How does the regular Kubernetes Autoscaler work in AWS?

When we create a regular Kubernetes cluster in AWS, each node group is managed by the AWS Auto-scaling group [Auto Scaling groups - Amazon EC2 Auto Scaling]. Cluster native autoscaler adjusts the desired size based on the load in the cluster to fit all unscheduled pods.

HorizontalPodAutoscaler (HPA) [Horizontal Pod Autoscaling | Kubernetes] is built into Kubernetes and it uses metrics like CPU usage, memory usage or custom metrics we can write to decide when to spin up or down additional pods in the node of the cluster. If our app is receiving more traffic, HPA will kick in and provision additional pods. 

VerticalPodAutoscaler (VPA) can also be installed in cluster where it manages the resource (like CPU and memory) allocation to pods that are already running.

What about when there's not enough capacity to schedule any more pods in the node? That's when we'll need an additional node. So we have a pod that needs to be scheduled but we don't know where to put it. We could call AWS API, spin up an additional EC2 node, get added it to our cluster or if we're using managed groups we can use Managed Node Group API, bump up the desired size but easier approach is to use cluster auto-scaler. There is a mature open-source solution called Cluster Auto-Scaler (CAS).

CAS was built to handle hundreds of different comninations of nodes types, zones, purchase options available in AWS. CAS works directly with managed node groups or self-managed managed nodes and auto-scaling groups which are AWS constructs to help us manage nodes. 

What are the issues with the regular Kubernetes Autoscaler?

Let's say CAS is installed on node, in cluster and manages one managed node group (MNG). It's filling up and we have an additional pod that needs to be provisioned so CAS tells MNG to bump up the number of nodes so it spins up another one so pod can now be scheduled. But this is not ideal. We have a single pod in a node, we don't need such a big node. 

This can be solved by creating a different MNG with a smaller instance type and now CAS recognizes that instance and provisions pod on a more appropriately-sized node.

Unfortunately, we might end up with many MNGs, based on requirements which might be a challenge to manage especially when looking best practices in terms of cost efficiency and high availability. 

How does Karpenter work?

Karpenter works differently, It doesn't use MNG or ASGs and manages each node directly. Let's say we have different pods, of different sizes. Let's say that HPA says that we need more of the smaller pods. Karpenter will intelligently pick the right instance type for that workload. If we need to spin up a larger pod it will again pick the right instance type. 

Karpenter picks exactly the right type of node for our workload. 

If we're using spot instances and spot capacity is not available, Karpenter does retries more quickly. Karpenter offers, faster, dynamic, more intelligent compute, using best practices without operational overhead of managing nodes ourselves. 

How to control how Karpenter operates?

There are many dimensions here. We can set constraints on Karpenter to limit the instances type, we can set up taints to isolate workloads to specific types of nodes. Different teams can have isolated access to different pods, one team can access billing pods, another GPU-based instances. 

Workload Consolidation feature: Pods are consolidated into fewer nodes.. let's say we have 3 nodes, two at 70% and one at 20% utilization. Karpenter detects this and will move pods from underutilized node to those two and shut down this now empty node (instances are terminated). This leads to lower costs.

Karpenter is making it easier to use spot and graviton instances which can also lead to lower costs. 

A feature to keep our nodes up to date. ttlSecondsUntilExpired parameter tells Karpenter to terminate nodes after a set amount of time. These nodes will automatically be replaced with new nodes, running the latest AMIs.

Karpenter:

1) lower costs

2) higher application availability 

3) lower operation overhead

Karpenter needs permissions to create EC2 instances in AWS. 

If we use a self-hosted (on bare metal boxes or EC2 instances), self-managed (you have full control over all aspects of Kubernetes) Kubernetes cluster, for example by using kOps (see also Is k8s Kops preferable than eks? : r/kubernetes), we can add additional IAM policies to the existing IAM role attached to Kubernetes nodes. 

If using EKS, the best way to grant access to internal service is with IAM roles for service accounts (IRSA).

How to configure Karpenter?

We can configure specific Karpenter NodePools or Provisioners.

How to know if node was provisioned by Karpenter?

Karpenter applies labels on nodes it provisions so let's check labels:

% kubectl get nodes --show-labels

If labels like karpenter.sh/nodepool or karpenter.sh/provisioner-name exist, Karpenter launched the node.

References:

Autoscaling - Amazon EKS

Karpenter for Kubernetes | Karpenter vs Cluster Autoscaler - YouTube

Cluster-Autoscaler - EKS Best Practices Guides

Karpenter Vs Cluster Autoscaler: The Essential Guide