elasticsearch-exporter

Elasticsearch doesn't speak Prometheus. It exposes its internals over its own REST API — GET /_nodes/stats, /_cluster/health, /_cat/indices — as JSON, in Elastic's own shape. Prometheus can't scrape that.

The elasticsearch-exporter is a small sidecar-or-Deployment-shaped translator that sits between the two:

• it polls those ES REST endpoints on an interval,
• flattens the JSON into Prometheus text-format metrics,
• and serves them on /metrics (conventionally :9114) for Prometheus to scrape.

The canonical implementation is prometheus-community/elasticsearch_exporter (formerly justwatchcom/elasticsearch_exporter), packaged as the prometheus-elasticsearch-exporter Helm chart. Elastic also ships a first-party alternative path — Metricbeat's elasticsearch module, or the newer Elastic Agent integration — but those ship into Elasticsearch/Kibana's own monitoring cluster, not into Prometheus, so they don't help a Grafana-alerting-on-Prometheus setup.

The metrics it produces are for example:

• metric
• what it gives you 

• search_jvm_memory_used_bytes{area="heap"}
• actual JVM heap in use — the thing that actually predicts an ES OOM
• elasticsearch_jvm_memory_max_bytes{area="heap"}
• configured heap ceiling (-Xmx), so you can take a real ratio
• elasticsearch_jvm_gc_collection_seconds_*
• GC pressure; sustained old-gen GC is the pre-OOM tell
• elasticsearch_breakers_tripped
• circuit breakers firing — ES rejecting work to avoid OOM
• elasticsearch_cluster_health_status
• green/yellow/red, unassigned shards                                 │

The exporter is a separate deployable — ECK does not install it.

Prometheus Metrics

Container Metrics

container_memory_rss

container_memory_rss is a Prometheus metric (exposed by cAdvisor) that measures a container's Resident Set Size—specifically, the amount of physical RAM allocated to non-reclaimable, non-file-backed memory.

Key Technical Breakdown

Under the hood in Linux cgroups, container_memory_rss tracks:

• Anonymous Memory: Process heap allocations, execution stack, and memory allocated via malloc or mmap(MAP_ANONYMOUS).  
• Swap Cache: Memory swapped out to disk that is being brought back into RAM.

What it EXCLUDES:

Unlike standard Linux host RSS metrics, container_memory_rss in cAdvisor excludes file-backed page caches (memory used to cache files read from disk). Because of this, it only measures memory that cannot be automatically freed by the Linux kernel under memory pressure.  

container_memory_rss vs. Other Container Metrics

To understand its role, it helps to see how it fits into cAdvisor's other core memory metrics:

Metric

• Includes
• Purpose / Key Characteristic

container_memory_rss

• Heap + Stack (Anonymous memory)
• Stable indicator of process memory footprint. Does not fluctuate with disk I/O.

container_memory_working_set_bytes

• Heap + Stack + Active Page Cache
• What Kubernetes actually monitors. Fluctuates with file reads/writes.

container_memory_usage_bytes

• Heap + Stack + Active Cache + Inactive Cache
• Raw total RAM usage. Can be misleading because inactive cache is easily reclaimed by the OS.

Why is container_memory_rss Important?

• Memory Leak Detection: Because it excludes cached file reads, container_memory_rss provides a much cleaner signal for application-level memory leaks. If this metric steadily climbs over time without dropping, your application (e.g., Go heap, JVM heap, Node process) is holding onto unmanaged memory. 
• Debugging OOM Kills: While Kubernetes triggers OOMKills based on container_memory_working_set_bytes hitting resource limits, container_memory_rss helps you determine why it happened:  
• High RSS + High Working Set --> Application memory leak or underestimated heap limit.
• Low RSS + High Working Set --> Heavy disk I/O / file caching (e.g., reading massive log files or database indexes into memory).  

When container_memory_rss is NOT a relevant metric?

Example:

WiredTiger is the default storage engine that MongoDB (here, the Percona Server for MongoDB / PSMDB cluster) uses to actually read and write data to disk. In <ticketID> it matters specifically because of how it uses memory, which is what breaks those four Grafana alert rules.

The relevant behavior:

WiredTiger keeps a large in-memory cache. It maintains its own cache of frequently-accessed data and indexes, and it deliberately sizes that cache to roughly half the container's memory limit (the issue cites ~10 GiB against a 21Gi limit on rs0/rs2, and it's tunable per <ticketID>). WiredTiger runs its own eviction, targeting about 80% cache fill under normal operation and triggering aggressive eviction around 95%.

That cached data shows up as page cache, not RSS. This is the crux of the ticket. container_memory_rss counts anonymous/resident process memory but excludes the OS page cache — and for a WiredTiger workload the file-backed pages (the cache) are the bulk of the real footprint. So rss / limit sits structurally pinned around 50-58% no matter how much memory pressure the pod is actually under. An alert keyed on container_memory_rss > 0.90 can therefore never fire for a WiredTiger container. It's dead. (alert is defined as: container_memory_rss{container="mongod"} / limit > 0.90)

But you can't just switch to working-set either. container_memory_working_set_bytes does include those active file pages, so on the busy replicas it reads 95-96%. That looks alarming but is expected and healthy: it's the WiredTiger cache sitting at its designed ~half-of-limit size and WT's own eviction watermark. The kernel reclaims those pages under real pressure, and the evidence is that nothing in the mongodb namespace has ever been OOMKilled. Flipping the metric would just move the rule from never-firing to always-firing.

The signal that actually matters is WiredTiger cache fill. Because WT stalls user operations when its cache can't evict fast enough, the meaningful early-warning metric is cache utilization (the existing MongoDB WiredTiger cache fill — above 95% for 30m rule from <ticketID>), not container RSS or working set. That's why the ticket argues the RSS rules are redundant, not just broken, and leans toward deleting them.

So in this issue's context, "WiredTiger" is essentially shorthand for "a workload whose memory lives mostly in a large, self-managed, page-cache-backed database cache" — which is exactly the profile that makes RSS-based memory alerting meaningless and makes cache-fill the correct signal instead. (The same logic applies to the Elasticsearch rules, where the JVM heap plays the analogous role.)

Example PromQL Query

To track application heap growth without the noise of filesystem caching, query RSS like this:

# Rate of container RSS growth over 5-minute intervals

rate(container_memory_rss{namespace="production", container!=""}[5m])

container_memory_working_set_bytes

...

container_memory_usage_bytes

...

Introduction to cAdvisor (Container Advisor)

cAdvisor (short for Container Advisor) is an open-source tool created by Google to collect, aggregate, process, and export resource usage and performance metrics for running containers.

It acts as a daemon that monitors resource isolation parameters, historical resource usage, and network statistics directly from the host node.

How It Works

cAdvisor doesn't require instrumenting code inside containers. Instead, it inspects the node environment where containers run:

• Queries Linux Kernel Structures: It pulls raw performance data directly from kernel mechanisms—primarily cgroups (control groups) for CPU, memory, and disk utilization, and network interfaces for throughput metrics.
• Discovers Running Containers: It automatically detects running containers across multiple runtimes (Docker, containerd, CRI-O, systemd containers).
• Exposes Metrics: It formats gathered data and exposes it over a /metrics HTTP endpoint (primarily in Prometheus format) for scrapers to ingest.

Role in Kubernetes

In Kubernetes, cAdvisor is not deployed as a standalone pod. Instead, it is built directly into the kubelet binary that runs on every node.

• The kubelet uses cAdvisor internally to monitor local container resource usage.
• It exposes cAdvisor metrics under the /metrics/cadvisor endpoint on the kubelet API port (typically 10250).
• Tools like Prometheus scrape this endpoint to collect system-wide container metrics (container_cpu_usage_seconds_total, container_memory_rss, container_network_transmit_bytes_total, etc.).

Core Capabilities

• Resource Usage Monitoring: Tracks real-time CPU utilization, memory breakdown (RSS, cache, swap), network I/O, and disk space usage per container.
• Historical Trend Aggregation: Keeps a small buffer of historical telemetry in memory for local inspection.
• Multi-Runtime Support: Works out of the box with containerd, Docker, and CRI-compliant container engines.
• Built-in Web UI: When run as a standalone binary or Docker container outside Kubernetes, it provides a lightweight built-in dashboard for quick visual inspection of container stats.

Summary of Metric Flow

Linux Kernel / cgroups --> cAdvisor --> Prometheus --> Grafana

cAdvisor sits right at the boundary between the underlying host/kernel layer and the monitoring stack, turning low-level kernel counters into structured metrics for alerting and visualization.

Kubernetes Job object

 

A Kubernetes Job is a controller object designed to run a batch task to completion.

Unlike Deployments or ReplicaSets (which keep applications running indefinitely) or CronJobs (which trigger tasks on a schedule), a Job creates one or more Pods, executes the workload, and ensures they terminate cleanly. Once the specified number of Pods complete successfully, the Job itself is marked as complete and stops.

Standard Job Manifest

apiVersion: batch/v1

kind: Job

metadata:

  name: data-migration-job

spec:

  backoffLimit: 4             # Number of retries before marking the job failed

  completions: 1              # Number of successful pod completions required

  parallelism: 1              # How many pods run concurrently

  ttlSecondsAfterFinished: 600 # Clean up job & pods 10 minutes after completion

  template:

    spec:

      containers:

      - name: migration-task

        image: python:3.11-slim

        command: ["python", "-c", "print('Running database migration...'); import time; time.sleep(10); print('Done!')"]

      restartPolicy: OnFailure # Required: OnFailure or Never (Always is invalid)

Core Execution Patterns

Kubernetes Jobs support three primary workload execution models:

• 1. Non-Parallel Jobs
• Behavior: Starts a single Pod and waits for it to complete successfully.
• Use Case: One-off database schema migrations, report generation, or administrative scripts.
• 2. Parallel Jobs with Fixed Completions
• Behavior: Runs multiple Pods in parallel until a total number of successful completions (spec.completions) is reached.
• Use Case: Batch processing where $N$ independent tasks need to be completed.
• 3. Parallel Jobs with Work Queue
• Behavior: Pods coordinate via an external message queue (e.g., RabbitMQ, Redis, SQS). Each Pod pulls work until the queue is empty, then exits.
• Use Case: High-throughput task processing, media transcoding, or distributed data transformation.

Key Configuration Fields

Field

• Default
• Description

restartPolicy

• Required
• Must be OnFailure (restarts container inside same Pod) or Never (spawns a new Pod on failure).

backoffLimit

• 6
• Maximum number of retries before marking the Job as failed.

completions

• 1
• Total number of successful Pod terminations needed for Job completion.

parallelism

• 1
• Max number of Pods allowed to run concurrently at any given moment.

activeDeadlineSeconds

• Unlimited
• Max time allowed for the entire Job (including retries) before terminating all running Pods.

completionMode

• NonIndexed
• Set to Indexed to assign each Pod a unique completion index ($0$ to $\text{completions}-1$) via environment variables.

ttlSecondsAfterFinished

• Disabled
• Automatically deletes the Job and its underlying Pods after $N$ seconds of finishing.

Essential kubectl Commands

Operation                                 Command

==================         ===========

Create a Job imperatively         kubectl create job my-job --image=busybox -- echo "Hello World"

Get Job status                            kubectl get jobs

Inspect job details                     kubectl describe job my-job

List Pods associated with Job   kubectl get pods --selector=batch.kubernetes.io/job-name=my-job

View logs of Job Pods              kubectl logs job/my-job

Delete Job and its Pods            kubectl delete job my-job

Jobs vs Deployments vs CronJobs        

                    ┌─────────────────────────┐

                    │      Workload Type      │

                    └────────────┬────────────┘

                                 │

           ┌─────────────────────┴─────────────────────┐

           ▼                                           ▼

   Long-Running Services                     Batch Tasks / One-Off

(Deployments, StatefulSets)                      (Jobs & CronJobs)

           │                                           │

  Maintains target Pod                       Executes task, then

  count indefinitely.                        terminates cleanly.

                                                       │

                                      ┌────────────────┴────────────────┐

                                      ▼                                 ▼

                                Single Run                         Scheduled Run

                                  (Job)                              (CronJob)

Kubernetes CronJob

A Kubernetes CronJob creates and manages short-lived Jobs on a scheduled, repeating basis. It is the Kubernetes equivalent of a standard Unix crontab file, making it ideal for periodic tasks like database backups, report generation, or maintenance scripts.

Minimal Example Manifest

apiVersion: batch/v1

kind: CronJob

metadata:

  name: nightly-backup

spec:

  schedule: "0 2 * * *" # Runs every day at 02:00 UTC

  timeZone: "Etc/UTC"   # Optional: set preferred timezone (Kubernetes 1.27+)

  concurrencyPolicy: Forbid

  startingDeadlineSeconds: 100

  successfulJobsHistoryLimit: 3

  failedJobsHistoryLimit: 1

  jobTemplate:

    spec:

      template:

        spec:

          containers:

          - name: backup-task

            image: alpine:latest

            command:

            - /bin/sh

            - -c

            - echo "Running database backup..."; sleep 5

          restartPolicy: OnFailure

Schedule Syntax Quick Reference

The schedule field uses standard cron syntax with 5 fields:

minute hour day-of-month month day-of-week

Schedule Format Interpretation

*    * * * *             Every minute

*/15 * * * *              Every 15 minutes

0    0 * * *              Every day at midnight

0    9 * * 1              Every Monday at 9:00 AM

Critical Settings

• concurrencyPolicy: Controls how overlapping executions are handled when a previous run hasn't finished:
• Allow (default): Runs concurrent jobs simultaneously. 
• Forbid: Skips the new job if the previous one is still running. 
• Replace: Cancels the currently running job and starts the new one. 
• startingDeadlineSeconds: The deadline (in seconds) for starting a job if it missed its scheduled time (e.g., cluster was temporarily down).
• successfulJobsHistoryLimit / failedJobsHistoryLimit: Number of completed or failed Job/Pod records to keep for auditing before automatic cleanup.
• restartPolicy: Must be set on the pod template spec to either OnFailure or Never (Always is invalid for Jobs).  

Helpful kubectl Commands

Task                                            Command 

====                                           ========

List CronJobs                              kubectl get cronjobs

Inspect configuration                  kubectl describe cronjob <name>

Manually trigger immediately    kubectl create job --from=cronjob/<cronjob-name>                        <manual-job-name>

Pause schedule                           kubectl patch cronjob <name> -p '{"spec":                             {"suspend":true}}'

View logs of latest run               kubectl logs job/<job-name>

CronJobs Inner Mechanism

Under the hood, Kubernetes CronJobs rely on a decentralized control loop pattern. They are not handled by a traditional Linux cron daemon running on a single server, but rather by the Kubernetes Control Plane through cascading controllers.

How CronJobs Are Implemented

The implementation follows a 3-tier hierarchical model:

CronJob Object --> Job Object --> Pod(s)

Rather than running code directly, a CronJob acts as a factory for Job objects, which in turn manage the Pods where your container actually executes. 

┌─────────────────────────────────────────────────────────┐

│                 kube-controller-manager                 │

│                                                         │

│   ┌───────────────────┐       Creates      ┌─────────┐  │

│   │ CronJob Controller│ ─────────────────> │   Job   │  │

│   └───────────────────┘                    └─────┬───┘  │

└──────────────────────────────────────────────────┼──────┘

                                                   │

                                                Creates

                                                   │

                                                   ▼

                                              ┌─────────┐

                                              │   Pod   │

                                              └─────────┘

The Control Loop Mechanism

1. Synchronization Loop: The CronJob Controller runs inside kube-controller-manager. Every ~10 seconds, it iterates through all CronJob objects defined in the cluster.  
2. Schedule Checking: The controller parses the schedule field (e.g., 0 * * * *) and compares the current time against the last time the job was executed.  
3. Job Spawning: If a run is due, the CronJob controller reads the embedded jobTemplate and creates an actual Job resource.  
4. Execution: The cluster's separate Job Controller detects the newly created Job resource and spawns one or more Pods to execute your container workload to completion.  
5. Garbage Collection: Depending on successfulJobsHistoryLimit and failedJobsHistoryLimit, the CronJob controller periodically deletes old completed Job objects (and their associated logs/pods).  

Who Controls Them?

Control over CronJobs is split between system components (automation) and users/roles (permissions).

System Component Control

• kube-controller-manager: The core control plane component where the CronJob controller code actually executes. If this component is down, scheduled triggers will pause until it recovers.  
• kube-apiserver: Stores the desired state in etcd and validates user manifests.
• kube-scheduler: Assigns the individual Pods spawned by the resulting Jobs to healthy worker nodes.

User & Permission Control (RBAC)

Human administrators and automated service accounts control CronJobs via Kubernetes Role-Based Access Control (RBAC):

Role / Action              Required API Permissions (batch/v1)

=============     ==============================

Manage Schedules     create, update, patch, delete on cronjobs

View Status                get, list, watch on cronjobs

Manual Trigger          create permissions on jobs (to invoke kubectl create job --from=cronjob/...)

Example RBAC Role for CronJob Operators:

apiVersion: rbac.authorization.k8s.io/v1

kind: Role

metadata:

  namespace: prod

  name: cronjob-operator

rules:

- apiGroups: ["batch"]

  resources: ["cronjobs", "jobs"]

  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Technical Considerations

• At-Least-Once Execution: Kubernetes schedules are designed around at-least-once execution semantics. Due to control loop timing or network hiccups, a scheduled job might occasionally run twice or run slightly late. Workloads should always be designed to be idempotent. 
• Timezones: Controller clocks default to UTC or the local time of kube-controller-manager unless explicit timezones are passed via spec.timeZone (supported in K8s 1.27+). 

CronJob is a controller object

In Kubernetes, a controller is a control loop that watches the state of your cluster through the API server and makes changes attempting to move the current state toward the desired state.

Here is how a CronJob fits into the controller pattern:

Why CronJob is a Controller

• Custom Resource / Spec & Status Model: Like Deployment, ReplicaSet, and Job, a CronJob has an API object schema (spec defining desired behavior, status tracking execution state).
• Control Loop Execution: The CronJob implementation runs as a control loop inside the kube-controller-manager component.
• Cascading Controller Pattern: CronJob sits at the top of a controller hierarchy:

CronJob Controller --[creates/manages]--> Job Controller --[creates/manages]--> Pods

• The CronJob Controller reconciles the CronJob spec: it checks the schedule, creates Job resources when a execution is due, cleans up old jobs based on history limits, and handles concurrency policies.
• The Job Controller reconciles those created Job resources to manage individual Pods to completion.

Summary Table

Controller                        API Group      What it Watches      What it Creates/Manages

========                       =========     =============      ===================

CronJob Controller            batch/v1          CronJob specs            Job objects

Job Controller                    batch/v1          Job specs                    Pod objects

Deployment Controller      apps/v1           Deployment specs      ReplicaSet objects

Grafana Alerting

 

How it works

• Grafana alerting periodically queries data sources and evaluates the condition defined in the alert rule
• If the condition is breached, an alert instance fires
• Firing instances are routed to notification policies based on matching labels
• Notifications are sent out to the contact points specified in the notification policy

How to set alerts

• Alert rules: Create an alert rule to query a data source and evaluate the condition defined in the alert rule. 
• There are two types of alerts rules:
• Grafana-managed. Examples:
• APM
• AWS
• Data Quality
• Kubernetes
• MongoDB
• Storage
• Synthetics
• Data source-managed. Data sources containing configured alerts rules are for example Mimir or Loki data sources where alert rules are stored and evaluated in the data sources itself. In these data sources you can select Manage alerts via Alerting UI to be able to manage these alerts rules in the Grafana UI as well as in the data source where they were configured.
• Prometheus
• Mimir
• Loki
• Define the condition that must be met before an alert rule fires
• Route alert notifications either directly to a contact point or through notification policies for more flexibility
• Contact points: Configure who receives notifications and how they are sent
• Notification policies: Configure how firing alert instances are routed to contact points
• Monitor your alert rules using dashboards and visualizations

Kubernetes Custom Resource Definition (CRD)

Why are Custom Resource Definitions (CRDs) stored in etcd?

Custom Resource Definitions (CRDs) are stored in etcd because etcd serves as the single source of truth and persistent data store for the entire Kubernetes control plane. Storing CRDs in etcd allows the kube-apiserver to dynamically register, validate, and manage custom resources exactly like native objects without requiring a separate database.

Why Kubernetes Stores CRDs in etcd

• Native API Integration: The kube-apiserver inherently reads from and writes to etcd. Storing CRDs here allows custom objects to instantly leverage native features like kubectl support, RBAC security, and namespaces.
• Consistent State: Like Pods or Deployments, custom resources represent the "desired state" of a system. etcd provides the strong consistency and distributed consensus needed to safely store this state across cluster nodes.
• Watch Mechanism: Custom controllers and operators rely on the Kubernetes watch API to listen for resource changes. Because etcd natively supports watch events, controllers can instantly react when a custom resource is created, updated, or deleted.

How custom controllers interact with the stored CRD data using the reconciliation loop?

Custom controllers interact with etcd through a continuous synchronization mechanism called the reconciliation loop. This loop constantly drives the actual state of the cluster toward your desired state.

Here is exactly how custom controllers interact with stored CRD data:

1. The Relationship to CRDs

A CRD (Custom Resource Definition) acts as the database schema. It defines the structure and validation rules for your custom data. When you create a CR (Custom Resource) based on that CRD, its data is stored directly in etcd.

The custom controller is the active software component (the brain) that watches that specific CR data and takes action based on what is stored.

2. The Reconciliation Loop Step-by-Step

The controller executes a continuous loop, often formalized as a Reconcile() function, which follows a Observe, Analyze, Act workflow:   

    +--------------------------------------------+
    |                                            |
    |               1. OBSERVE                   |
    |     (Watch etcd via API Server)            |
    |                                            |
    +---------------------+----------------------+
                          |
                          v
    +---------------------+----------------------+
    |                                            |
    |               2. ANALYZE                   |
    |    (Compare Desired vs. Actual State)      |
    |                                            |
    +---------------------+----------------------+
                          |
                          v
    +---------------------+----------------------+
    |                                            |
    |                 3. ACT                     |
    |     (Create/Update/Delete Infrastructure)  |
    |                                            |
    +--------------------------------------------+

• Step 1: Observe (The Watch Mechanism): Instead of constantly polling etcd, the controller establishes a long-lived HTTP connection (a watch) to the kube-apiserver. When a user creates, updates, or deletes a CR in etcd, the API server instantly broadcasts an event. The controller captures this event and queues the resource's key for processing.
• Step 2: Analyze (Diffing the State): The controller fetches the full CR data from etcd. It reads the .spec section, which represents the user's desired state (e.g., "replicas": 3). It then looks at the real-world cluster infrastructure to see the actual state (e.g., only 2 pods are running).
• Step 3: Act (Fixing the Drift): The controller executes the necessary logic to close the gap. In this case, it talks to the API server to spin up a 3rd pod.
• Step 4: Update Status: Once the action is complete, the controller updates the .status subresource of the CR. The API server writes this new status back to etcd, confirming the actual state now matches the desired state.

Key Characteristics

• Level-Triggered: The loop reacts not just to isolated events, but to the current state of the system. If the controller crashes and restarts, it reads etcd to immediately understand the current state and resume fixing any drift.
• Asynchronous: The user gets an immediate "Success" response from the API server when saving a CR to etcd. The controller processes the request in the background asynchronously.