Model Context Protocol (MCP)

 

Model Context Protocol (MCP)

The Model Context Protocol (MCP):

• Open-source standard
• Enables AI models to seamlessly connect with external data sources, tools, and software systems
• Acts as a universal "USB-C port" for AI, allowing LLMs to securely access local files, databases, and APIs to enhance context-aware responses. 
• Introduced by Anthropic in late 2024

Key Aspects of MCP:

• Purpose: Replaces fragmented, custom integrations with a single, open standard, making it easier to connect AI assistants to enterprise data, tools, and development environments.
• Components: Consists of:
• MCP Clients
• MCP Hosts - AI apps like Claude or coding agents 
• MCP Servers - programs that bridge specific data sources
• Security: MCP supports secure, two-way connections, allowing developers to control exactly what data is exposed to the AI.
• Functionality: Enables models to read files, query databases, use search engines, and call external APIs, providing live, relevant context for tasks.
• Open Standard: Hosted by the Linux Foundation, the protocol is designed for broad industry adoption. 

MCP differs from RAG (Retrieval-Augmented Generation) by focusing on active, two-way interaction with systems, whereas RAG is focused on retrieving documents for context.

For developers, it provides SDKs in Python and TypeScript.

MCP clients

MCP clients are the components within AI applications (AI Hosts) that manage one-to-one connections with MCP servers, translating AI requests into protocol-standardized messages. 

Popular MCP Client Applications

Several major AI-powered platforms and editors have integrated MCP client support to allow users to pull in their own tools and context: 

• Claude Desktop: Anthropic’s flagship app provides a built-in interface for managing local and remote MCP servers (e.g., Google Drive, Slack, GitHub).
• Cursor: An AI-native code editor that uses MCP to give its internal AI models direct access to project files, local databases, and custom developer tools.
• Windsurf Editor: A developer environment that supports tool invocation through MCP servers, allowing it to seamlessly interact with external scripts and APIs during coding sessions.
• Visual Studio Code (Agent Mode): Developers can use extensions to register MCP servers, enabling chat assistants to interact with internal enterprise tools directly within the editor.
• JetBrains IDEs: Platforms like IntelliJ IDEA feature an MCP-client UI where users can paste server configurations to bring external tool catalogues into the AI Assistant pane.
• BeeAI: An open-source desktop AI assistant from IBM that supports tool integration via built-in or custom MCP servers. 

Core Client Features

In the MCP architecture, clients don't just consume data; they provide specific features that enable complex, "agentic" workflows: 

• Sampling: Allows a server to request that the client (and its LLM) generate a completion, enabling the AI to "ask back" for clarification or more information.
• Elicitation: Provides a structured way for servers to ask users for specific information (like a password or preference) through the client’s UI.
• Roots: Allows the client to define specific file system boundaries, telling servers which directories they are permitted to access for safety and context scoping. 

Developer SDKs for Building Clients 

If you are building your own AI application, you can implement an MCP client using official software development kits: 

• TypeScript/Node.js SDK: The most comprehensive SDK for building web-based or local Node.js clients using standard JSON-RPC 2.0.
• Python SDK: Ideal for data science or automation workflows, allowing easy integration with Python-based AI agents.

MCP Hosts

Model Context Protocol (MCP) hosts are AI applications or environments that manage, connect, and run MCP clients to interact with external data sources (servers). Key examples include Claude Desktop, Claude Code, Zed, Replit, Cursor, and Sourcegraph. 

Key Roles of MCP Hosts:

• Orchestration: Act as the container for AI models to coordinate and communicate with multiple MCP servers.
• Security & Permissions: Manage user consent and enforce security policies (e.g., in sandboxed iframes) to control what data servers can access.
• Workflow Management: Aggregate user prompts, conversation state, and tool results from various sources to enable seamless interaction.
• Session Control: Initialize, maintain, and terminate connections between LLMs and tools via the MCP client. 

Primary MCP Host Examples:

• Claude Desktop: Anthropic's desktop application is a primary implementation of an MCP host, allowing the Claude model to utilize local tools and data.
• Claude Code: An AI agentic coding tool that acts as a host to interact with development-focused MCP servers.
• AI IDEs (Zed, Cursor, Replit): Integrated Development Environments that use MCP to connect coding assistants with files, documentation, and databases.
• Platforms & Agents: Other platforms like Workato or custom agents that orchestrate data access via the protocol.

What's the difference between mcp host and mcp client?

In the Model Context Protocol (MCP), the Host is the overall application you interact with, while the Client is the specific component inside that application that handles the technical connection to a server.

Key Differences at a Glance

• What it is
• MCP Host: The end-user application (e.g., Claude Desktop, Cursor).
• MCP Client: A protocol handler or "adapter" inside the host.
• Primary Role
• MCP Host: Manages the user interface and coordinates multiple clients.
• MCP Client: Maintains a 1:1 connection with a single MCP server.
• Responsibility
• MCP Host: Security policies, user consent, and aggregating data for the AI model
• MCP Client: Translating protocol messages (JSON-RPC) between the host and server.
• Hierarchy
• MCP Host: A single Host can contain multiple Clients.
• MCP Client: A Client is a subsidiary of the Host.

The "Restaurant" Analogy

To make it simpler, imagine a restaurant setting: 

• The Host is the Executive Chef: They decide what needs to be cooked and oversee everything, but they don't leave the kitchen to buy ingredients.
• The Client is the Waiter: They take the Chef's specific order, run to the source (the Server), and bring back exactly what was requested in a format the Chef can use. 

Why the distinction matters

While you will often hear people refer to applications like Claude Desktop as "the client," technically they are hosts. This architecture allows one app to connect to many different data sources (like Google Drive, Slack, and local files) simultaneously by instantiating a separate client for each one.

What is a MCP Client in Claude Desktop?

In Claude Desktop, the MCP client is the internal software layer that allows the app to "talk" to the tools you've added. While you might call the whole app "the client," it actually functions as a host that manages multiple individual client connections. 

How it works in Claude Desktop

• The Translator: When you ask Claude to "read a file," the client translates that human request into a technical JSON-RPC message that the Filesystem server understands.
• The Connection Manager: Claude Desktop can run several clients at once. For example, one client might be connected to a GitHub server while another is connected to a Google Drive server.
• Permission Gatekeeper: The client facilitates the security handshake. Before a tool executes, the client triggers the UI popup in Claude Desktop asking for your explicit permission. 

How to see them

You can see your active MCP clients and their available tools by clicking the "hammer" or "plug" icon (the MCP server indicator) in the bottom-right corner of the chat input box. 

Configuration

Claude Desktop's clients are configured via a local JSON file (claude_desktop_config.json). This file tells the internal clients exactly how to launch and communicate with your servers.

Config File Location:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

MCP Servers

Model Context Protocol (MCP) servers:

• Lightweight programs that connect AI models (like Claude or ChatGPT) to external data sources and tools, such as local files, databases, GitHub, or Slack
• Provide a standardized interface, enabling AI agents to securely access, read, and manipulate data beyond their training sets. 

Key Aspects of MCP Servers:

• Functionality: They expose specific capabilities—resources, prompts, and tools—to AI applications.
• Use Cases: Common implementations include file system access for documentation, database querying, and API interactions for services like GitHub or Google Tasks.
• Security: They provide controlled, authorized access to local or remote resources, with user permission required for actions.
• Architecture: As part of the Model Context Protocol, they act as the "server" in a client-server model, connecting to "hosts" like desktop apps. 

Common MCP Server Examples:

• Local File System: Allows AI to read, write, and organize local documents.
• GitHub/GitLab: Enables AI to manage repositories, create issues, and pull code.
• Database/API Connectors: Connects AI to SQL databases, HubSpot CRM, or AWS services.
• Developer Tools: Includes servers for Terraform, Angular CLI, and Home Assistant. 

You can build your own MCP server using Python or TypeScript, often utilizing tools like uv for environment setup.

How to start using Agentic AI in DevOps and Platform Engineering

The next frontier of DevOps and Platform Engineering is Agentic AI. We need to learn how autonomous agents reason and adapt to reduce cognitive load and accelerate the SDLC as we want to move beyond simple automation to build self-optimizing ecosystems that scale with confidence, innovation, and enterprise governance.


We should be able to:

• Explain the shift from automation to agentic AI and articulate what makes an AI system truly
• “agentic”
• Design agent-aware workflows in GitHub Actions, integrating LLMs with events, logs, APIs, and quality gates to create intelligent CI/CD pipelines
• Build AI-powered diagnostic loops that ingest failure context, reason about root causes, and generate structured remediation proposals or self-healing fixes
• Implement intelligent release decisions using multi-signal quality gates (test coverage, performance, security, cost) and generate auditable release rationale reports
• Deploy our own end-to-end platform engineering agent, capable of diagnosing pipeline failures, evaluating release readiness, and autonomously opening a fix PR or escalating with structured context.

Learning while Doing

• Identify Platform engineering pain points and the AI opportunity
• How can we get from static scripts and CI/CD automation to agentic AI
• Make a comparison of manual vs. AI-driven diagnosis
• Understand how platform engineering is evolving from static automation toward AI-driven systems that proactively diagnose and resolve operational issues
• Agentic AI fundamentals - how agents reason and act?
• Learn about core agent components (LLMs, memory, and tools)
• Compare event-driven vs. polling architectures
• Balance autonomous actions with human oversight
• Understand how agentic systems combine reasoning, memory, and tools to perceive events, make decisions, and act within engineering workflows
• How to setup the environment and create our first agentic workflow
• Set up an agentic runtime that responds to CI/CD events
• Connect an AI agent to our pipeline's event stream and context
• Trigger our first agent run and interpret its reasoning logs
• Learn how to connect AI agents to CI/CD events and platform context to trigger automated reasoning and actions in real time
• AI-powered diagnosis and remediation
• Compare manual vs. AI-driven incident diagnosis 
• Build agents that read logs, reason about failures, and propose fixes 
• Define escalation boundaries: when the agent self-heals vs. asks a human
• Understand how AI agents analyze logs, diagnose failures, and determine whether to self-heal or escalate issues to humans
• Intelligent CI/CD & adaptive delivery
• How to move beyond pass/fail pipelines to AI-driven release decision
• Automate rollback decisions using AI quality gates
• Query pipeline state and release history using natural language
• How AI transforms CI/CD pipelines into adaptive systems that make context-aware release and rollback decisions
• Operational intelligence & conversational observability
• Replace complex dashboards with AI anomaly detection
• Check platform health via chat interfaces
• Shift from reactive alerts to predictive management
• Understand how AI enables conversational access to platform health and detects anomalies to support proactive operations.
• Multi-agent coordination & implementation strategy
• Architect multi-agent systems for our platform workflows
• Handle agent conflicts, failures, and graceful degradation 
• Design a phased enterprise rollout with guardrails and audit trails
• Build our platform engineering agent
• Learn how to design coordinated multi-agent systems that handle complex platform workflows with governance and reliability
• Wire together diagnosis, quality gates, and observability into one agent pipeline
• Implement self-healing PRs with confidence thresholds
• Shift our role from platform operator to AI supervisor
• Learn how to combine diagnosis, delivery intelligence, and observability into a unified agent that automates key platform workflows

---

Kubernetes StatefulSet

 

In Kubernetes, a StatefulSet is a specialized workload API object designed to manage stateful applications. Unlike standard Deployments, where Pods are interchangeable "cattle," StatefulSets treat Pods as unique "pets" with a persistent identity that is maintained even if they are rescheduled or restarted.

Key Features

• Stable Network Identity: Each Pod is assigned a unique, ordinal index (e.g., web-0, web-1) and a corresponding stable DNS name through a Headless Service.
• Stable Storage: By using volumeClaimTemplates, each Pod is automatically paired with its own PersistentVolume. If a Pod dies, the replacement Pod with the same identity will automatically remount the same storage.
• Ordered Deployment: Pods are created and scaled sequentially from 0 to N-1. Kubernetes ensures that the previous Pod is "Running and Ready" before starting the next one.
• Ordered Termination: Scaling down or deleting the StatefulSet occurs in reverse order, starting from the highest ordinal (e.g., web-2 is deleted before web-1). 

When to Use StatefulSets

StatefulSets are the standard choice for applications that require consistent data and unique identities, such as: 

• Databases: Systems like MySQL, PostgreSQL, MongoDB, and Cassandra.
• Distributed Systems: Tools like ZooKeeper, Kafka, and Elasticsearch that need a quorum or master election.
• Clustered Applications: Any software where instances need to know each other’s specific addresses to sync data. 

Comparison: StatefulSet vs. Deployment

Feature         StatefulSet                                                 Deployment

----------           --------------                                                     ---------------

Pod Identity: Unique and stable (ordinal names)                 Randomly generated and ephemeral

Storage:         Dedicated volume per Pod (via template) Typically shared or transient

Network: Fixed DNS per Pod (via Headless Service) Single Load Balancer for the whole set

Scaling:         Sequential (0, then 1, then 2...)                 Parallel (multiple Pods at once)

Best Practices

• Use Headless Services: Always pair our StatefulSet with a Service that has clusterIP: None to ensure Pods are individually addressable.
• Persistent Storage: Ensure our StorageClass is correctly configured for dynamic provisioning so that each Pod gets its own disk automatically.
• Manual Data Sync: Note that while Kubernetes manages the infrastructure, we are still responsible for configuring internal application logic like data replication or master/slave sync. 

YAML manifest example for a basic MySQL StatefulSet

Below is a standard YAML manifest for a MySQL StatefulSet. It includes a Headless Service for network identity and a volumeClaimTemplate to automatically provision unique storage for each replica.

apiVersion: v1

kind: Service

metadata:

  name: mysql

  labels:

    app: mysql

spec:

  ports:

  - port: 3306

    name: mysql

  clusterIP: None # Defines this as a Headless Service

  selector:

    app: mysql

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: mysql

spec:

  selector:

    matchLabels:

      app: mysql

  serviceName: "mysql"

  replicas: 3

  template:

    metadata:

      labels:

        app: mysql

    spec:

      containers:

      - name: mysql

        image: mysql:8.0

        env:

        - name: MYSQL_ROOT_PASSWORD

          value: "password" # Use Secrets in production!

        ports:

        - containerPort: 3306

          name: mysql

        volumeMounts:

        - name: mysql-data

          mountPath: /var/lib/mysql

  volumeClaimTemplates:

  - metadata:

      name: mysql-data

    spec:

      accessModes: [ "ReadWriteOnce" ]

      resources:

        requests:

          storage: 1Gi

Why this works:

• Stable DNS: Each Pod gets a predictable name: mysql-0.mysql, mysql-1.mysql, etc.
• Unique Storage: Kubernetes creates three separate PersistentVolumeClaims. mysql-0 will always mount the first disk, even after a reboot.
• Ordered Startup: Pods launch one after another (0, then 1, then 2), which is critical for forming database clusters.

To use a Kubernetes Secret (like mysql-secret) instead of hardcoding passwords, we need to create a Secret object and then reference it in our StatefulSet. This is the standard practice for distributing credentials securely in Kubernetes.

1. Create the Secret

We can define our password in a YAML file. Note that values in the data field must be base64 encoded.

apiVersion: v1

kind: Secret

metadata:

  name: mysql-secret

type: Opaque

data:

  # 'password' encoded in base64 is 'cGFzc3dvcmQ='

  root-password: cGFzc3dvcmQ=

Alternatively, we can use stringData to provide the password in plain text; Kubernetes will handle the encoding for us when we apply it:

apiVersion: v1

kind: Secret

metadata:

  name: mysql-secret

type: Opaque

stringData:

  root-password: "our-secure-password"

2. Update the StatefulSet

Modify the env section of our MySQL container to use valueFrom and secretKeyRef. This tells the Pod to pull the value of MYSQL_ROOT_PASSWORD from the secret we just created.

 containers:

      - name: mysql

        image: mysql:8.0

        env:

        - name: MYSQL_ROOT_PASSWORD

          valueFrom:

            secretKeyRef:

              name: mysql-secret   # Name of our Secret object

              key: root-password   # The specific key inside the Secret

Key Considerations

• Initialization Only: For MySQL, the MYSQL_ROOT_PASSWORD environment variable is typically only used during the first-time initialization of the data directory. Changing the Secret later will not automatically update the root password in an existing database.
• Security: Ensure our cluster has encryption at rest enabled for Secrets to truly protect sensitive data.
• Alternative for Multiple Variables: If we have many credentials (user, password, DB name), we can use envFrom to map all keys in a Secret to environment variables at once.

Changing Storage Spec

Changing spec.volumeClaimTemplate updates the StatefulSet template but will not resize already-created PVCs. If the goal is to fix an existing CrashLoopBackOff due to disk-full, we still need to expand the current PVC(s) (and ensure the general StorageClass allows volume expansion), or recreate the PVC/StatefulSet so the new size takes effect.

References:

StatefulSets | Kubernetes

Amazon EBS CSI Driver

The Amazon EBS CSI Driver is a standard interface that allows Amazon Elastic Kubernetes Service (EKS) clusters to manage the full lifecycle of Amazon EBS volumes as persistent storage for containers. It replaces the older, deprecated "in-tree" Kubernetes storage plugin with a more flexible, decoupled model. 

Key Features

• Dynamic Provisioning: Automatically creates and attaches EBS volumes when a PersistentVolumeClaim (PVC) is made.
• Volume Lifecycle Management: Handles the creation, attachment, mounting, and deletion of volumes.
• Resizing & Snapshots: Supports online volume resizing (for gp3 and others) and taking volume snapshots for data backup.
• EKS Auto Mode Support: In EKS Auto Mode, routine block storage tasks are automated, and you don't even need to manually install the driver. 

Deployment Methods

You can install and manage the driver through several channels: 

• EKS Managed Add-on (Recommended): Simplifies installation and updates via the AWS Console, CLI, or Terraform.
• Helm Chart: Provides highly customizable installation options.
• Kustomize: Direct deployment using manifests from the official GitHub repository. 

Core Requirements

• IAM Permissions: The driver requires an IAM role with the AmazonEBSCSIDriverPolicy to interact with EBS resources.
• Service Accounts: Typically uses IAM Roles for Service Accounts (IRSA) to securely provide AWS credentials to the driver pods.
• Compatibility: Supports Linux and Windows worker nodes, as well as ARM64 architectures.

Driver Components

The driver is typically deployed into the kube-system namespace and consists of two main parts: 

• Controller Deployment: Runs as a set of replicas (ebs-csi-controller) to communicate with the AWS EC2 API and manage volume operations.
• Node DaemonSet: Runs on every worker node (ebs-csi-node) to handle the actual mounting and unmounting of volumes to pods on that specific host. 

In the Amazon EBS CSI driver architecture, the ebs-csi-controller and ebs-csi-node are the two primary components that work together to manage the lifecycle of EBS volumes in a Kubernetes cluster.

Core Feature Differences

ebs-csi-controller

• Deployment Type: 
• Deployment (typically 2 replicas for HA)
• Main Function: 
• Control Plane operations: Creating, deleting, attaching, and detaching volumes
• AWS Interaction: 
• Calls the AWS EC2 API to manage EBS resources
• IAM Permissions: 
• Requires an IAM role with permissions like ec2:CreateVolume and ec2:AttachVolume

ebs-csi-node

• Deployment Type: 
• DaemonSet (runs on every worker node)
• Main Function: 
• Node-level operations: Mounting and unmounting volumes to the local filesystem
• AWS Interaction: 
• Interacts with the local OS (privileged system calls) to handle block devices
• IAM Permissions: 
• Generally requires fewer/no AWS API permissions, as it mostly performs local mount actions

How They Work Together

• Provisioning & Attachment: When you create a PersistentVolumeClaim (PVC), the ebs-csi-controller watches the request and calls the AWS API to create the EBS volume and attach it to the correct EC2 instance.
• Mounting: Once the volume is physically attached to the EC2 instance, the ebs-csi-node pod running on that specific node detects the new block device and mounts it into the container’s path so your application can use it. 

Key Considerations

• Security: For better security, you can schedule the ebs-csi-controller on hardened management nodes, while the ebs-csi-node must run everywhere your workloads need storage.
• Fargate: You can run the controller on Fargate nodes, but the ebs-csi-node (as a DaemonSet) only runs on EC2 instances.
• Troubleshooting: If a volume fails to "attach," check the controller logs; if it fails to "mount" or "format," check the node logs.

Pods for both the ebs-csi-controller and ebs-csi-node typically share the same value for the app.kubernetes.io/name label. 

In standard deployments (such as via the official Helm chart or EKS add-on), both components use this label to identify that they belong to the same overarching application: the Amazon EBS CSI Driver.

Label Comparisons

While they share the same application name, they use the app.kubernetes.io/component label to distinguish between their specific roles.

Label Key                           ebs-csi-controller Pods     ebs-csi-node Pods

------------                                  ------------------------------    ----------------------

app.kubernetes.io/name           aws-ebs-csi-driver       aws-ebs-csi-driver

app.kubernetes.io/instance   aws-ebs-csi-driver       aws-ebs-csi-driver

app.kubernetes.io/component   csi-driver (or controller)     csi-driver (or node)

app (Legacy label)                   ebs-csi-controller               ebs-csi-node

How to Verify in Your Cluster

You can check these labels yourself using kubectl. This is useful if you are writing Prometheus rules or network policies that need to target the entire driver or just one part of it. 

To see labels for all EBS CSI pods:

kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver --show-labels

To target only the controller:

kubectl get pods -n kube-system -l app=ebs-csi-controller

---

Monitoring and Observability

 

Monitoring vs Observability

In the world of IT and DevOps, monitoring and observability are two related but distinct concepts used to manage system health and performance. 

Core Difference

The simplest way to distinguish them is:

• Monitoring tells you what is happening (and when). It is reactive and focuses on known problems using predefined metrics.
• Observability tells you why it is happening. It is proactive and uses the system's outputs to understand its internal state, especially for "unknown unknowns". 

Key Comparison Table

Feature         Monitoring                 Observability

----------           ---------------                    -----------------    

Purpose         Detect known issues Diagnose root causes

Perspective External (symptoms) Internal (system state)

Question         "Is the system healthy?" "Why is it behaving this way?"

Approach Reactive                         Proactive

Focus         "Known knowns"         "Unknown unknowns"

Data Types Metrics, logs                 Metrics, logs, and traces

Analogy: The Car

• Monitoring is your dashboard. It has dials for speed and fuel, and a "check engine" light. It tells you if you are speeding or if something is broken.
• Observability is the mechanic’s diagnostic tool. When the "check engine" light comes on, the mechanic plugs in a tool to see exactly which sensor failed and why, without having to take the entire engine apart. 

Common Tools

• Monitoring Tools: Nagios, Zabbix, Prometheus.
• Observability Platforms: Datadog, New Relic, Honeycomb, Dynatrace.

Three Pillars of Observability

The three pillars of observability—metrics, logs, and traces—are essential telemetry data types used to understand the internal state of complex, distributed systems. They enable teams to detect, investigate, and resolve performance issues by providing high-level trends, granular event details, and full request-flow paths. 

Metrics

Quantitative measurements over time (e.g., CPU usage, error rates).

Numerical measurements that describe the health, performance, and behavior of a system over time (e.g., CPU usage, error rates, throughput). They are ideal for alerting, capacity planning, and spotting trends or symptoms.

Logs

Granular, timestamped records of discrete events.

Timestamped, granular records of discrete events. They provide the detailed context (text or structured data) necessary to understand exactly what happened within an application or service.

Traces

End-to-end journeys of a single request through a distributed system, showing how different 

Records showing the journey of a single request as it travels through a distributed system, encompassing multiple services. They are critical for pinpointing bottlenecks, latency, or failures in microservices architectures. 

Why They Are Used Together

While metrics indicate that a problem exists, logs provide the context of why it happened, and traces show where it is occurring. Correlating these three data types provides actionable insights rather than just raw data.

---

How to architecture highly available and fault tolerant AWS EKS Kubernetes cluster

We should follow a layered approach—from the network up to the application.

The "Highly Available EKS" Design Framework

1. Networking & Control Plane (The Foundation)

• Multi-AZ VPC: 
• Design a VPC with at least 3 Private Subnets across 3 different Availability Zones (AZs).
• EKS Endpoint (API server endpoint) Access: 
• Enable "Private Access" for the EKS control plane so worker nodes communicate with API sever within the VPC, reducing exposure and latency.

• NAT Gateways: 
• Use one NAT Gateway per AZ (3 total) to ensure that if one AZ fails, the nodes in other zones still have outbound internet access for image pulls. This means that we need to have 3 public subnets, one per each AZ.

Application Pod Disruption Budget (PDB) in Kubernetes

A Pod Disruption Budget (PDB) in Kubernetes:

• It's an API object
• Sets for an application deployed in cluster:
• Either
• Minimum number of pods an application needs to function smoothly...
• Or
• Maximum number of replicated pods that are down simultaneously... 
• ...during voluntary disruptions (e.g., node upgrades, maintenance, draining)
• Ensures high availability by guaranteeing a minimum number or percentage of pods remain active. 

Key Aspects of PDBs:

In general, disruptions can be:

• voluntary, such as maintenance operations or node scaling, or
• involuntary, such as hardware failures or system crashes

Voluntary Focus: PDBs only protect against voluntary disruptions, such as kubectl drain or node repairs, not against involuntary, unavoidable failures.

Configuration: You define a PDB using either

• either minAvailable 
• minimum pods that must run
• or maxUnavailable
• maximum pods allowed to be down
• PDB configuration setting defining the maximum number of pods that can be voluntarily taken down simultaneously.

Use Case: Ideal for quorum-based applications (e.g., Elasticsearch, Zookeeper) to ensure quorum is never lost during node maintenance.

Mechanism: When a cluster administrator drains a node, the system checks the PDB. If removing a pod violates the budget, the action is delayed until enough replicas are available elsewhere. 

Example PDB Configuration:

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

  name: web-pdb

spec:

  minAvailable: 2 # At least 2 pods must remain running

  selector:

    matchLabels:

      app: web-app

Name is usually set to refer to the application for which PDB is created ("web" in the example above).

Best Practice: 

Use PDBs in conjunction with pod anti-affinity rules to ensure pods are spread across nodes.

How to check PDB in cluster?

Example:

% kubectl get pdb -A                          

NAMESPACE     NAME                        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE

clickhouse    chi-clickhouse-ch           N/A             1                 1                     132d

kube-system   ws-cluster-autoscaler       N/A             1                 1                     133d

kube-system   coredns                     N/A             1                 1                     140d

kube-system   ebs-csi-controller          N/A             1                 1                     140d

kube-system   karpenter                   N/A             1                 1                     139d

ALLOWED DISRUPTIONS:

• dynamic value, constantly re-calculated by the disruption controller
• the real-time status indicator 
• showing how many pods can currently be evicted without violating the set maxUnavailable or minAvailable constraints
• The non-zero value means that the disruption controller has seen the pods, counted the matching pods, and updated the status of the PDB

To see the number of current and desired healthy pods (and how ALLOWED DISRUPTIONS is actually calculated) run this command (in the example below it was for karpenter application):

% kubectl get poddisruptionbudgets karpenter -n kube-system -o yaml

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

  annotations:

    meta.helm.sh/release-name: karpenter

    meta.helm.sh/release-namespace: kube-system

  creationTimestamp: "2025-10-21T14:05:33Z"

  generation: 1

  labels:

    app.kubernetes.io/instance: karpenter

    app.kubernetes.io/managed-by: Helm

    app.kubernetes.io/name: karpenter

    app.kubernetes.io/version: 1.3.2

    helm.sh/chart: karpenter-1.3.2

  name: karpenter

  namespace: kube-system

  resourceVersion: "2664456"

  uid: 2b58340a-fd07-4567-95a9-2a43b5dd4bca

spec:

  maxUnavailable: 1

  selector:

    matchLabels:

      app.kubernetes.io/instance: karpenter

      app.kubernetes.io/name: karpenter

status:

  conditions:

  - lastTransitionTime: "2025-10-27T10:52:01Z"

    message: ""

    observedGeneration: 1

    reason: SufficientPods

    status: "True"

    type: DisruptionAllowed

  currentHealthy: 2

  desiredHealthy: 1

  disruptionsAllowed: 1

  expectedPods: 2

  observedGeneration: 1

PDB and Rolling Update of Node Group

ALLOWED DISRUPTIONS = 1 is generally the safest and most standard setting for a rolling node group update (which can be triggered if we e.g. upgrade k8s version of aws_eks_node_group), especially for high-availability workloads. 

Key Considerations for maxUnavailable: 1

Safety First: This setting ensures only one node is updated at a time (node - if all pod replicas are distribute across all nodes). This is ideal for maintaining quorum in stateful applications like databases (e.g., Consul or ZooKeeper) where losing multiple nodes 

simultaneously could cause data loss or service failure.

Default Behavior: In Amazon EKS managed node groups, maxUnavailable defaults to 1 if not specified.

Resource Availability: For this to work, your cluster must have enough spare capacity (CPU/Memory) on the remaining nodes to host the pods evicted from the node being updated.

Update Speed: While safe, updating one node at a time is the slowest method. For very large clusters, you might consider a higher absolute number or a percentage (e.g., 10%) to speed up the process. 

When 1 is NOT Enough

Blocking Drains: If you have a Pod Disruption Budget (PDB) where minAvailable equals your total replicas, the node drain will be blocked, and the update will stall because no pods can be legally moved.

Timeouts: Amazon EKS has a 15-minute timeout for draining pods. If pods take too long to terminate, the update may fail unless you have configured pre-stop hooks or adjusted your PDBs.

In-Depth Explanation

In a Pod Disruption Budget (PDB), the setting ALLOWED DISRUPTIONS = 1 (which results from either maxUnavailable: 1 or a minAvailable value that leaves one "slot" free) acts as a safety valve that synchronizes infrastructure changes with application health. 

Here is exactly how it makes a rolling node group update safe:

1. It Hooks into the "Eviction API" 

When a node group update begins, the automation doesn't just "kill" pods; it calls the Kubernetes Eviction API. 

The Check: Before a pod is removed from a node being updated, the Eviction API checks your PDB.

The Logic: If ALLOWED DISRUPTIONS is 1, the API allows exactly one pod to be terminated. Once that pod is gone, the "Allowed Disruptions" counter drops to 0. 

2. It Forces a Sequential Wait 

This is the most critical safety feature. If a second pod eviction is requested while the first one is still being replaced: 

• The Block: The API sees ALLOWED DISRUPTIONS = 0 and rejects the eviction request.
• The Wait: The node "drain" process pauses and retries. It will stay paused until the first pod's replacement is scheduled on a different node and passes its Readiness Probe.
• The Reset: Only when the new pod is "Ready" does the ALLOWED DISRUPTIONS count return to 1, allowing the next pod to be safely evicted. 

3. It Prevents "Brain-Dead" Automation

Without this PDB setting, a node group update might try to drain a node that holds all replicas of your app. 

• Without PDB: The node drains, kills all pods at once, and you have a total outage.
• With ALLOWED DISRUPTIONS = 1: The automation is physically unable to kill the second pod until the first one is safely back online elsewhere, ensuring your app always has at least some capacity. 

Summary of the "Safety Loop"

Step Action                                                                 PDB State (Allowed Disruptions)

1 Node update starts; first pod eviction requested 1 → Eviction granted

2 First pod is terminating; replacement is starting 0 → All further evictions blocked

3 Replacement pod passes Readiness Probe         0 → 1 → Block lifted

4 Next pod eviction requested                                 1 → Process repeats

Important Note: If your app only has 1 replica total, a PDB with minAvailable: 1 will block the node update forever because it can never safely evict that single pod. You generally need at least 2 replicas for this safety mechanism to work without stalling your updates.

Can pod disruption budget be defined in deployment manifest?

No, a Pod Disruption Budget (PDB) cannot be defined directly inside a Deployment manifest. In Kubernetes, a PDB is a separate API resource (Kind: PodDisruptionBudget) that exists independently of the Deployment. 

While they are separate resources, they are linked by a Label Selector. You define the PDB to target specific pods by matching the labels specified in your Deployment's pod template. 

Why they are separate?

• Decoupled Lifecycle: You can create, update, or delete a PDB without needing to redeploy your application or trigger a rolling update of your pods.
• Multiple Controllers: A single PDB can protect pods across multiple Deployments, ReplicaSets, or StatefulSets as long as they share the same labels.
• Administrative Control: Cluster administrators may manage PDBs separately from application developers to ensure cluster-wide stability during maintenance like node drains.

How to link them?

To make them work together, ensure the spec.selector.matchLabels in your PDB YAML matches the spec.template.metadata.labels in your Deployment YAML. 

Deployment snippet:

metadata:

  labels:

    app: my-app

spec:

  template:

    metadata:

      labels:

        app: my-app

PDB snippet:

spec:

  selector:

    matchLabels:

      app: my-app

  minAvailable: 2

Here is an example of a complete multi-resource YAML that includes both the Deployment and the PDB:

apiVersion: apps/v1

kind: Deployment

metadata:

  name: web-server

  labels:

    app: nginx

spec:

  replicas: 3

  selector:

    matchLabels:

      app: nginx

  template:

    metadata:

      labels:

        app: nginx

    spec:

      containers:

      - name: nginx

        image: nginx:latest

        ports:

        - containerPort: 80

---

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

  name: web-server-pdb

spec:

  # The PDB finds the pods using this selector

  selector:

    matchLabels:

      app: nginx

  # Ensures at least 2 pods stay up during voluntary disruptions

  minAvailable: 2

Key Takeaways:

• The Link: The spec.selector.matchLabels in the PDB must exactly match the spec.template.metadata.labels in the Deployment.
• Voluntary Disruptions: This PDB will protect your pods during actions like node drains or cluster upgrades, but it won't prevent "involuntary" issues like a hardware crash.

Using maxUnavailable is often better for autoscaling environments because it scales with your replica count. Instead of saying "I need X pods alive," you’re saying "I can afford to lose X pods."

Here is the updated manifest:

apiVersion: apps/v1

kind: Deployment

metadata:

  name: web-server

spec:

  replicas: 5

  selector:

    matchLabels:

      app: nginx

  template:

    metadata:

      labels:

        app: nginx

    spec:

      containers:

      - name: nginx

        image: nginx:latest

---

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

  name: web-server-pdb

spec:

  selector:

    matchLabels:

      app: nginx

  # Only 1 pod can be taken down at a time during maintenance

  maxUnavailable: 1

Pro Tips for maxUnavailable:

• Absolute Number: Using 1 ensures that even if you have 100 replicas, Kubernetes will only drain one node at a time for this app.
• Percentage: You can use a string like "25%" if you want the "allowed downtime" to grow or shrink as your Deployment scales.
• The "One" Rule: You cannot use both minAvailable and maxUnavailable in the same PDB; you have to pick one strategy.

How to use a percentage to make the PDB more flexible for Horizontal Pod Autoscaling (HPA)?

Using percentages for a Pod Disruption Budget (PDB) is highly recommended when combined with a Horizontal Pod Autoscaler (HPA). This allows the disruption budget to scale dynamically as your application grows or shrinks based on traffic. 

Dynamic YAML Example (Deployment + HPA + PDB)

This configuration ensures that no matter how many replicas the HPA creates, at least 80% of them will always remain available during maintenance.

apiVersion: apps/v1

kind: Deployment

metadata:

  name: dynamic-app

spec:

  replicas: 2 # Initial count

  selector:

    matchLabels:

      app: web

  template:

    metadata:

      labels:

        app: web

    spec:

      containers:

      - name: nginx

        image: nginx:latest

---

apiVersion: autoscaling/v2

kind: HorizontalPodAutoscaler

metadata:

  name: web-hpa

spec:

  scaleTargetRef:

    apiVersion: apps/v1

    kind: Deployment

    name: dynamic-app

  minReplicas: 2

  maxReplicas: 10

  metrics:

  - type: Resource

    resource:

      name: cpu

      target:

        type: Utilization

        averageUtilization: 50

---

apiVersion: policy/v1

kind: PodDisruptionBudget

metadata:

  name: web-pdb

spec:

  selector:

    matchLabels:

      app: web

  # Scales with HPA: if HPA scales to 10 pods, 8 must stay up.

  minAvailable: 80%

Key Behaviours to Note:

• Rounding Logic: Kubernetes always rounds up the result of the percentage to the nearest integer. For example, if you have 7 pods and set minAvailable: 50%, Kubernetes requires 4 pods to remain available.
• Maintenance Blockers: Be careful with 100% or 0%. Setting minAvailable: 100% or maxUnavailable: 0% will completely block node drains, preventing cluster administrators from performing upgrades or maintenance.
• Single Replica Limitation: If your HPA scales down to 1 replica, a PDB with minAvailable: 1 or 100% will prevent that single pod from being evicted, which can stall cluster maintenance. 

How to troubleshoot a node drain that is stuck because of a PDB?

A kubectl drain typically gets stuck because the ALLOWED DISRUPTIONS for a specific PDB is 0. This means evicting even one more pod would violate your availability rules. 

1. Identify the Blocking PDB

Run this command to see which PDB is preventing the drain: 

kubectl get pdb -A

Look for any PDB where ALLOWED DISRUPTIONS is 0. If it’s 0, the eviction API will reject any attempt to move those pods, and the drain will wait indefinitely. 

2. Common Reasons for "Allowed Disruptions: 0"

• Too Few Replicas: You have minAvailable: 1 but only 1 pod is currently running.
• Unhealthy Pods: Your PDB requires 3 pods to be healthy, but one is crashing or not yet "Ready." The PDB won't allow any more to be removed until that one is fixed.
• Misconfiguration: You set minAvailable equal to your total replicas (e.g., 3 out of 3), meaning you've effectively told Kubernetes "never take a pod down". 

3. How to Unstick the Drain

Depending on your situation, use one of these three strategies:

• Strategy: Scale Up
• Command:  kubectl scale deployment <name> --replicas=<current+1>
• Best For: Safest: Adds a new pod on a different node so the PDB is satisfied.
• Strategy: Relax PDB
• Command: kubectl edit pdb <pdb-name>
• Best For: Cleanest: Temporarily lower minAvailable or increase maxUnavailable.
• Strategy: Temporary Delete
• Command: kubectl delete pdb <pdb-name>
• Best For: Emergency: Delete the PDB, finish the drain, then re-apply the PDB later.

Warning: The "Force" Flag

Using kubectl drain <node> --force does not bypass PDBs. It only forces the deletion of pods not managed by a controller (like standalone pods) or pods using emptyDir. To truly bypass a PDB, you must either delete the PDB or manually kubectl delete pod the blocking pods

How to find all pods on a specific node that are currently protected by a PDB?

To find pods on a specific node that are protected by a Pod Disruption Budget (PDB), you need to cross-reference the pods on that node with the label selectors defined in your PDBs. 

Option 1: The Quickest Way (Manual Check)

The most direct way to see all pods on a node and identify their labels is to use the describe node command. 

kubectl describe node <node-name>

Look for the Non-terminated Pods section. You can then check if those pods' labels match your PDB selectors. 

Option 2: Filtered Search (Best for Large Nodes)

If the node has many pods, use this command to list all pods on that node along with their labels:

kubectl get pods -A \

   -o wide \

   --field-selector spec.nodeName=<node-name> \

   --show-labels

-A: Searches across all namespaces.

--field-selector spec.nodeName: Limits results to the specific node.

--show-labels: Displays the labels you need to compare against your PDB's spec.selector. 

Option 3: Identifying the PDB Selector

To see exactly what labels a PDB is looking for, run:

kubectl get pdb <pdb-name> \

   -o jsonpath='{.spec.selector.matchLabels}'

Any pod on your node that has these exact labels is protected by that PDB. 

Summary of Useful Flags

Flag                 Purpose

-----------------         ----------

-o wide                 Shows the NODE column to verify where pods are running.

--field-selector Filters the list server-side by the node name.

-l or --selector         Filters pods by specific labels (e.g., kubectl get pods -l app=nginx).

---