Kubernetes Custom Resource Definition (CRD)

Why are Custom Resource Definitions (CRDs) stored in etcd?

Custom Resource Definitions (CRDs) are stored in etcd because etcd serves as the single source of truth and persistent data store for the entire Kubernetes control plane. Storing CRDs in etcd allows the kube-apiserver to dynamically register, validate, and manage custom resources exactly like native objects without requiring a separate database.

Why Kubernetes Stores CRDs in etcd

• Native API Integration: The kube-apiserver inherently reads from and writes to etcd. Storing CRDs here allows custom objects to instantly leverage native features like kubectl support, RBAC security, and namespaces.
• Consistent State: Like Pods or Deployments, custom resources represent the "desired state" of a system. etcd provides the strong consistency and distributed consensus needed to safely store this state across cluster nodes.
• Watch Mechanism: Custom controllers and operators rely on the Kubernetes watch API to listen for resource changes. Because etcd natively supports watch events, controllers can instantly react when a custom resource is created, updated, or deleted.

How custom controllers interact with the stored CRD data using the reconciliation loop?

Custom controllers interact with etcd through a continuous synchronization mechanism called the reconciliation loop. This loop constantly drives the actual state of the cluster toward your desired state.

Here is exactly how custom controllers interact with stored CRD data:

1. The Relationship to CRDs

A CRD (Custom Resource Definition) acts as the database schema. It defines the structure and validation rules for your custom data. When you create a CR (Custom Resource) based on that CRD, its data is stored directly in etcd.

The custom controller is the active software component (the brain) that watches that specific CR data and takes action based on what is stored.

2. The Reconciliation Loop Step-by-Step

The controller executes a continuous loop, often formalized as a Reconcile() function, which follows a Observe, Analyze, Act workflow:   

    +--------------------------------------------+
    |                                            |
    |               1. OBSERVE                   |
    |     (Watch etcd via API Server)            |
    |                                            |
    +---------------------+----------------------+
                          |
                          v
    +---------------------+----------------------+
    |                                            |
    |               2. ANALYZE                   |
    |    (Compare Desired vs. Actual State)      |
    |                                            |
    +---------------------+----------------------+
                          |
                          v
    +---------------------+----------------------+
    |                                            |
    |                 3. ACT                     |
    |     (Create/Update/Delete Infrastructure)  |
    |                                            |
    +--------------------------------------------+

• Step 1: Observe (The Watch Mechanism): Instead of constantly polling etcd, the controller establishes a long-lived HTTP connection (a watch) to the kube-apiserver. When a user creates, updates, or deletes a CR in etcd, the API server instantly broadcasts an event. The controller captures this event and queues the resource's key for processing.
• Step 2: Analyze (Diffing the State): The controller fetches the full CR data from etcd. It reads the .spec section, which represents the user's desired state (e.g., "replicas": 3). It then looks at the real-world cluster infrastructure to see the actual state (e.g., only 2 pods are running).
• Step 3: Act (Fixing the Drift): The controller executes the necessary logic to close the gap. In this case, it talks to the API server to spin up a 3rd pod.
• Step 4: Update Status: Once the action is complete, the controller updates the .status subresource of the CR. The API server writes this new status back to etcd, confirming the actual state now matches the desired state.

Key Characteristics

• Level-Triggered: The loop reacts not just to isolated events, but to the current state of the system. If the controller crashes and restarts, it reads etcd to immediately understand the current state and resume fixing any drift.
• Asynchronous: The user gets an immediate "Success" response from the API server when saving a CR to etcd. The controller processes the request in the background asynchronously.