Tuesday, 23 December 2025

Kubernetes EndpointSlices

 

EndpointSlices:
  • a modern, scalable way to track which Pods back a Service and how to reach them
  • split Service endpoints into small, scalable chunks that Kubernetes networking components can efficiently consume
  • replace (and improve on) the older Endpoints object

The problem they solve


Originally, Kubernetes used a single Endpoints object per Service that listed all Pod IPs and ports.

This caused issues at scale:
  • Large Services (hundreds/thousands of Pods) created huge objects
  • Frequent updates caused API server and etcd pressure
  • Harder to extend with metadata (zone, topology, readiness, etc.)

What EndpointSlices are


An EndpointSlice is:

  • A Kubernetes API object (discovery.k8s.io/v1)
  • Owned by a Service
  • Contains a subset of endpoints (Pods or external IPs)
  • Typically holds up to ~100 endpoints per slice (configurable)

So instead of:

Service → 1 big Endpoints object

We have:

Service → multiple EndpointSlice objects

What’s inside an EndpointSlice


An EndpointSlice includes:

  • Endpoints
    • IP addresses (IPv4 / IPv6)
    • Ready / serving / terminating status
    • Zone & node info
  • Ports
    • Name, port number, protocol
  • AddressType
    • IPv4, IPv6, or FQDN
  • Labels
    • Including kubernetes.io/service-name

Example (simplified):

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: my-service-abcde
  labels:
    kubernetes.io/service-name: my-service
addressType: IPv4
ports:
- name: http
  port: 80
  protocol: TCP
endpoints:
- addresses:
  - 10.244.1.12
  conditions:
    ready: true
    serving: true
    terminating: false


How they’re used


  • kube-proxy reads EndpointSlices to program iptables/IPVS rules
  • CoreDNS uses them for Service DNS resolution
  • Controllers watch them instead of Endpoints
  • The old Endpoints object still exists for backward compatibility

Ingress → Service → EndpointSlice → Pod IPs

Since modern clusters use EndpointSlices by default, lacking access to them breaks:
  • kubectl describe ingress
  • kubectl describe service
  • Some kubectl get ... -o wide outputs

Why EndpointSlices are better


✅ Scales much better
✅ Smaller, more frequent updates
✅ Topology-aware routing (zones, nodes)
✅ Supports dual-stack (IPv4 + IPv6)
✅ Extensible for future networking features

Relationship to Endpoints (important)


  • Modern clusters use EndpointSlices by default
  • Kubernetes still creates Endpoints objects unless disabled
  • You should:
    • Read EndpointSlices
    • Avoid writing Endpoints directly in new tooling

When you’ll notice them (DevOps angle)


As a DevOps engineer, you’ll run into EndpointSlices when:

  • Debugging Services with many Pods
  • Investigating kube-proxy or networking issues
  • Watching API load in large clusters
  • Writing controllers or operators
  • Tuning Service performance at scale

Useful command:

kubectl get endpointslices
kubectl describe endpointslice <name>


---

Kubernetes Objects

  


Kubernetes objects can be divided into two groups:

  • API objects = persistent, declarative resources stored in etcd
  • Non-API objects = runtime, node-local, or in-memory artifacts created to implement API objects

Kubernetes is fundamentally an API-driven control plane + runtime realization.

What are Kubernetes API objects?


Kubernetes API objects are persistent entities stored in etcd that represent the desired state and current state of your cluster.

They are:
  • Defined by a schema
  • Exposed via the Kubernetes API server
  • Created, read, updated, deleted via kubectl, controllers, or the API
  • Continuously reconciled by controllers
In short, if it’s stored in etcd and managed via the API server, it’s an API object.

Common Kubernetes API objects (by category)


Workloads

  • Pod
  • Deployment
  • ReplicaSet
  • StatefulSet
  • DaemonSet
  • Job
  • CronJob

Networking

  • Service
  • Ingress
  • NetworkPolicy
  • EndpointSlice
  • Endpoints (legacy)

Configuration & storage

  • ConfigMap
  • Secret
  • PersistentVolume (PV)
  • PersistentVolumeClaim (PVC)
  • StorageClass
  • VolumeSnapshot

Cluster & node

  • Node
  • Namespace
  • ResourceQuota
  • LimitRange
  • RuntimeClass

Security & access

  • ServiceAccount
  • Role, ClusterRole
  • RoleBinding, ClusterRoleBinding
  • PodSecurityPolicy (deprecated)
  • PodSecurityAdmission (via config)

Custom & extension

  • CustomResourceDefinition (CRD)
  • Any Custom Resource (e.g. Certificate, Prometheus, IngressRoute)

Key properties of API objects


Every API object has:

apiVersion: ...
kind: ...
metadata:
  name: ...
  namespace: ...
spec:        # desired state
status:      # current state (set by controllers)


And importantly:

Declarative: you describe what you want
Reconciled: controllers try to make reality match spec
Durable: survive restarts

What are “non-API” Kubernetes objects?


This is where the term object becomes fuzzy.

There are many things in Kubernetes that are real, important, and named — but are NOT API objects because they:
  • Are not stored in etcd
  • Do not have YAML
  • Are not CRUD’d via the API
  • Are usually runtime, internal, or node-local

Major categories of non-API Kubernetes “objects”


1️⃣ Containers

  • Individual containers inside a Pod
  • Managed by the container runtime (containerd / CRI-O)
  • Kubernetes does NOT store container objects
Example:
  • You can’t do kubectl get container
  • Containers exist only while the Pod is running

2️⃣ Volumes (runtime volumes)


While PV/PVC are API objects, the actual mounted volumes are not.

Examples:
  • emptyDir
  • hostPath
  • Mounted CSI volumes
These are:
  • Created on the node
  • Managed by kubelet + CSI
  • Invisible to the API as concrete objects

3️⃣ kube-proxy rules (iptables / IPVS)


  • iptables chains
  • IPVS virtual servers
  • eBPF maps (Cilium)
These implement Services but are:
  • Node-local
  • Ephemeral
  • Derived from API objects

4️⃣ Scheduler & controller internals


Examples:
  • Scheduling queues
  • Controller work queues
  • Leader election locks (in-memory)
  • Cached informers
These exist:
  • In process memory
  • For performance and correctness
  • Never stored in etcd

5️⃣ CNI networking artifacts


Examples:
  • Network namespaces
  • veth pairs
  • Routes
  • eBPF programs
  • Overlay tunnels

Created by:
  • kubelet
  • CNI plugins (Calico, Cilium, Flannel)

Not visible as API objects.

6️⃣ DNS records


  • Service A/AAAA records
  • Pod DNS entries
They’re derived from:
  • Services
  • EndpointSlices
But the records themselves:
  • Live inside CoreDNS
  • Are regenerated dynamically

7️⃣ Events (kind of special)


Event is an API object, but:
  • Short-lived
  • Often garbage-collected quickly
  • Not something you “own” declaratively
This blurs the line between API and runtime artifacts.

8️⃣ Admission & runtime state


Examples:
  • Mutated Pod specs (post-admission)
  • Pod sandbox state
  • cgroups
  • seccomp / AppArmor profiles applied

These exist at runtime, not as first-class objects.


A useful mental model


Three layers of “things” in Kubernetes
┌─────────────────────────────┐
│  API Objects (etcd)         │  ← declarative, durable
│  Pods, Services, Secrets    │
├─────────────────────────────┤
│  Controllers & Reconcilers  │  ← logic
├─────────────────────────────┤
│  Runtime Artifacts          │  ← ephemeral, node-local
│  Containers, iptables, CNI  │
└─────────────────────────────┘

Rule of thumb


Ask these questions:

Can I kubectl get it?
→ Yes → API object
→ No → probably runtime

Is it stored in etcd?
→ Yes → API object

Does it survive a full cluster restart?
→ Yes → API object
→ No → runtime artifact

Why this distinction matters (DevOps relevance)

  • Explains why “kubectl says it exists but it’s not working”
  • Helps debug node-local vs cluster-wide issues
  • Critical when writing controllers, operators, or CRDs
  • Clarifies what’s source of truth vs derived state

---

Thursday, 18 December 2025

Security Hardening of AWS EC2 Instances



Are password authentication and root login on Amazon EC2 instance disabled by default?


From Manage system users on your Amazon EC2 Linux instance - Amazon Elastic Compute Cloud:

By default, password authentication and root login are disabled, and sudo is enabled. To log in to your instance, you must use a key pair. For more information about logging in, see Connect to your Linux instance using SSH.

You can allow password authentication and root login for your instance. For more information, see the documentation for your operating system.


On a standard Amazon EC2 Linux instance, both are disabled by default, and SSH key-based login with a non-root user is required.​

Default SSH access

By default, you connect to an EC2 Linux instance using a non-root account such as ec2-user (Amazon Linux) or ubuntu (Ubuntu) with an SSH key pair, not a password. This design enforces public key authentication and avoids exposing password-based logins on the internet.​

Password authentication

Password authentication over SSH is disabled by default on EC2 Linux instances, so you cannot log in with a username and password until you explicitly enable it in sshd_config. To log in initially, you must use the key pair specified when the instance was launched.​

Root login

Direct root SSH login is also disabled by default; you are expected to log in as the default user and then use sudo to gain root privileges. Root login can be enabled later by changing PermitRootLogin in sshd_config, but this is discouraged from a security standpoint.


To verify the current settings, checkout out the /etc/ssh/sshd_config and look for these settings (their value can be yes or no):
  • PermitRootLogin
  • PasswordAuthentication

---