Kubernetes Architecture From First Principles: Control Plane, Self-Healing Pods, and Zero-Downtime Networking

Why Kubernetes Exists (And the Mental Model That Clicks)

Running a single Go binary inside a Docker container on one Linux VM is simple until two production realities hit:

1. Processes crash and nodes die at 3:00 AM.
2. Traffic spikes and zero-downtime deploys require routing requests across multiple instances without dropping in-flight connections.

Kubernetes is not magic; it is a distributed desired-state reconciliation loop backed by an immutable key-value store (etcd). Instead of running imperative shell scripts (ssh node-01 && docker run ...), you hand the API Server a declarative YAML manifest stating: "I want 3 replicas of go-api:v1.4.0 listening on port 8080." Every component in the diagram above exists to make actual cluster state converge on that desired state.

1. The Control Plane vs. Worker Nodes

Click between 🟢 Steady (3/3 Ready) and 💥 Kill Pod (Self-Heal) in the interactive visual instance above to see how the two planes divide responsibility:

The Control Plane (The Brain)

• kube-apiserver (:6443): The front door to the cluster. Every kubectl command, controller watch, and kubelet heartbeat goes through the API Server over TLS. It is the only component that talks directly to etcd.
• etcd: A strongly consistent Raft key-value store holding the entire cluster truth under /registry/. If it isn't recorded in etcd, it doesn't exist to Kubernetes.
• kube-controller-manager: Runs background control loops (ReplicaSet, Deployment, Node, Endpoint controllers). Each loop watches the API Server and asks one question: Does actual == desired?
• kube-scheduler: Watches for newly created Pods that have nodeName: "" (unscheduled), filters out nodes lacking CPU/memory requests, scores the remaining nodes, and binds the Pod to the best worker node.

The Worker Nodes (The Muscle)

• kubelet: The node agent running on every worker. It watches the API Server for Pods bound to its node, instructs the container runtime (containerd via CRI) to pull images and start containers, and continuously executes livenessProbe and readinessProbe checks.
• kube-proxy: Programs node networking rules (iptables or IPVS) so traffic sent to a virtual Service IP (ClusterIP) is load-balanced only to healthy Pod endpoints.

2. Core Mental Models: Pods, Deployments, and Services

3. Anatomy of a Self-Healing Go Deployment

When you click 💥 Kill Pod (Self-Heal) in the simulator above, trace the exact sequence that executes in under two seconds:

1. pod/go-api-7f8c-2 suffers an OOMKill or node eviction (Status: Terminating).
2. kubelet reports the container exit to kube-apiserver, which updates etcd.
3. Traffic Protection: The Endpoint controller immediately removes 10.244.1.18 from go-api-svc. kube-proxy updates IPVS tables so zero user requests hit the dead pod.
4. Reconciliation: ReplicaSet Controller sees Ready = 2 < Desired = 3 and POSTs a new Pod spec (pod/go-api-4) to kube-apiserver.
5. Scheduling & Boot: kube-scheduler places pod/go-api-4 onto Node-02. kubelet starts the Go container, waits for GET /healthz (readinessProbe) to return 200 OK, and adds the pod back into the ClusterIP rotation.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-api
  namespace: production
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: go-api
  template:
    metadata:
      labels:
        app: go-api
    spec:
      containers:
        - name: api
          image: ghcr.io/divineforge/go-api:v1.4.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "100m"
              memory: "64Mi"
            limits:
              cpu: "500m"
              memory: "128Mi"
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 2
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /livez
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10

4. Field Triage Runbook (With kubectl & k9s)

When a deployment does not turn green, 95% of production failures fall into four states:

• Pending: The scheduler cannot find a node that fits. Run kubectl describe pod <name> and check Events: for Insufficient cpu, Insufficient memory, or an unbound PersistentVolumeClaim.
• ImagePullBackOff: Bad image tag, missing registry imagePullSecrets, or rate-limited public pulls.
• CrashLoopBackOff: The container starts and exits non-zero (missing env var, panicking database connection, or an overly aggressive livenessProbe). Inspect the previous crash with kubectl logs <pod> --previous.
• OOMKilled (Exit Code 137): The Linux kernel cgroup killer terminated the process for exceeding resources.limits.memory. For Go services, pair container memory limits with GOMEMLIMIT (e.g., GOMEMLIMIT=110MiB on a 128Mi container) so the Go garbage collector runs before the kernel OOM-kills the pod.