Kubernetes the Hard Way in 2026: Build a Cluster From Scratch

Most people meet Kubernetes through a managed service or an installer. You run one command, wait a few minutes, and a cluster appears. That is great for getting work done. It is not great for learning, because everything interesting happened somewhere you could not see.

This guide does the opposite. You will build a small cluster by hand from the official release binaries. You will create every certificate, write every config file and start every component yourself. When something breaks later in your career, you will know which piece to look at, because you will have installed each one.

The idea is not ours. Kelsey Hightower’s Kubernetes The Hard Way has taught this lesson to a generation of engineers. This is our own walkthrough of the same journey, written for Kubernetes 1.37 and Debian 13. We ran every command in this post, top to bottom, on fresh machines before publishing it.

Set aside two to three hours. Copy and paste is fine, but read the text between the commands. That is where the learning is.

What you will build

Four Linux machines, each with one job.

The jumpbox is where you prepare everything. The other three only receive finished files and run them. Keeping the two apart makes it obvious what each machine actually needs.

These are the versions we tested with.

CoreDNS and the CNI plugins are the exact versions the Kubernetes 1.37 release pins, and etcd is a newer patch release of the 3.7 line it pins, so you are building with a combination the project tests.

Before you start

You need four machines running Debian 13 (trixie). Cloud servers, local virtual machines or old laptops all work. A few rules matter.

• All four must reach each other on a private network.
• All four must use the same CPU architecture, either amd64 or arm64. The commands below detect it.
• The jumpbox can be small (1 vCPU, 512 MB of memory). Give the server and both workers at least 2 vCPU and 2 GB of memory each, and 20 GB of disk.
• The machine network must not overlap with the two ranges this cluster uses inside itself. Pods get addresses from 10.200.0.0/16 and services from 10.32.0.0/24.
• You can log in to the jumpbox as root, and root on the jumpbox can SSH into the other three machines as root with a key.

If the last point is not set up yet, create a key on the jumpbox with ssh-keygen -t ed25519, then add the contents of /root/.ssh/id_ed25519.pub to /root/.ssh/authorized_keys on the server and both workers. Debian’s SSH server allows root to log in with a key by default, though some cloud images add their own default user and expect you to use sudo. In that case, run sudo -i first and set up root’s key the same way.

Every command in this guide runs as root. Each code block is labelled with the machine it runs on.

Step 1. Prepare the jumpbox

Install the few tools you need and create a working directory.

apt-get update
apt-get install -y curl openssl openssh-client
mkdir -p ~/kthw

Now describe your machines in one small file. Each line holds the private IP address, the hostname and, for workers, the pod subnet that machine will hand out. Replace the example addresses with your own.

cd ~/kthw
cat > machines.txt <<'EOF'
10.240.0.10 server
10.240.0.20 node-0 10.200.0.0/24
10.240.0.21 node-1 10.200.1.0/24
EOF

Every later step reads this file, so you never have to type an IP address again.

Next, give each machine its hostname and teach all four machines each other’s names. Kubernetes identifies nodes by hostname, and the certificates you create later contain these names, so they have to match exactly.

cd ~/kthw
awk '{print $1, $2}' machines.txt > hosts
cat hosts >> /etc/hosts

while read IP HOST SUBNET; do
  ssh -n -o StrictHostKeyChecking=accept-new root@${IP} "hostnamectl set-hostname ${HOST}"
  scp hosts root@${IP}:/tmp/hosts
  ssh -n root@${IP} "cat /tmp/hosts >> /etc/hosts && rm /tmp/hosts"
done < machines.txt

for HOST in server node-0 node-1; do
  ssh -n -o StrictHostKeyChecking=accept-new root@${HOST} hostname
done

The -n flag matters. Without it, ssh reads the rest of machines.txt as its own input and the loop quietly stops after the first machine.

The last loop should print server, node-0 and node-1. If it does, name resolution and SSH both work.

Step 2. Download the binaries

Kubernetes is a handful of programs. You download each one once on the jumpbox, check it against the checksum its project publishes, and copy it out later.

cd ~/kthw
K8S_VERSION=v1.37.1
ETCD_VERSION=v3.7.2
CONTAINERD_VERSION=2.4.2
RUNC_VERSION=v1.5.2
CNI_VERSION=v1.9.1
CRICTL_VERSION=v1.37.0
ARCH=$(dpkg --print-architecture)
GH=https://github.com

mkdir -p downloads && cd downloads

for BIN in kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy kubectl; do
  curl -fsSLO "https://dl.k8s.io/${K8S_VERSION}/bin/linux/${ARCH}/${BIN}"
  echo "$(curl -fsSL "https://dl.k8s.io/${K8S_VERSION}/bin/linux/${ARCH}/${BIN}.sha256")  ${BIN}" | sha256sum --check
done

ETCD_TGZ=etcd-${ETCD_VERSION}-linux-${ARCH}.tar.gz
curl -fsSLO "${GH}/etcd-io/etcd/releases/download/${ETCD_VERSION}/${ETCD_TGZ}"
curl -fsSL "${GH}/etcd-io/etcd/releases/download/${ETCD_VERSION}/SHA256SUMS" | grep " ${ETCD_TGZ}$" | sha256sum --check

CONTAINERD_TGZ=containerd-${CONTAINERD_VERSION}-linux-${ARCH}.tar.gz
curl -fsSLO "${GH}/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/${CONTAINERD_TGZ}"
curl -fsSL "${GH}/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/${CONTAINERD_TGZ}.sha256sum" | sha256sum --check
curl -fsSL -o containerd.service "https://raw.githubusercontent.com/containerd/containerd/v${CONTAINERD_VERSION}/containerd.service"

curl -fsSLO "${GH}/opencontainers/runc/releases/download/${RUNC_VERSION}/runc.${ARCH}"
curl -fsSL "${GH}/opencontainers/runc/releases/download/${RUNC_VERSION}/runc.sha256sum" | grep " runc.${ARCH}$" | sha256sum --check
mv runc.${ARCH} runc

CNI_TGZ=cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgz
curl -fsSLO "${GH}/containernetworking/plugins/releases/download/${CNI_VERSION}/${CNI_TGZ}"
curl -fsSL "${GH}/containernetworking/plugins/releases/download/${CNI_VERSION}/${CNI_TGZ}.sha256" | sha256sum --check

CRICTL_TGZ=crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gz
curl -fsSLO "${GH}/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/${CRICTL_TGZ}"
echo "$(curl -fsSL "${GH}/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/${CRICTL_TGZ}.sha256")  ${CRICTL_TGZ}" | sha256sum --check

tar -xzf ${ETCD_TGZ} --strip-components=1 etcd-${ETCD_VERSION}-linux-${ARCH}/etcd etcd-${ETCD_VERSION}-linux-${ARCH}/etcdctl
chmod +x kube-* kubelet kubectl runc
install -m 0755 kubectl /usr/local/bin/kubectl

Every check should print OK. If one prints FAILED, delete that file and download it again. Never install a binary that failed its checksum.

Notice what you just downloaded. Five Kubernetes programs, a database, a container runtime, a low-level container launcher, some network plugins and the command-line tools that talk to them. That is the whole system. Everything else in Kubernetes runs as pods on top of these.

Step 3. Create the certificates

Every connection inside Kubernetes uses TLS, and almost every component proves who it is with a client certificate. The API server reads the certificate’s common name as the user name and the organisation field as the group. So the certificates are not only encryption. They are the identity system.

You start with a certificate authority (CA), a key pair that signs everything else. Then you create one certificate per identity. This whole block runs in a single shell session, because it defines a small helper function first.

cd ~/kthw
mkdir -p pki && cd pki

# The certificate authority that signs everything else
openssl genpkey -quiet -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out ca.key
openssl req -x509 -new -key ca.key -sha256 -days 3650 \
  -subj "/CN=kubernetes-ca" \
  -addext "basicConstraints=critical,CA:TRUE" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -out ca.crt

# make_cert NAME SUBJECT USAGE [SUBJECT_ALT_NAMES]
make_cert() {
  local name=$1 subject=$2 usage=$3 sans=$4
  openssl genpkey -quiet -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out "${name}.key"
  openssl req -new -key "${name}.key" -subj "${subject}" -out "${name}.csr"
  {
    echo "basicConstraints=critical,CA:FALSE"
    echo "keyUsage=critical,digitalSignature,keyEncipherment"
    echo "extendedKeyUsage=${usage}"
    if [ -n "${sans}" ]; then echo "subjectAltName=${sans}"; fi
  } > "${name}.ext"
  openssl x509 -req -in "${name}.csr" -CA ca.crt -CAkey ca.key \
    -days 365 -sha256 -extfile "${name}.ext" -out "${name}.crt"
  rm "${name}.csr" "${name}.ext"
}

SERVER_IP=$(awk '$2 == "server" {print $1}' ../machines.txt)

# Clients of the API server
make_cert admin "/CN=admin/O=system:masters" clientAuth
make_cert kube-controller-manager "/CN=system:kube-controller-manager" clientAuth
make_cert kube-scheduler "/CN=system:kube-scheduler" clientAuth
make_cert kube-proxy "/CN=system:kube-proxy" clientAuth

# The API server's own certificate, and the one it uses to call kubelets
make_cert kube-apiserver "/CN=kube-apiserver" serverAuth \
  "IP:10.32.0.1,IP:127.0.0.1,IP:${SERVER_IP},DNS:server,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.cluster.local"
make_cert apiserver-kubelet-client "/CN=kube-apiserver-kubelet-client" clientAuth

# One certificate per worker, used both as a client and as the kubelet's server
while read IP HOST SUBNET; do
  if [ "${HOST}" = "server" ]; then continue; fi
  make_cert "${HOST}" "/CN=system:node:${HOST}/O=system:nodes" serverAuth,clientAuth "DNS:${HOST},IP:${IP}"
done < ../machines.txt

# The key pair that signs and verifies service account tokens
openssl genpkey -quiet -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out service-account.key
openssl pkey -in service-account.key -pubout -out service-account.pub

ls

The names in those subjects are not decoration. Each one maps to a permission that already exists in every Kubernetes cluster.

• system:masters is a group with full access to everything. It is how admin becomes a cluster administrator.
• system:kube-controller-manager, system:kube-scheduler and system:kube-proxy are users that Kubernetes ships ready-made roles for.
• system:node:node-0 in the group system:nodes is how a kubelet identifies itself. The Node authorizer uses that exact pattern to let each kubelet read only what its own pods need.

The API server certificate lists every name and address a client might use to reach it. 10.32.0.1 is the first address in the service range, which becomes the kubernetes service that pods use to talk to the API. If a name is missing from this list, clients that use it get a TLS error.

Take a look at one of the worker certificates to see the identity inside it.

cd ~/kthw/pki
openssl x509 -in node-0.crt -noout -subject -ext subjectAltName

subject=CN=system:node:node-0, O=system:nodes
X509v3 Subject Alternative Name:
    DNS:node-0, IP Address:10.240.0.20

One more thing about system:masters. That group skips every permission check, so you cannot take its access away by changing roles. The only ways out are waiting for the certificate to expire or replacing the CA. That is fine for a lab. Real clusters keep this certificate locked away and give people narrower access.

Step 4. Create the kubeconfig files

A kubeconfig file bundles three things. Where the API server is, which CA to trust, and which certificate to log in with. Each component gets its own.

cd ~/kthw
mkdir -p configs && cd configs

# make_kubeconfig NAME USER SERVER_URL
make_kubeconfig() {
  local name=$1 user=$2 server=$3
  kubectl config set-cluster kthw --certificate-authority=../pki/ca.crt \
    --embed-certs=true --server="${server}" --kubeconfig="${name}.kubeconfig"
  kubectl config set-credentials "${user}" --client-certificate="../pki/${name}.crt" \
    --client-key="../pki/${name}.key" --embed-certs=true --kubeconfig="${name}.kubeconfig"
  kubectl config set-context default --cluster=kthw --user="${user}" --kubeconfig="${name}.kubeconfig"
  kubectl config use-context default --kubeconfig="${name}.kubeconfig"
}

make_kubeconfig node-0 system:node:node-0 https://server:6443
make_kubeconfig node-1 system:node:node-1 https://server:6443
make_kubeconfig kube-proxy system:kube-proxy https://server:6443
make_kubeconfig kube-controller-manager system:kube-controller-manager https://127.0.0.1:6443
make_kubeconfig kube-scheduler system:kube-scheduler https://127.0.0.1:6443
make_kubeconfig admin admin https://127.0.0.1:6443

The workers reach the API server by its hostname. The controller manager and scheduler live on the server itself, so they use the loopback address. The user name inside a kubeconfig is only a local label. The identity the API server sees always comes from the certificate.

Step 5. Encrypt secrets at rest

By default, Kubernetes stores Secrets in etcd without any encryption. Anyone who copies the etcd data directory can read every password in the cluster. An encryption config tells the API server to encrypt Secrets before it writes them.

cd ~/kthw/configs
ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)

cat > encryption-config.yaml <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - secretbox:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

The first provider in the list encrypts new writes. Of the providers that keep their key in a local file, secretbox is the one the Kubernetes documentation rates strong without a warning attached. Both AES options are marked as not recommended there. The identity provider at the end lets the API server still read anything written before encryption was turned on. In production you would keep this key in an external key management service instead of a file.

Step 6. Write the control plane services

The control plane is four programs, each run by systemd. You write their unit files on the jumpbox and copy them over in the next step.

etcd is the database. Every object in the cluster lives here and nowhere else.

mkdir -p ~/kthw/control-plane && cd ~/kthw/control-plane

cat > etcd.service <<'EOF'
[Unit]
Description=etcd
Documentation=https://etcd.io/docs/

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name=server \
  --data-dir=/var/lib/etcd \
  --listen-client-urls=http://127.0.0.1:2379 \
  --advertise-client-urls=http://127.0.0.1:2379 \
  --listen-peer-urls=http://127.0.0.1:2380 \
  --initial-advertise-peer-urls=http://127.0.0.1:2380 \
  --initial-cluster=server=http://127.0.0.1:2380 \
  --initial-cluster-state=new \
  --initial-cluster-token=kthw
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

This etcd listens only on the loopback address, so nothing outside the server can reach it, and that is why plain HTTP is acceptable here. A production control plane runs three or five etcd members across machines, and then every connection between them, and from the API server, needs TLS.

The API server is the front door. Every other component, and you, talk to the cluster only through it.

cd ~/kthw/control-plane
SERVER_IP=$(awk '$2 == "server" {print $1}' ../machines.txt)

cat > kube-apiserver.service <<'EOF'
[Unit]
Description=Kubernetes API Server
Documentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/
After=etcd.service
Wants=etcd.service

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
  --advertise-address=SERVER_IP \
  --allow-privileged=true \
  --authorization-mode=Node,RBAC \
  --client-ca-file=/var/lib/kubernetes/ca.crt \
  --enable-admission-plugins=NodeRestriction \
  --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \
  --etcd-servers=http://127.0.0.1:2379 \
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.crt \
  --kubelet-client-certificate=/var/lib/kubernetes/apiserver-kubelet-client.crt \
  --kubelet-client-key=/var/lib/kubernetes/apiserver-kubelet-client.key \
  --service-account-issuer=https://server:6443 \
  --service-account-key-file=/var/lib/kubernetes/service-account.pub \
  --service-account-signing-key-file=/var/lib/kubernetes/service-account.key \
  --service-cluster-ip-range=10.32.0.0/24 \
  --tls-cert-file=/var/lib/kubernetes/kube-apiserver.crt \
  --tls-private-key-file=/var/lib/kubernetes/kube-apiserver.key \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sed -i "s/SERVER_IP/${SERVER_IP}/" kube-apiserver.service

The heredoc is quoted so the shell leaves the line-ending backslashes alone, and sed fills in the one value that differs per cluster. A few flags are worth understanding.

• --authorization-mode=Node,RBAC turns on the two permission systems. Node limits each kubelet to its own pods. RBAC handles everyone else.
• --enable-admission-plugins=NodeRestriction limits each kubelet to changing its own node and its own pods, so a stolen kubelet credential cannot reach the rest of the cluster.
• --client-ca-file is what makes the client certificates from step 3 work as logins.
• The --kubelet-* flags let the API server open verified connections to kubelets. That is how kubectl logs and kubectl exec reach a container.

The controller manager runs the control loops. It notices that a Deployment wants three pods and only two exist, and creates the third.

cd ~/kthw/control-plane

cat > kube-controller-manager.service <<'EOF'
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/
After=kube-apiserver.service

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
  --bind-address=127.0.0.1 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/var/lib/kubernetes/ca.crt \
  --cluster-signing-key-file=/var/lib/kubernetes/ca.key \
  --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \
  --leader-elect=true \
  --root-ca-file=/var/lib/kubernetes/ca.crt \
  --service-account-private-key-file=/var/lib/kubernetes/service-account.key \
  --service-cluster-ip-range=10.32.0.0/24 \
  --use-service-account-credentials=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

--use-service-account-credentials=true gives each control loop its own identity instead of sharing one powerful one. When you read audit logs later, you can see which loop did what.

The scheduler decides which node each new pod runs on. It reads its settings from a small config file.

cd ~/kthw/control-plane

cat > kube-scheduler.yaml <<'EOF'
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /var/lib/kubernetes/kube-scheduler.kubeconfig
leaderElection:
  leaderElect: true
EOF

cat > kube-scheduler.service <<'EOF'
[Unit]
Description=Kubernetes Scheduler
Documentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/
After=kube-apiserver.service

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
  --config=/var/lib/kubernetes/kube-scheduler.yaml \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

Leader election matters once you run several control plane machines. Only one scheduler and one controller manager act at a time, and the others wait to take over. With one server it simply always wins.

Step 7. Start the control plane

Copy the binaries, certificates and configs to the server.

cd ~/kthw
scp downloads/etcd downloads/etcdctl downloads/kube-apiserver \
  downloads/kube-controller-manager downloads/kube-scheduler downloads/kubectl \
  pki/ca.crt pki/ca.key pki/kube-apiserver.crt pki/kube-apiserver.key \
  pki/apiserver-kubelet-client.crt pki/apiserver-kubelet-client.key \
  pki/service-account.key pki/service-account.pub \
  configs/kube-controller-manager.kubeconfig configs/kube-scheduler.kubeconfig \
  configs/admin.kubeconfig configs/encryption-config.yaml \
  control-plane/* \
  root@server:~/

Now log in to the server and put each file where its unit expects it.

cd ~
install -m 0755 etcd etcdctl kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/
rm etcd etcdctl kube-apiserver kube-controller-manager kube-scheduler kubectl

mkdir -p /var/lib/kubernetes /var/lib/etcd
chmod 700 /var/lib/etcd
mv ca.crt ca.key kube-apiserver.crt kube-apiserver.key \
  apiserver-kubelet-client.crt apiserver-kubelet-client.key \
  service-account.key service-account.pub \
  kube-controller-manager.kubeconfig kube-scheduler.kubeconfig \
  encryption-config.yaml kube-scheduler.yaml \
  /var/lib/kubernetes/
chmod 600 /var/lib/kubernetes/*.key /var/lib/kubernetes/*.kubeconfig /var/lib/kubernetes/encryption-config.yaml
mv etcd.service kube-apiserver.service kube-controller-manager.service kube-scheduler.service /etc/systemd/system/

systemctl daemon-reload
systemctl enable --now etcd
etcdctl endpoint health

127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.2ms

etcd is running. Start the rest of the control plane and wait until the API server reports ready.

systemctl enable --now kube-apiserver kube-controller-manager kube-scheduler

until kubectl --kubeconfig ~/admin.kubeconfig get --raw='/readyz' >/dev/null 2>&1; do sleep 2; done
kubectl --kubeconfig ~/admin.kubeconfig get --raw='/readyz?verbose' | tail -n 3

[+]poststarthook/apiservice-openapiv3-controller ok
[+]shutdown ok
readyz check passed

The controller manager and scheduler prove they are alive by holding a lease, a small object they renew every few seconds. It can take up to half a minute after startup before both appear.

kubectl --kubeconfig ~/admin.kubeconfig -n kube-system get leases

NAME                                   HOLDER                                                AGE
apiserver-xxxxxxxxxxxxxxxxxxxxxxxxxx   apiserver-xxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxx-xxxx-   30s
kube-controller-manager                server_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx           25s
kube-scheduler                         server_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx           25s

If a component is missing, journalctl -u kube-scheduler (or the matching unit name) shows why. Do not worry about a burst of “forbidden” errors in the first seconds of the scheduler’s log. The API server creates its built-in roles just after it starts, and the errors stop once it has.

Step 8. Connect kubectl from the jumpbox

From here on you drive the cluster from the jumpbox, like you would from your own laptop.

cd ~/kthw
kubectl config set-cluster kthw --certificate-authority=pki/ca.crt \
  --embed-certs=true --server=https://server:6443
kubectl config set-credentials admin --client-certificate=pki/admin.crt \
  --client-key=pki/admin.key --embed-certs=true
kubectl config set-context kthw --cluster=kthw --user=admin
kubectl config use-context kthw

kubectl version

Client Version: v1.37.1
Kustomize Version: v5.8.1
Server Version: v1.37.1

There is one permission left to grant. The API server calls each kubelet as the user kube-apiserver-kubelet-client, and by default that user may do nothing. Kubernetes ships a role for exactly this job, so you only bind it.

kubectl create clusterrolebinding kube-apiserver-kubelet-client \
  --clusterrole=system:kubelet-api-admin \
  --user=kube-apiserver-kubelet-client

Without this binding, kubectl logs and kubectl exec fail with a forbidden error later on, even though everything else works.

Step 9. Prepare the worker configs

A worker needs four things. A container runtime, a network for its pods, the kubelet and kube-proxy. You write their configs on the jumpbox first.

containerd is the runtime. It pulls images and asks runc to start containers.

mkdir -p ~/kthw/workers && cd ~/kthw/workers

cat > containerd-config.toml <<'EOF'
version = 4

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
  runtime_type = 'io.containerd.runc.v2'

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins.'io.containerd.cri.v1.runtime'.cni]
  bin_dirs = ['/opt/cni/bin']
  conf_dir = '/etc/cni/net.d'
EOF

cat > crictl.yaml <<'EOF'
runtime-endpoint: unix:///run/containerd/containerd.sock
EOF

SystemdCgroup = true makes containerd and the kubelet agree on who manages resource limits. On a systemd machine both should use systemd. containerd’s own default is false, so this line matters. When the two disagree, the machine ends up with two views of what is using its resources, and the node can become unstable under pressure.

The kubelet is the agent on every worker. It watches the API server for pods assigned to its node and makes them run.

cd ~/kthw/workers

cat > kubelet-config.yaml <<'EOF'
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: /var/lib/kubelet/ca.crt
authorization:
  mode: Webhook
cgroupDriver: systemd
clusterDomain: cluster.local
clusterDNS:
  - 10.32.0.10
containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
resolvConf: /etc/resolv.conf
tlsCertFile: /var/lib/kubelet/kubelet.crt
tlsPrivateKeyFile: /var/lib/kubelet/kubelet.key
EOF

cat > kubelet.service <<'EOF'
[Unit]
Description=Kubernetes Kubelet
Documentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \
  --config=/var/lib/kubelet/kubelet-config.yaml \
  --kubeconfig=/var/lib/kubelet/kubeconfig \
  --node-ip=NODE_IP \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

The authentication and authorization blocks lock the kubelet down. Anonymous requests are refused, and every request is checked with the API server first. Without them, anyone who can reach the kubelet’s port could run commands inside your containers.

--node-ip tells the kubelet which address to publish for its node. Debian maps every machine’s own hostname to 127.0.1.1, and without this flag the kubelet could pick the wrong address.

kube-proxy turns Services into packet rules. When a pod connects to a service address, those rules send the packet to one of the pods behind it.

cd ~/kthw/workers

cat > kube-proxy-config.yaml <<'EOF'
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clientConnection:
  kubeconfig: /var/lib/kube-proxy/kubeconfig
mode: nftables
clusterCIDR: 10.200.0.0/16
EOF

cat > kube-proxy.service <<'EOF'
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \
  --config=/var/lib/kube-proxy/kube-proxy-config.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

We use the nftables mode. It is the modern replacement for the older iptables mode and matches the firewall framework Debian 13 uses.

Step 10. Plan the pod network

This is the part most installers hide completely, and the part people find hardest when it breaks.

Kubernetes networking rests on one core rule. Every pod gets its own IP address, and every pod can reach every other pod at that address without any address translation in between. Kubernetes itself does not implement this. It hands the job to a CNI plugin.

Real clusters use plugins like Cilium or Calico. Here you will do it by hand with the basic bridge plugin, so you can see what those plugins automate.

Each worker owns one slice of the pod range from machines.txt. The bridge plugin creates a virtual switch called cni0 on the worker and gives each pod an address from that worker’s slice.

cd ~/kthw/workers

cat > 10-bridge.conflist <<'EOF'
{
  "cniVersion": "1.0.0",
  "name": "bridge",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": false,
      "hairpinMode": true,
      "ipam": {
        "type": "host-local",
        "ranges": [[{ "subnet": "POD_SUBNET" }]],
        "routes": [{ "dst": "0.0.0.0/0" }]
      }
    },
    {
      "type": "portmap",
      "capabilities": { "portMappings": true }
    }
  ]
}
EOF

That covers pods on the same worker. Pods on different workers need two more things.

First, each machine needs a route to the other workers’ pod slices. A packet for 10.200.1.5 has to go to node-1, because only node-1 knows that pod.

Second, pods need to reach the internet, for example to resolve outside DNS names. Pod addresses are private, so traffic leaving the cluster must be translated to the worker’s own address. Traffic between pods must not be translated, or the receiving pod sees the worker’s address instead of the real sender. That is why ipMasq is off in the bridge config above. Its built-in rule would also translate traffic going to pods on other workers. Instead, you add one precise rule.

cd ~/kthw/workers

cat > pod-nat.nft <<'EOF'
# Translate pod traffic that leaves the cluster, and nothing else.
table ip pod-nat
delete table ip pod-nat
table ip pod-nat {
  chain postrouting {
    type nat hook postrouting priority srcnat; policy accept;
    ip saddr 10.200.0.0/16 ip daddr != 10.200.0.0/16 masquerade
  }
}
EOF

The first two lines make the file safe to load twice. They create the table if it is missing and then delete it, so the definition below always starts from scratch.

Routes added by hand disappear on reboot, so you wrap the routes and the rule in a small systemd service per machine. The loop reads machines.txt and writes one route per other worker.

cd ~/kthw/workers

while read IP HOST SUBNET; do
  {
    cat <<'EOF'
[Unit]
Description=Pod network routes and NAT
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
EOF
    while read PEER_IP PEER_HOST PEER_SUBNET; do
      if [ -n "${PEER_SUBNET}" ] && [ "${PEER_HOST}" != "${HOST}" ]; then
        echo "ExecStart=/usr/sbin/ip route replace ${PEER_SUBNET} via ${PEER_IP}"
      fi
    done < ../machines.txt
    if [ -n "${SUBNET}" ]; then
      echo "ExecStart=/usr/sbin/nft -f /etc/kubernetes/pod-nat.nft"
    fi
    cat <<'EOF'

[Install]
WantedBy=multi-user.target
EOF
  } > pod-network-${HOST}.service
done < ../machines.txt

cat pod-network-node-0.service

[Unit]
Description=Pod network routes and NAT
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/ip route replace 10.200.1.0/24 via 10.240.0.21
ExecStart=/usr/sbin/nft -f /etc/kubernetes/pod-nat.nft

[Install]
WantedBy=multi-user.target

The server gets routes to both workers’ slices but no translation rule, because no pods run there. These routes only work because all machines share one network. Across networks or clouds, this is exactly the problem an overlay plugin like Cilium or Flannel solves.

Step 11. Copy everything to the workers

Fill in the per-worker values and copy each worker its files.

cd ~/kthw

while read IP HOST SUBNET; do
  if [ "${HOST}" = "server" ]; then continue; fi
  mkdir -p workers/${HOST}
  sed "s|POD_SUBNET|${SUBNET}|" workers/10-bridge.conflist > workers/${HOST}/10-bridge.conflist
  sed "s|NODE_IP|${IP}|" workers/kubelet.service > workers/${HOST}/kubelet.service

  scp downloads/kubelet downloads/kube-proxy downloads/runc \
    downloads/containerd-*.tar.gz downloads/cni-plugins-*.tgz downloads/crictl-*.tar.gz \
    downloads/containerd.service \
    pki/ca.crt pki/${HOST}.crt pki/${HOST}.key \
    configs/${HOST}.kubeconfig configs/kube-proxy.kubeconfig \
    workers/containerd-config.toml workers/crictl.yaml workers/kubelet-config.yaml \
    workers/kube-proxy-config.yaml workers/kube-proxy.service workers/pod-nat.nft \
    workers/${HOST}/10-bridge.conflist workers/${HOST}/kubelet.service \
    workers/pod-network-${HOST}.service \
    root@${HOST}:~/
done < machines.txt

scp workers/pod-network-server.service root@server:~/

Turn on the server’s routes now.

mv ~/pod-network-server.service /etc/systemd/system/pod-network.service
systemctl daemon-reload
systemctl enable --now pod-network
ip route | grep 10.200

10.200.0.0/24 via 10.240.0.20 dev eth0
10.200.1.0/24 via 10.240.0.21 dev eth0

Step 12. Bring up the workers

Run the next three blocks on node-0, then on node-1. They are identical on both.

First the operating system settings. The kubelet refuses to start while swap is on, and pod networking needs two kernel modules and packet forwarding.

apt-get update
apt-get install -y conntrack nftables

swapoff -a
sed -i '/\sswap\s/ s/^/#/' /etc/fstab

cat > /etc/modules-load.d/kubernetes.conf <<'EOF'
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter

cat > /etc/sysctl.d/99-kubernetes.conf <<'EOF'
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sysctl --system

br_netfilter is subtle. Pods on the same worker talk through the cni0 bridge, and by default bridged traffic skips the host’s packet rules entirely. The module and the two bridge-nf-call settings make it pass through them, which kube-proxy needs when a pod talks to a service backed by a pod on the same worker.

Next the container runtime and the pod network.

cd ~
tar -xzf containerd-*.tar.gz -C /usr/local
install -m 0755 runc /usr/local/sbin/runc
mkdir -p /opt/cni/bin /etc/cni/net.d /etc/containerd /etc/kubernetes
tar -xzf cni-plugins-*.tgz -C /opt/cni/bin
tar -xzf crictl-*.tar.gz -C /usr/local/bin
rm containerd-*.tar.gz cni-plugins-*.tgz crictl-*.tar.gz runc

mv containerd-config.toml /etc/containerd/config.toml
mv crictl.yaml /etc/crictl.yaml
mv 10-bridge.conflist /etc/cni/net.d/
mv pod-nat.nft /etc/kubernetes/pod-nat.nft
mv containerd.service /etc/systemd/system/containerd.service
mv pod-network-$(hostname).service /etc/systemd/system/pod-network.service

systemctl daemon-reload
systemctl enable --now containerd pod-network
crictl info >/dev/null && echo "containerd is ready"

Finally the kubelet and kube-proxy.

cd ~
install -m 0755 kubelet kube-proxy /usr/local/bin/
rm kubelet kube-proxy
mkdir -p /var/lib/kubelet /var/lib/kube-proxy

mv ca.crt /var/lib/kubelet/ca.crt
mv $(hostname).crt /var/lib/kubelet/kubelet.crt
mv $(hostname).key /var/lib/kubelet/kubelet.key
mv $(hostname).kubeconfig /var/lib/kubelet/kubeconfig
mv kubelet-config.yaml /var/lib/kubelet/kubelet-config.yaml
mv kube-proxy.kubeconfig /var/lib/kube-proxy/kubeconfig
mv kube-proxy-config.yaml /var/lib/kube-proxy/kube-proxy-config.yaml
chmod 600 /var/lib/kubelet/kubelet.key /var/lib/kubelet/kubeconfig /var/lib/kube-proxy/kubeconfig

# When systemd-resolved runs, /etc/resolv.conf points at a local stub that pods cannot use
if systemctl is-active --quiet systemd-resolved; then
  sed -i 's|^resolvConf: .*|resolvConf: /run/systemd/resolve/resolv.conf|' /var/lib/kubelet/kubelet-config.yaml
fi

mv kubelet.service kube-proxy.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now kubelet kube-proxy

The resolv.conf check guards against a classic failure. When systemd-resolved is running, /etc/resolv.conf sends queries to 127.0.0.53. Inside a pod that address is the pod itself, so CoreDNS would forward queries to itself, detect the loop and crash. Pointing the kubelet at the real upstream servers avoids it. kubeadm makes exactly the same check. Only test whether the service is running, not whether its file exists. A leftover file can vanish at the next reboot, and then no pod on that worker can start.

Back on the jumpbox, wait for both workers to register and report ready.

kubectl wait --for=create node/node-0 node/node-1 --timeout=120s
kubectl wait --for=condition=Ready node/node-0 node/node-1 --timeout=180s
kubectl get nodes

NAME     STATUS   ROLES    AGE   VERSION
node-0   Ready    <none>   30s   v1.37.1
node-1   Ready    <none>   30s   v1.37.1

The first wait is needed because a node object only exists once its kubelet has registered it, and kubectl wait cannot wait for the condition of an object that does not exist yet. Add -o wide to the last command to see each node’s address, operating system and container runtime.

Notice that the server is not in this list. It runs no kubelet, so it is not a node. Its control plane programs run as plain systemd services, outside Kubernetes.

Step 13. Add cluster DNS

Pods find services by name, like nginx.default.svc.cluster.local. CoreDNS answers those questions. It is the first thing you install the normal way, as pods, using the cluster you just built.

The Kubernetes repository ships a CoreDNS manifest template. You fill in the cluster domain, the DNS service address you gave the kubelets and a memory limit.

cd ~/kthw
curl -fsSL https://raw.githubusercontent.com/kubernetes/kubernetes/v1.37.1/cluster/addons/dns/coredns/coredns.yaml.base \
  | sed -e 's/__DNS__DOMAIN__/cluster.local/g' \
        -e 's/__DNS__SERVER__/10.32.0.10/g' \
        -e 's/__DNS__MEMORY__LIMIT__/170Mi/g' \
  > coredns.yaml

kubectl apply -f coredns.yaml
kubectl -n kube-system rollout status deployment/coredns --timeout=180s
kubectl -n kube-system get pods -o wide

NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
coredns-xxxxxxxxxx-xxxxx   1/1     Running   0          20s   10.200.1.2   node-1   <none>           <none>

That pod is the first proof your cluster works end to end. The scheduler picked a node, the kubelet asked containerd to pull the image and start it, the bridge plugin gave it an address, and CoreDNS used its service account token to read services from the API server through the kubernetes service.

Step 14. Prove that it works

Run each test from the jumpbox. Each one exercises a different part of what you built.

Secrets are encrypted in etcd. Create a Secret, then read it straight from the database on the server.

kubectl create secret generic kthw --from-literal=mykey=mydata
ssh root@server "etcdctl get /registry/secrets/default/kthw --print-value-only | head -c 26"; echo

k8s:enc:secretbox:v1:key1:

The stored value starts with the provider and key name, followed by encrypted bytes. Your password is not in there in readable form.

Deployments schedule onto both workers.

kubectl create deployment nginx --image=nginx:1.30 --replicas=2
kubectl rollout status deployment/nginx --timeout=180s
kubectl get pods -l app=nginx -o wide

Pods reach pods on other workers, without translation. Start a small client pod and call each nginx pod by its address.

kubectl run client --image=busybox:1.37 --restart=Never -- sleep 3600
kubectl wait --for=condition=Ready pod/client --timeout=120s

for POD_IP in $(kubectl get pods -l app=nginx -o jsonpath='{.items[*].status.podIP}'); do
  kubectl exec client -- wget -qO- -T 5 http://${POD_IP} | grep -m1 -o 'Welcome to nginx!'
done

kubectl get pod client -o jsonpath='{.status.podIP}{"\n"}'
kubectl logs -l app=nginx --tail=1

The nginx access logs show the client pod’s own address as the caller. That is the networking rule from step 10, working.

Services and DNS work.

kubectl expose deployment nginx --port=80
kubectl exec client -- nslookup nginx.default.svc.cluster.local
kubectl exec client -- wget -qO- -T 5 http://nginx | grep -m1 -o 'Welcome to nginx!'
kubectl exec client -- nslookup kubernetes.io

The first lookup is answered by CoreDNS from cluster data. The short name nginx works because the kubelet gave the pod a search list. The last lookup goes out to the internet through the translation rule.

Port forwarding, logs and exec work. All three travel from kubectl to the API server and on to the kubelet.

kubectl port-forward deployment/nginx 8080:80 >/dev/null &
PF_PID=$!
sleep 3
curl -s http://127.0.0.1:8080 | grep -m1 -o 'Welcome to nginx!'
kill ${PF_PID}

kubectl logs deployment/nginx --tail=2
kubectl exec deployment/nginx -- nginx -v

NodePort services are reachable from outside the cluster.

kubectl expose deployment nginx --name=nginx-nodeport --type=NodePort --port=80
NODE_PORT=$(kubectl get service nginx-nodeport -o jsonpath='{.spec.ports[0].nodePort}')
curl -s --retry 5 --retry-connrefused http://node-0:${NODE_PORT} | grep -m1 -o 'Welcome to nginx!'
curl -s --retry 5 --retry-connrefused http://node-1:${NODE_PORT} | grep -m1 -o 'Welcome to nginx!'

kube-proxy needs a moment to write the rules for a new service, which is why curl retries if the first attempt is refused. Both workers answer, even one that runs no nginx pod at all, because kube-proxy on every worker knows where the pods are.

If all of these printed Welcome to nginx! and real addresses, you have built a working Kubernetes cluster from scratch.

When something does not work

Most problems show up in a component’s logs. On the machine that runs it, journalctl -u <unit> -e shows the latest lines. A few failures are common enough to name.

• A node never becomes Ready. Check journalctl -u kubelet on that worker. A certificate whose name does not match the hostname is the usual cause.
• Pods stay in ContainerCreating. Run kubectl describe pod and read the events. Network setup errors point at /etc/cni/net.d or /opt/cni/bin.
• Cross-node traffic times out. Run ip route on each worker and check that the routes from step 10 exist. Some cloud providers also drop traffic for addresses they did not assign to the machine. AWS calls the setting the source and destination check, and Google Cloud calls it IP forwarding.
• kubectl logs fails with Forbidden (user=kube-apiserver-kubelet-client, ...). The role binding from step 8 is missing. Kubelets remember permission answers for a few minutes, so a fix can take that long to show.
• CoreDNS keeps restarting. Its logs usually mention a loop. Recheck the resolvConf step on the workers.

What this cluster does not have

You now understand the moving parts better than most people who run Kubernetes every day. It is also worth being honest about what a production cluster needs on top of this.

• No high availability. One server and one etcd member means one machine failure takes the control plane down. Production runs three control plane machines behind a load balancer.
• No backups. Nothing here snapshots etcd. Lose the server’s disk and the cluster’s state is gone.
• Certificates expire in a year. Every one of them must be renewed and redistributed before then, without downtime.
• Upgrades are manual. Kubernetes ships three minor releases a year, and each is supported for about fourteen months. Every component must be upgraded in the right order, every few months, forever.
• No network policies, ingress, storage or monitoring. Each is another component to choose, install, configure and keep up to date.

None of this is out of reach. You could add each piece by hand, the same way you built the rest. That is a good way to learn, and a hard way to run a business.

The easy way, once you know the hard way

This is the work we built Ankra to take off your hands. Ankra creates clusters on Hetzner, OVHcloud, UpCloud, DigitalOcean, Scaleway, AWS or your own Proxmox hardware, using kubeadm or k3s under the hood. Underneath sit the same certificates, etcd, control plane and workers you just set up by hand, with three control plane machines from day one if you want high availability. Ankra can also import a cluster you already run.

From there the day-2 work becomes routine. A Kubernetes version upgrade is a platform operation, not a weekend of SSH sessions. Add-ons like ingress, certificates, monitoring and databases are pinned, versioned parts of a stack, stored in your own Git repository. Secrets stay encrypted in Git. And when something breaks, Ankra’s AI reads the same logs, events and metrics you learned to read in this guide, and tells you what it found.

The knowledge from today still pays off. When Ankra flags a node that is not ready, or a pod stuck in ContainerCreating, you know which pieces sit underneath and where to look.

If you want a cluster to keep practising on without four machines, every Ankra organisation gets a free Trial Playground, and the free plan includes 30 vCPU of worker capacity for real clusters.

When you are done with your hard-way cluster, delete the four machines. Then build it again in a week without looking at this guide. That second time is when it really sticks.

Get started: Create a free account on Ankra.

Join our community: Slack

Follow us on: LinkedIn | GitHub

Contact us: hello@ankra.ai