The Cluster That Almost Wasn't

Build, Break & Operate · Ship Boring Lab · Stage 1.

I set out to build a small Kubernetes cluster from scratch:

2 VMs → Ansible hardening → pinned Kubernetes v1.31 → `kubeadm init` + `join`

It sounded straightforward.

It wasn't.

What fought back

1. SSH — three rounds of debugging

The OrbStack VMs didn't have openssh-server installed.

Then I hit two more problems:

• The machines were running in isolated-machine mode
• The configured username was `spk`, not ubuntu

The breakthrough came through /mnt/mac for copying the required SSH key.

The fingerprints matched, but the server still refused the connection.

The sshd logs finally told me why.

Lesson: Before debugging SSH authentication, make sure there is actually an SSH server to authenticate against.

2. The clone trap

sb-node1 was created as a clone of sb-cp.

That meant it inherited:

• SSH host keys
• machine-id

Two machines should not share the same identity.

I regenerated the machine identity and SSH host keys.

New fingerprint.

Connection worked.

Lesson: A cloned VM isn't automatically a new machine.

3. `swapoff -a` wasn't actually a failure

Ansible reported:

swapoff -a

The catch?

There was no swap.

The machine was already in the desired state.

The automation needed to understand:

Already compliant = success.

That's the difference between a script that runs and automation that is actually idempotent.

4. Ansible exposed another assumption

This:

changed_when: 'stdout'

ran into problems with newer ansible-core.

The more portable pattern is to explicitly register the command result:

register: result

and evaluate the registered result.

Another small failure, another assumption removed.

5. One tab broke YAML

One tab inside a YAML heredoc was enough to kill the file.

Now this is part of the debugging reflex:

grep -Pn '\t'

Sometimes the problem isn't Kubernetes.

Sometimes it's one character.

The Win

Eventually:

kubectl get nodes

returned the state I was after:

sb-cp       Ready
sb-node1    Ready

2/2 nodes Ready.

Kubernetes v1.31.4.

Running locally.

Zero dollars.

And every line of the setup is mine.

What I actually learned

The Kubernetes installation wasn't the interesting part.

The interesting part was debugging the assumptions underneath it:

A VM isn't necessarily SSH-ready. A cloned machine isn't necessarily uniquely identified. A non-zero return code doesn't always mean the system is wrong. Automation needs to model desired state, not just command output. YAML doesn't care how small the mistake looks.

The automation was the easy part. The debugging is the skill.

That's what I want to capture with Ship Boring Lab.

Not just the final architecture.

The failures, the assumptions, the fixes, and the reasoning that gets the system from "should work" to actually working.