k3s HA on Raspberry Pi: Zero-Downtime Cluster in 45 Minutes (Multi-Master + MySQL)

Why High Availability?

The standard K3s setup — the kind covered in the earlier Pi Cluster series — runs a single master node. That node holds the Kubernetes API server, the scheduler, and the controller manager. If it goes down, the cluster stops accepting new workloads. Existing pods keep running, but you cannot deploy, scale, or inspect anything until the master recovers.

For a homelab this is usually fine. But if you are running services that need to stay up even during maintenance or an unexpected reboot — or if you just want to learn how production-grade Kubernetes handles control plane redundancy — HA is the answer.

K3s supports HA through an external datastore. Instead of the default embedded SQLite (which is node-local and cannot be shared), you point multiple K3s server nodes at the same MySQL database. Each node is a full control plane member. An Nginx load balancer in front of them distributes API server traffic and provides a single stable endpoint for worker nodes.

Architecture

Worker nodes
  └─▶ Nginx (port 6443)
        ├─▶ K3s Master Node 1 (port 6443)
        └─▶ K3s Master Node 2 (port 6443)
              Both connected to: MySQL (homelab DB)

Embedded etcd vs External Database: Why MySQL?

Modern k3s (v1.19+) supports two HA approaches:

1. Embedded etcd — k3s runs its own etcd cluster on the master nodes (no external database needed)
2. External datastore — point multiple masters at a shared MySQL, PostgreSQL, or MariaDB instance

This guide uses external MySQL because:

Pros of External MySQL

• Simpler disaster recovery — database backups are a solved problem. mysqldump, replication, point-in-time recovery all work out of the box.
• Familiar tooling — if you already run MySQL for other services (Home Assistant, Grafana, etc.), you can reuse that knowledge and infrastructure.
• Separate failure domain — database can live on a VM with its own snapshot/backup schedule, independent of k3s node lifecycles.
• Lower resource usage on Pi nodes — etcd is CPU and disk-I/O intensive. Offloading it to a beefier VM or dedicated node keeps Pi resources free for workloads.

Cons of External MySQL

• Single point of failure — if MySQL goes down, the entire control plane stops. You'd need MySQL replication or a managed DB to eliminate this (which adds complexity).
• Network dependency — masters need stable connectivity to the DB. A network partition can take down the control plane even if all masters are healthy.
• One more thing to maintain — you're now responsible for MySQL upgrades, security patches, and backups.

When to Use Embedded etcd Instead

Embedded etcd makes sense if:

• You want true distributed consensus with no single database bottleneck
• You're comfortable managing etcd directly (snapshots, compaction, defrag)
• Your Pi nodes have good SD cards or NVMe storage (etcd is sensitive to slow disks)
• You prefer fewer moving parts (no external DB to secure/backup)

For this homelab: I already had MySQL running for other services, backups were automated, and I wanted to keep Pi CPU/disk usage low. If you're starting fresh and comfortable with etcd, the embedded approach is cleaner — just add --cluster-init to the first master and --server https://master-1:6443 to the others.

Bottom line: External MySQL was the pragmatic choice for this setup. Embedded etcd is architecturally cleaner but adds operational complexity I didn't want in a homelab. Pick the one that fits your existing infrastructure and comfort level.

Prerequisites

• Two Raspberry Pi 4 (4GB or 8GB) or equivalent nodes for the master role
• Two additional nodes for workers (Pi 4 or Proxmox VMs work fine)
• A machine to host MySQL — this can be a Proxmox VM or a dedicated node
• Ubuntu 23.04 LTS on each node
• A machine to run Nginx as the load balancer (can be one of the above)

Step 1: Set Up MySQL

On your database host:

sudo apt install mysql-server
sudo systemctl start mysql.service
sudo systemctl enable mysql.service

Log into MySQL and create the K3s database and user. Replace the network range with your actual home subnet:

CREATE DATABASE homelab;
CREATE USER 'k8s' IDENTIFIED BY 'your_strong_password';
GRANT ALL PRIVILEGES ON *.* TO 'k8s'@'10.0.0.0/255.255.255.0';
FLUSH PRIVILEGES;

The grant uses a subnet mask rather than a single IP so that both master nodes can connect using the same user account, regardless of their individual IPs.

Step 2: Install K3s on Both Master Nodes

Run this on each master node, substituting SQL_DB_IP with the IP of your MySQL host and the password you set:

curl -sfL https://get.k3s.io | sh -s - server \
  --datastore-endpoint="mysql://k8s:your_strong_password@tcp(SQL_DB_IP:3306)/homelab"

K3s will install, connect to MySQL, and register itself as a control plane member. Run the same command on the second master — it will join the same datastore and both will participate in leader election.

Verify both masters are ready:

kubectl get nodes

You should see two nodes with the control-plane,master role.

Step 3: Set Up the Nginx Load Balancer

Install Nginx on a separate node (or one of the masters if resources are tight):

sudo apt install nginx

Edit /etc/nginx/nginx.conf. The key addition is a stream block, which proxies raw TCP rather than HTTP — this is important because the Kubernetes API server uses its own TLS and Nginx should not try to terminate it:

stream {
  server {
    listen 6443;
    proxy_pass stream_master_nodes;
  }

  upstream stream_master_nodes {
    server 10.0.0.X:6443;   # IP of master node 1
    server 10.0.0.Y:6443;   # IP of master node 2
  }
}

Replace the IPs with your actual master node addresses, then reload Nginx:

sudo nginx -t
sudo systemctl reload nginx

Step 4: Join Worker Nodes

On each worker node, point K3s at the Nginx load balancer rather than a specific master:

curl -sfL https://get.k3s.io | sh -s - agent \
  --server http://NGINX_LB_IP:6443

Workers now connect through the load balancer. If a master goes down, Nginx will route new connections to the remaining healthy master — the workers stay connected.

Testing Failover

With both masters and workers running, try stopping K3s on one master:

sudo systemctl stop k3s

On another machine, verify the cluster is still responding:

kubectl get nodes
kubectl get pods -A

The stopped master will show NotReady, but the cluster continues to function. Workloads keep running and the API server is reachable through the load balancer. Bring the master back with sudo systemctl start k3s and it will rejoin automatically.

What This Adds to the Pi Cluster Series

The original Pi Cluster setup builds a solid single-master K3s cluster. This extends it to handle control plane failures gracefully:

The trade-offs are real: you need more hardware, a MySQL instance to maintain, and an Nginx LB. For a homelab running critical services it is worth it. For a pure learning environment, single master is simpler.

---

Pi Cluster Series

• Part 1 — Introduction: Pi Cluster
• Part 2 — Setting Up the Cluster
• Part 3 — Persistent Storage with Longhorn
• Part 4 — Offsite Backups with AWS S3
• Part 5 — Load Balancing with MetalLB
• Part 6 — Ingress and TLS with Traefik v3
• Part 7 — Edge Protection with Cloudflare
• Part 8 — High Availability Control Plane ← you are here