k3s HA on Raspberry Pi: Zero-Downtime Cluster in 45 Minutes (Multi-Master + MySQL)


estimated read time: 5 minutes

Why High Availability?

The standard K3s setup — the kind covered in the earlier Pi Cluster series — runs a single master node. That node holds the Kubernetes API server, the scheduler, and the controller manager. If it goes down, the cluster stops accepting new workloads. Existing pods keep running, but you cannot deploy, scale, or inspect anything until the master recovers.

For a homelab this is usually fine. But if you are running services that need to stay up even during maintenance or an unexpected reboot — or if you just want to learn how production-grade Kubernetes handles control plane redundancy — HA is the answer.

K3s supports HA through an external datastore. Instead of the default embedded SQLite (which is node-local and cannot be shared), you point multiple K3s server nodes at the same MySQL database. Each node is a full control plane member. An Nginx load balancer in front of them distributes API server traffic and provides a single stable endpoint for worker nodes.

Architecture

Worker nodes
  └─▶ Nginx (port 6443)
        ├─▶ K3s Master Node 1 (port 6443)
        └─▶ K3s Master Node 2 (port 6443)
              Both connected to: MySQL (homelab DB)

Prerequisites

Step 1: Set Up MySQL

On your database host:

sudo apt install mysql-server
sudo systemctl start mysql.service
sudo systemctl enable mysql.service

Log into MySQL and create the K3s database and user. Replace the network range with your actual home subnet:

CREATE DATABASE homelab;
CREATE USER 'k8s' IDENTIFIED BY 'your_strong_password';
GRANT ALL PRIVILEGES ON *.* TO 'k8s'@'10.0.0.0/255.255.255.0';
FLUSH PRIVILEGES;

The grant uses a subnet mask rather than a single IP so that both master nodes can connect using the same user account, regardless of their individual IPs.

Step 2: Install K3s on Both Master Nodes

Run this on each master node, substituting SQL_DB_IP with the IP of your MySQL host and the password you set:

curl -sfL https://get.k3s.io | sh -s - server \
  --datastore-endpoint="mysql://k8s:your_strong_password@tcp(SQL_DB_IP:3306)/homelab"

K3s will install, connect to MySQL, and register itself as a control plane member. Run the same command on the second master — it will join the same datastore and both will participate in leader election.

Verify both masters are ready:

kubectl get nodes

You should see two nodes with the control-plane,master role.

Step 3: Set Up the Nginx Load Balancer

Install Nginx on a separate node (or one of the masters if resources are tight):

sudo apt install nginx

Edit /etc/nginx/nginx.conf. The key addition is a stream block, which proxies raw TCP rather than HTTP — this is important because the Kubernetes API server uses its own TLS and Nginx should not try to terminate it:

stream {
  server {
    listen 6443;
    proxy_pass stream_master_nodes;
  }

  upstream stream_master_nodes {
    server 10.0.0.X:6443;   # IP of master node 1
    server 10.0.0.Y:6443;   # IP of master node 2
  }
}

Replace the IPs with your actual master node addresses, then reload Nginx:

sudo nginx -t
sudo systemctl reload nginx

Step 4: Join Worker Nodes

On each worker node, point K3s at the Nginx load balancer rather than a specific master:

curl -sfL https://get.k3s.io | sh -s - agent \
  --server http://NGINX_LB_IP:6443

Workers now connect through the load balancer. If a master goes down, Nginx will route new connections to the remaining healthy master — the workers stay connected.

Testing Failover

With both masters and workers running, try stopping K3s on one master:

sudo systemctl stop k3s

On another machine, verify the cluster is still responding:

kubectl get nodes
kubectl get pods -A

The stopped master will show NotReady, but the cluster continues to function. Workloads keep running and the API server is reachable through the load balancer. Bring the master back with sudo systemctl start k3s and it will rejoin automatically.

What This Adds to the Pi Cluster Series

The original Pi Cluster setup builds a solid single-master K3s cluster. This extends it to handle control plane failures gracefully:

TopologyControl plane failureWorkloads affected
Single masterCluster API unavailableExisting pods run, no new deploys
HA (this guide)Leader election, failover in secondsNone — transparent to workloads

The trade-offs are real: you need more hardware, a MySQL instance to maintain, and an Nginx LB. For a homelab running critical services it is worth it. For a pure learning environment, single master is simpler.

Pi Cluster Series