Crafting a Local Kubernetes Cluster Using K3S and Raspberry Pies

Introduction

Recently, I stumbled upon a video by NetworkChuck detailing the process of constructing a Kubernetes cluster using Raspberry Pis. It’s a good watch, I can’t recommend it enough. Since I had a few Raspberry Pis gathering dust from a previous endeavor, coupled with a spare mini-computer, I thought, “Why not give it a try?” This post will guide you through my journey of setting up K3S — a lightweight iteration of Kubernetes. While my setup doesn’t mirror NetworkChuck’s to the letter, the foundational logic remains similar, making it easy for you to follow along.

Components

Though my configuration might not be a carbon copy of NetworkChuck’s, the principle stays consistent. Here’s a breakdown of the devices I’ve employed:

PCs:

• Mini PC (8GB DDR4, 256GB SSD) powered by a Gemini Lake J4105 processor.
• Two Raspberry Pi 4 units, each boasting 4GB of RAM.
• A Raspberry Pi 3, equipped with 2GB RAM.
• Raspberry Pi Model B+ v1.2 with 500MB RAM (Note: I faced challenges when attempting to install k3s on this model).

Network Infrastructure:

• TP-Link AC1200: A versatile dual-band wireless Wi-Fi router.
• TP-Link 16-Port Desktop Switch: A robust Gigabit Ethernet hub that catered to my connectivity needs.

Master Node Setup

I used the Mini PC as a master node, installing it was incredibly easy. The Mini PC is running Ubuntu 22.04,

Prerequisite Ubuntu / Debian​

It is recommended to turn off UFW (uncomplicated firewall):

ufw disable

If you wish to keep ufw enabled, by default, the following rules are required:

ufw allow 6443/tcp #apiserver
ufw allow from 10.42.0.0/16 to any #pods
ufw allow from 10.43.0.0/16 to any #services

With that in place, the only other command was:

curl -sfL https://get.k3s.io | sh -

For more information on k3s check out their documentation https://docs.k3s.io/quick-start

After the installation is complete, you can execute kubectl to retrieve the following output.

NAMESPACE     NAME                                     READY   STATUS      RESTARTS   AGE
kube-system   local-path-provisioner-957fdf8bc-z7srp   1/1     Running     0          20h
kube-system   coredns-77ccd57875-zpsrh                 1/1     Running     0          20h
kube-system   metrics-server-648b5df564-mt62g          1/1     Running     0          20h
kube-system   helm-install-traefik-crd-r7vn9           0/1     Completed   0          20h
kube-system   helm-install-traefik-t5pst               0/1     Completed   1          20h
kube-system   svclb-traefik-99d3fde5-smk76             2/2     Running     0          20h
kube-system   traefik-64f55bb67d-947fk                 1/1     Running     0          20h
kube-system   svclb-traefik-99d3fde5-4ph59             2/2     Running     0          111m

That’s it, the master node is set up!

Only one node is available at the moment, so, If you run kubectl get nodes you will get the following:

kubectl get nodes
NAME           STATUS   ROLES                  AGE    VERSION
pc-mini       Ready    control-plane,master   21h    v1.27.6+k3s1

To ensure the master node maintains a consistent IP address, I’ve configured a static IP through the router. You can achieve this by accessing your router at http://192.168.0.1/, navigating to 'DHCP', and selecting 'Address Reservation

Once you have set the static IP the only other item you will need from the master node is the k3s node token located in /var/lib/rancher/k3s/server/node-token

The token will look something like this but longer:
a1b2c3d4f5…::server:a1b2c3d4f5…

Setting Up Additional Nodes

Configuring additional nodes involves a bit more preparation. Given that we’re using Raspberry Pis for these nodes, there are specific adjustments to consider during setup.

Prepping Your Raspberry Pi

To install Raspbian OS, use the Raspberry Pi Imager:

1. Under the “Operating System” section, choose “Raspberry Pi OS Lite”.
2. Within “Raspberry Pi OS (other)”, select your storage device and proceed.
3. When prompted to use OS customizations, opt for “Edit Settings”.

Configuring Service Settings and Installation

1. Navigate to the “Services” tab and enable “SSH”.
2. Under the “General” tab, set both the hostname and your desired username and password.
3. After confirming these settings, proceed to install Raspbian.

Identifying Raspberry Pi’s IP Address

1. Once the Raspberry Pi is up and running on, check for its assigned IP address via your router’s interface: http://192.168.0.1/ > DHCP > DHCP client list. This might take a couple of minutes on the first boot up.
2. With the IP address identified, you can then SSH into the Raspberry Pi. In this instance, the IP address is 192.168.0.105.
3. With the IP you can SSH into it using the following command. The default password, if not changed in the installation process, will be raspberry . The command to ssh is ssh pi@192.168.0.105

Installation via SSH

Once you are logged in to the Raspberry Pi there are 3 main actions we need to take:

• Modify /boot/cmdline.txt
• Modify /boot/config.txt
• Reboot
• Install K3S

Modify /boot/cmdline.txtand add cgroup_memory=1 cgroup_enable=memory at the end of the file. In my case it went from this:

cat /boot/cmdline.txt 
console=serial0,115200 console=tty1 root=PARTUUID=0b679d81–02 rootfstype=ext4 fsck.repair=yes rootwait

to

cat /boot/cmdline.txt 
console=serial0,115200 console=tty1 root=PARTUUID=0b679d81–02 rootfstype=ext4 fsck.repair=yes rootwait cgroup_memory=1 cgroup_enable=memory

Next, add arm_64bit=1 at the end of the file /boot/config.txt

Once this is done reboot and log back in

Installing k3s will be similar but we have to provide the IP of the master node and the node-token when installing

curl -sfL https://get.k3s.io | K3S_URL=https://192.168.0.115:6443 K3S_TOKEN=a1b2c3d4f5...::server:a1b2c3d4f5... sh -

In my case the master node IP is 192.168.0.115
K3S_URL=https://192.168.0.115:6443

And the token will look something like this:
K3S_TOKEN=a1b2c3d4f5…::server:a1b2c3d4f5…

A successful output should look like this:

pi@raspberrypi2:~ $ curl -sfL https://get.k3s.io | K3S_URL=https://192.168.0.115:6443 K3S_TOKEN=a1b2c3d4f5...::server:a1b2c3d4f5... sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.27.6+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.27.6+k3s1/sha256sum-arm64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.27.6+k3s1/k3s-arm64
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO]  systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO]  systemd: Starting k3s-agent

That’s it! Now, if you go back to the main machine running the master node and run kubectl get nodes you will get the following:

kubectl get nodes

NAME           STATUS   ROLES                  AGE    VERSION
pc-mini       Ready    control-plane,master   21h    v1.27.6+k3s1
raspberrypi1   Ready    <none>                 159m   v1.27.6+k3s1

And there you have it — a fully operational Kubernetes cluster at your fingertips!

Adding additional nodes is easy just loop back to the steps above and watch your cluster expand.