Skip to main content

3 node Rancher cluster

Using Ubuntu and RKE2

Setup DNS

Your RKE2/Rancher nodes need to be able to find each other by name. This means DNS.

Follow the instructions here to setup a suitable DNS configuration.

Build VMs

Use the guide here to provision 3 VMs.

Ideally:

  • The VMs should be on seperate physical hypervisors (for HA)
  • The VMs should have either static IPs or reservations in DHCP
  • You can probably get started with:
    • 2x vCPUs each
    • 8GiB of RAM each
    • 100GiB Virtual Disk each

When you get to the "Optional: Cloud-Init" section, return to this page for example cloud-init configuration files.

With Cloud-Init

These cloud-init files include:

  • Install the "helm-stable" package repository
  • Run the equivelant of "apt update" on first boot
  • Run the equivelant of "apt upgrade" on first boot
  • Install the following packages:
    • helm
    • openssh-server
    • avahi-daemon
    • vim-tiny
    • ufw
  • Drop in a RKE2 configuration template with some values pre-filled
    • In the configuration file, we insert a list of TLS SAN (Subject Alternative Name) values to cover a lot of bases
      • The common name for the cluster. Mine will be rke.ghanima.net
      • The name of this secific VM (from the VMNAME environment variable
      • A short name for this specific VM (e.g. "rke1")
      • The above short name appended with ".local" for Avahi name resolution
  • Add a 10GiB swap file in /

These cloud-init files do not:

  • Actually bootstrap the RKE2 cluster. We will do that manually, when we're ready, maybe after testing things a bit first.

Download the following template files:

Set the following environment variables:

export CLUSTERNAME="rke.ghanima.net"
export FIRSTNODE="rke1.ghanima.net"
export VMSHORTNAME=$(echo "${VMNAME}" | awk -F "." '{print $1}')

For the first node, generate a new random token

export TOKEN=$(dd if=/dev/urandom bs=4k count=1 2>/dev/null | sha512sum | dd bs=64 count=1 2>/dev/null); echo "${TOKEN}"

For the remaining nodes, use the token generated for the first node

export TOKEN="<the token>"

Without Cloud-Init

If your hypervisor doesn't support Cloud-Init or you just don't want to use it for some other reason, you can follow these manual steps instead.

Pre-Bootstrap Testing

Test DNS name resolution

Make sure each node and resolve and contact (ping) each other node.

ping rke1.ghanima.net
ping rke2.ghanima.net
ping rke3.ghanima.net

Copy the configuration template to it's proper location

sudo cp /etc/rancher/rke2/config_template.yaml /etc/rancher/rke2/config.yaml

Bootstrap the RKE2 cluster

Reference: https://docs.rke2.io/install/ha/

Run the bootstrap on the first node (only):

curl -sfL https://get.rke2.io | sudo sh -
sudo systemctl enable rke2-server.service
sudo systemctl start rke2-server.service

The start command may take a few moments to return a command prompt. That's OK. Once the service is running it's going to bootstrap the etcd cluster and start a bunch of system pods. This will take a few more minutes.

You can watch it's progress by repeatedly running:

/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get pods --all-namespaces

When all pods are either Running or Completed, the cluster is ready to have the additional nodes added.

Make sure you have a cluster, not standalone nodes

On any node run the following:

/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes

If you don't see a list with multiple nodes, you may have forgotten the "Copy the configuration template to it's proper location" step. Either way, something has gone wrong.

"ubuntu" user kubectl configuration

mkdir .kube
sudo cp /etc/rancher/rke2/rke2.yaml ~/.kube/config
sudo chown -R $(id -u):$(id -g) ~
sudo ln -s /var/lib/rancher/rke2/bin/kubectl /usr/local/bin/kubectl

OPTIONAL: Install the default Kubernetes Dashboard

GITHUB_URL=https://github.com/kubernetes/dashboard/releases
VERSION_KUBE_DASHBOARD=$(curl -w '%{url_effective}' -I -L -s -S ${GITHUB_URL}/latest -o /dev/null | sed -e 's|.*/||')
kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/${VERSION_KUBE_DASHBOARD}/aio/deploy/recommended.yaml

dashboard.admin-user.yml

apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard

dashboard.admin-user-role.yml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
kubectl create -f dashboard.admin-user.yml -f dashboard.admin-user-role.yml

OPTIONAL: Kubernetes Dashboard access via the Nginx Ingress Controller

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dashboard-nginx-ingress
namespace: kubernetes-dashboard
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "false"
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
rules:
- host: e7470-k3s.local
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kubernetes-dashboard
port:
number: 443

Install Rancher Prerequisites

helm repo add rancher-latest https://releases.rancher.com/server-charts/latest kubectl create namespace cattle-system kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.1/cert-manager.crds.yaml helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager \ --namespace cert-manager \ --create-namespace \ --version v1.5.1


## Install Rancher

helm install rancher rancher-latest/rancher \ --namespace cattle-system \ --set hostname=rms.orro.cloud \ --set bootstrapPassword=admin


## Next Steps
**TODO**

* [Install the "local-path-provisioner"](https://github.com/rancher/local-path-provisioner)
* Install Longhorn for HA persistant storage
* Install the "monitoring" package