Kubernetes Cluster Installation

To learn some of the details about the kubeadm toolbox, the best way is to use it to manage a k8s cluster. This includes the whole life cycle management: new installation (HA), scale up/down, and upgrade. All information was current as of May 2022.

Preparations

You will need these preparations on all of your nodes.

1. https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin I use terraform to setup 5 Ubuntu 20.04 nodes (2 vCPU, 6Gi RAM).

2. Basic OS setup/update:

   sudo hostnamectl set-hostname node1
   sudo apt update
   sudo apt upgrade

3. Container runtimes (containerd) Use the following commands to install containerd on your system:

   a. Install and configure prerequisites:

   cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
   overlay
   br_netfilter
   EOF
   
   sudo modprobe overlay
   sudo modprobe br_netfilter
   
   # Setup required sysctl params, these persist across reboots.
   cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
   net.bridge.bridge-nf-call-iptables  = 1
   net.ipv4.ip_forward                 = 1
   net.bridge.bridge-nf-call-ip6tables = 1
   EOF
   
   # Apply sysctl params without reboot
   sudo sysctl --system

   b. Install containerd Visit Getting started with containerd and follow the instructions there.

   wget https://github.com/containerd/containerd/releases/download/v1.6.4/containerd-1.6.4-linux-amd64.tar.gz
   sudo tar Cxzvf /usr/local containerd-1.6.4-linux-amd64.tar.gz
   wget https://github.com/containerd/containerd/raw/main/containerd.service
   sudo mkdir -p /usr/local/lib/systemd/system
   sudo cp containerd.service /usr/local/lib/systemd/system/containerd.service
   sudo systemctl daemon-reload
   sudo systemctl enable --now containerd

   c. Install runc

   wget https://github.com/opencontainers/runc/releases/download/v1.1.1/runc.amd64
   sudo install -m 755 runc.amd64 /usr/local/sbin/runc

   d. Install CNI plugins

   wget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
   sudo mkdir -p /opt/cni/bin
   sudo tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.1.1.tgz

Install kubeadm, kubelet and kubectl

You will install these packages on all of your nodes.

1. Install crictl (required for kubeadm/kubelet Container Runtime Interface (CRI))

   CRICTL_VERSION="v1.24.0"
   ARCH="amd64"
   DOWNLOAD_DIR=/usr/local/bin
   sudo mkdir -p $DOWNLOAD_DIR
   curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gz" | sudo tar -C $DOWNLOAD_DIR -xz

2. Install kubeadm, kubelet, kubectl and add a kubelet systemd service

   #RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
   RELEASE="v1.23.0"
   ARCH="amd64"
   cd $DOWNLOAD_DIR
   sudo curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/${ARCH}/{kubeadm,kubelet,kubectl}
   sudo chmod +x {kubeadm,kubelet,kubectl}
   
   RELEASE_VERSION="v0.4.0"
   curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service
   sudo mkdir -p /etc/systemd/system/kubelet.service.d
   curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

3. Dependencies installation/setup (Ubuntu 22.04) sudo apt install socat conntrack -y

Initialize Control Plane

sudo kubeadm init --apiserver-advertise-address {node-ip-address}

ignored: [WARNING SystemVerification]: missing optional cgroups: blkio

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Setup CNI

1. Install Cilium CLI

  curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum}
  sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
  sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
  rm cilium-linux-amd64.tar.gz{,.sha256sum}

2. Install Cilium: cilium install

Setup Worker Nodes

1. Report preparation and kubeadm/kubelet, kubectl setup.
2. Join the node.

Calico Notes

1. Make sure the default “CALICO_IPV4POOL_CIDR” does not conflict with the node CIDR (e.g., in KVM default net, the subnet is 192.168.100/200.0/24). I ended up setting 172.16.0.0/16. Otherwise, the conflict will cause nodes to be unreachable and Calico pods not ready.

2. Issue: calico/node is not ready: BIRD is not ready: BGP not established Solution: Modify calico.yaml by searching for autodetect.

   - name: IP
     value: "autodetect"

   Add the following lines after it:

   - name: IP_AUTODETECTION_METHOD
     value: "interface=ens*"

   So it will look like:

   # Auto-detect the BGP IP address.
   - name: IP
     value: "autodetect"
   - name: IP_AUTODETECTION_METHOD
     value: "interface=ens*"