# Kubernetes Cluster Installation

To learn some of the details about the `kubeadm` toolbox, the best way is to use it to manage a k8s cluster. This includes the whole life cycle management: new installation (HA), scale up/down, and upgrade. All information was current as of May 2022.

## Preparations
You will need these preparations on all of your nodes.

1. https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin
   I use `terraform` to setup 5 Ubuntu 20.04 nodes (2 vCPU, 6Gi RAM).

2. Basic OS setup/update:
    ```bash
    sudo hostnamectl set-hostname node1
    sudo apt update
    sudo apt upgrade
    ```

3. Container runtimes ([containerd](https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd))
   Use the following commands to install containerd on your system:

   a. Install and configure prerequisites:
    ```bash
    cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
    overlay
    br_netfilter
    EOF

    sudo modprobe overlay
    sudo modprobe br_netfilter

    # Setup required sysctl params, these persist across reboots.
    cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
    net.bridge.bridge-nf-call-iptables  = 1
    net.ipv4.ip_forward                 = 1
    net.bridge.bridge-nf-call-ip6tables = 1
    EOF

    # Apply sysctl params without reboot
    sudo sysctl --system
    ```

   b. Install containerd
    Visit Getting started with containerd and follow the instructions there.
    ```bash
    wget https://github.com/containerd/containerd/releases/download/v1.6.4/containerd-1.6.4-linux-amd64.tar.gz
    sudo tar Cxzvf /usr/local containerd-1.6.4-linux-amd64.tar.gz
    wget https://github.com/containerd/containerd/raw/main/containerd.service
    sudo mkdir -p /usr/local/lib/systemd/system
    sudo cp containerd.service /usr/local/lib/systemd/system/containerd.service
    sudo systemctl daemon-reload
    sudo systemctl enable --now containerd
    ```

   c. Install runc
    ```bash
    wget https://github.com/opencontainers/runc/releases/download/v1.1.1/runc.amd64
    sudo install -m 755 runc.amd64 /usr/local/sbin/runc
    ```

   d. Install CNI plugins
    ```bash
    wget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
    sudo mkdir -p /opt/cni/bin
    sudo tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.1.1.tgz
    ```

## Install kubeadm, kubelet and kubectl
You will install these packages on all of your nodes.

1. Install crictl (required for kubeadm/kubelet Container Runtime Interface (CRI))
   ```bash
   CRICTL_VERSION="v1.24.0"
   ARCH="amd64"
   DOWNLOAD_DIR=/usr/local/bin
   sudo mkdir -p $DOWNLOAD_DIR
   curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gz" | sudo tar -C $DOWNLOAD_DIR -xz
   ```

2. Install kubeadm, kubelet, kubectl and add a kubelet systemd service
   ```bash
   #RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
   RELEASE="v1.23.0"
   ARCH="amd64"
   cd $DOWNLOAD_DIR
   sudo curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/${ARCH}/{kubeadm,kubelet,kubectl}
   sudo chmod +x {kubeadm,kubelet,kubectl}
   
   RELEASE_VERSION="v0.4.0"
   curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service
   sudo mkdir -p /etc/systemd/system/kubelet.service.d
   curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
   ```

3. Dependencies installation/setup (Ubuntu 22.04)
   `sudo apt install socat conntrack -y`

## Initialize Control Plane   
  `sudo kubeadm init --apiserver-advertise-address {node-ip-address}`
  > ignored: [WARNING SystemVerification]: missing optional cgroups: blkio

  ```bash
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
  ```

## Setup CNI
  1. Install Cilium CLI
  ```bash
    curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum}
    sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
    sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
    rm cilium-linux-amd64.tar.gz{,.sha256sum}
  ```
  2. Install Cilium: `cilium install`
    
## Setup Worker Nodes
1. Report preparation and kubeadm/kubelet, kubectl setup.
2. Join the node.

## Calico Notes
1. Make sure the default "CALICO_IPV4POOL_CIDR" does not conflict with the node CIDR (e.g., in KVM default net, the subnet is 192.168.100/200.0/24). I ended up setting 172.16.0.0/16. Otherwise, the conflict will cause nodes to be unreachable and Calico pods not ready.

2. Issue: calico/node is not ready: BIRD is not ready: BGP not established
   Solution: Modify `calico.yaml` by searching for `autodetect`.

   ```yaml
   - name: IP
     value: "autodetect"
   ```

   Add the following lines after it:

   ```yaml
   - name: IP_AUTODETECTION_METHOD
     value: "interface=ens*"
   ```

   So it will look like:

   ```yaml
   # Auto-detect the BGP IP address.
   - name: IP
     value: "autodetect"
   - name: IP_AUTODETECTION_METHOD
     value: "interface=ens*"
   ```