k8s Certificates Renewal Verification

Change to a Past Time

/opt/pnt/venv/bin/ansible-playbook -i /opt/pnt/kubespray/inventory/nsp-deployer-default/hosts.yml /opt/pnt/kubespray/change-notes-time.yml -e new_time="2020-02-17 16:27:00"

[root@nsp-k8s-deployer ~]# cat /opt/pnt/kubespray/change-notes-time.yml 
- name: Prepare and reset the required environment for a new build
  hosts: all
  gather_facts: yes
  user: root
  vars:
    new_time: "2021-02-18 12:50:00"
  tasks:
    - name: Disable ntp
      command: "timedatectl set-ntp no"

    - name: Set the new time
      command: "timedatectl set-time '{{ new_time }}'"

    - name: Enable ntp
      command: "timedatectl set-ntp yes"

Deploy a New Cluster

nsp-kubernetes-deployer.bash -i 135.121.159.105 -l node-labels-standard-3nodes.yml

Change Time to Now

/opt/pnt/venv/bin/ansible-playbook -i /opt/pnt/kubespray/inventory/nsp-deployer-default/hosts.yml /opt/pnt/kubespray/change-notes-time.yml -e new_time="2021-02-17 16:27:00"

Renew Certificates (On All Master Nodes)

1. https://v1-16.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/ For 1.16.3, somehow renew all does not work well. Renew each component one-by-one.

   [root@nsp-k8s-deployer ~]# cat /opt/pnt/kubespray/renew-certs.yml 
   - name: Prepare and reset the required environment for a new build
     hosts: "{{ groups['kube-master'] }}"
     gather_facts: yes
     user: root
     vars:
       cluster_components:
       - admin.conf
       - apiserver
       - apiserver-kubelet-client
       - controller-manager.conf
       - front-proxy-client
       - scheduler.conf
     tasks:
       - name: Renew certificates
         command: "kubeadm alpha certs renew {{ item }} --config /etc/kubernetes/kubeadm-config.yaml"
         loop: "{{ cluster_components }}"

2. On nodes created with kubeadm init, prior to kubeadm version 1.17, there is a bug where you manually have to modify the contents of kubelet.conf. After kubeadm init finishes, you should update kubelet.conf to point to the rotated kubelet client certificates by replacing client-certificate-data and client-key-data with:

   client-certificate: /etc/kubernetes/pki/apiserver-kubelet-client.crt
   client-key: /etc/kubernetes/pki/apiserver-kubelet-client.key

3. Restart kube-apiserver

   docker stop $(docker ps | grep kube-apiserver | awk {'print $1'}) 

4. Update .kube/config for errors like “error: You must be logged in to the server (Unauthorized)”

   ...
   current-context: kubernetes-admin@cluster.local
   kind: Config
   preferences: {}
   users:
   - name: kubernetes-admin
     user:
       client-certificate: /etc/kubernetes/pki/apiserver-kubelet-client.crt
       client-key: /etc/kubernetes/pki/apiserver-kubelet-client.key