Setup Harbor with Self-Signed Certificates

I previously had a docker-registry setup for images and a simple httpd service for a Helm chart repo. It worked as expected most of the time until we found that a layered (aka prefix) chart does not work. The solution is to replace the above two services with Harbor (2.0+). Harbor is an OCI-compliant registry capable of serving OCI-compliant artifacts, including Helm charts (requires Helm 3.0+).

The general setup should work with other Ingress controllers like Nginx/nginx-ingress.

Setup the Basic k3s Cluster

export https_proxy=http://proxy.com:8000        # optional
curl -sfL https://get.k3s.io | sh -s - --node-name=nsp-k8s-deployer --disable=traefik --disable=metrics-server

Install Helm Client

A local Helm client is optional if you would like to install charts from a remote client.

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

Install Traefik

kubectl create namespace traefik
helm install --repo https://repo.chartcenter.io --version 9.1.1 --namespace traefik traefik traefik/traefik

Generate Self-Signed Certificates

Reference: https://goharbor.io/docs/2.1.0/install-config/configure-https/

1. Generate a CA certificate private key: openssl genrsa -out ca.key 4096
2. Generate the CA certificate:

openssl req -x509 -new -nodes -sha512 -days 3650 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=yourdomain.com" -key ca.key -out ca.crt

Note: Use the interactive interface for precise settings.

3. Generate an x509 v3 extension file.

Regardless of whether you’re using an FQDN or an IP address to connect to your Harbor host, you must create this file so that you can generate a certificate for your Harbor host that complies with the Subject Alternative Name (SAN) and x509 v3 extension requirements. Replace the DNS entries to reflect your domain.

cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=yourdomain.com
DNS.2=yourdomain
DNS.3=hostname
EOF

4. Use the v3.ext file to generate a certificate for your Harbor host.

Replace yourdomain.com in the CSR and CRT file names with the Harbor host name.

openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in yourdomain.com.csr -out yourdomain.com.crt

5. Convert yourdomain.com.crt to yourdomain.com.cert for use by Docker. (This step might not be needed).

The Docker daemon interprets .crt files as CA certificates and .cert files as client certificates.

openssl x509 -inform PEM -in yourdomain.com.crt -out yourdomain.com.cert

Note: Docker only requires:

ls /etc/docker/certs.d/registry.nspos.nokia.local/ 

Registry.nspos.nokia.local.crt 

6. For Red Hat/CentOS Error: x509: certificate signed by unknown authority

Solution (RHEL): Copy certificate to /etc/pki/ca-trust/source/anchors/

update-ca-trust force-enable 
update-ca-trust extract # (might not be needed) 

Create TLS Secret

In the same namespace as your applications:

kubectl create secret tls traefiktest-cert --cert registry.nspos.nokia.local.crt --key registry.nspos.nokia.local.key -n nsp-k8s-deployer

Install Harbor

helm install harbor center/harbor/harbor -n nsp-k8s-deployer --set externalURL=https://registry.nspos.nokia.local --set expose.tls.certSource=none --set chartmuseum.enabled=false --set clair.enabled=false --set trivy.enabled=false --set trivy.enabled=false --set notary.enabled=false  --set expose.ingress.hosts.core=registry.nspos.nokia.local --set persistence.persistentVolumeClaim.registry.size=10G --set harborAdminPassword=Harbor12235 

The setting for ingress.hosts.core might not be needed; this needs to be verified.

Apply the Ingress Route

# harbor-ingress.yml

kind: IngressRoute 
metadata: 
  name: harbor-ingressroute 
  namespace: nsp-k8s-deployer 

spec: 
  entryPoints: 
    - websecure 

  routes: 
  - match: Host(`registry.nspos.nokia.local`) 
    kind: Rule
    services: 
    - name: harbor-harbor-portal 
      port: 80 
  - match: Host(`registry.nspos.nokia.local`) && PathPrefix(`/api/`, `/service/`, `/v2/`, `/chartrepo/`, `/c/`) 
    kind: Rule 
    services: 
    - name: harbor-harbor-core 
      port: 80 
  tls: 
    certResolver: default 
    domains: 
    - main: 'registry.nspos.nokia.local' 
    secretName: traefik-cert 

Reference:

1. https://github.com/goharbor/harbor-helm/issues/582
2. https://traefik.io/blog/install-and-configure-traefik-with-helm/

Summary

The most important step is the Harbor-specific IngressRoute setting. The details should be available from the Harbor and/or harbor-helm project.