Setup self-signed TLS certs for a registry POD - macos

Question

I created a private Docker registry POD in my Kubernetes cluster.
Here are the relevant settings for the pod:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
private-repository-k8s-686564966d-8grr8 1/1 Running 2 (7h10m ago) 9d
$ kubectl describe pods private-repository-k8s-686564966d-8grr8
...
Containers:
private-repository-k8s:
Container ID: docker://faadba7513c6a1bae6ab96480fcc230ae94a1c8e27c20928f3f93bfd2e7b7714
Image: registry:2
Image ID: docker-pullable://registry#sha256:265d4a5ed8bf0df27d1107edb00b70e658ee9aa5acb3f37336c5a17db634481e
Port: 5000/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 05 Oct 2021 14:49:07 -0700
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Sun, 26 Sep 2021 16:36:48 -0700
Finished: Tue, 05 Oct 2021 14:48:43 -0700
Ready: True
Restart Count: 2
Environment:
REGISTRY_HTTP_TLS_CERTIFICATE: /certs/registry.crt
REGISTRY_HTTP_TLS_KEY: /certs/registry.key
Mounts:
/certs from certs-vol (rw)
/var/lib/registry from registry-vol (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mqknd (ro)
...
Volumes:
certs-vol:
Type: HostPath (bare host directory volume)
Path: /opt/certs
HostPathType: Directory
registry-vol:
Type: HostPath (bare host directory volume)
Path: /opt/registry
HostPathType: Directory
kube-api-access-mqknd:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
I generated the certs on the master as follows:
$ sudo openssl req -newkey rsa:4096 -nodes -sha256 -keyout \
/opt/certs/registry.key -x509 -days 365 -out /opt/certs/registry.crt
Then the folder with the crt and key files are shared via NFS mount across all of the workers.
When I try to push an image from outside the k8s cluster and I get the following error:
$ docker push k8s-master:31320/nginx:1.17 The push refers to repository [k8s-master:31320/nginx]
Get "https://k8s-master:31320/v2/": x509: certificate is not valid for any names, but wanted to match k8s-master
The logs from the POD show this:
$ kubectl logs private-repository-k8s-686564966d-8grr8 -f
...
2021/10/06 05:06:02 http: TLS handshake error from 10.108.82.192:28058: remote error: tls: bad certificate
This proves to me that the request is hitting the POD, but TLS certs weren't setup properly.
I'm doing trying to push the Docker image from my MacOS client to this private Docker registry on a k8s server (each node in the server running Ubuntu).
I'm a bit shaky on the TLS stuff, but my understanding is that I'm using a self-signed cert (which should be fine as I'm only accessing this from my internal network). But I assume I need to do something from my Mac client to setup the TLS certs in order to access the registry. I have already tried adding the crt and key files to my Keychain and that didn't work. I cannot figure out what to do here.
I'm using these instructions:
https://www.linuxtechi.com/setup-private-docker-registry-kubernetes/
I'm running k8s v1.22.0. I have 4 VMs running Ubuntu 20.04.2 LTS inside a single rack server using VMware ESXi: 1 master, 3 worker VMs. I'm trying to push the docker image from my MacBook.

Answer 1

First, I found the CN (Common Name) was not setup property in the certificate (reference: https://github.com/docker/for-linux/issues/248). Once I regenerated the certificate I hit this issue:
$ docker push k8s-master:31320/nginx:1.17 The push refers to repository [k8s-master:31320/nginx]
Get "https://k8s-master:31320/v2/": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
Then I found I needed to add SAN (subjectAltName) to the certificate. I did this as follows:
$ sudo openssl req -newkey rsa:4096 -nodes -sha256 -keyout /opt/certs/registry.key -x509 -days 365 -out /opt/certs/registry.crt -addext "subjectAltName = DNS:k8s-master, DNS:k8s-master.local"
I restarted the registry pod and then I ran into this error:
$ docker push k8s-master:31320/nginx:1.17
The push refers to repository [k8s-master:31320/nginx]
Get "https://k8s-master:31320/v2/": x509: certificate signed by unknown authority
At this point, I realized MacOS client needed the certificate installed into the Keychain. I downloaded the registry.crt file and install it in Keychain (drag and drop). I also had to go into the Keychain, double-clicked on the certificate, opened the "Trust" drop down and selected "Always Trust". Then I restarted Docker on my MacOS.
At this point push started to work:
$ docker push k8s-master:31320/nginx:1.17
The push refers to repository [k8s-master:31320/nginx]
65e1ea1dc98c: Pushed
88891187bdd7: Pushed
6e109f6c2f99: Pushed
0772cb25d5ca: Pushed
525950111558: Pushed
476baebdfbf7: Pushed
1.17: digest: sha256:39065444eb1acb2cfdea6373ca620c921e702b0f447641af5d0e0ea1e48e5e04 size: 1570