flux deployment error X509 certificate signed by unknown authority - gitops

Question

My aim is to deploy a container-labelling-webhook solution onto my AKS cluster using flux CD v2. Once I have it operational, I want to rollout to multiple clusters.
Command used to bootstrap AKS Cluster(Flux Installation I mean)
flux bootstrap git --url=https://github.xxxxxx.com/user1/test-repo.git --username=$GITHUB_USER --password=$GITHUB_TOKEN --token-auth=true --path=clusters/my-cluster
✔ Kustomization reconciled successfully
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy
Now, I am trying to deploy my helm charts, note, helm chart deployment by itself works fine, not via Flux though.
flux create source helm label-webhook --url https://github.xxxxxx.com/user1/test-repo/tree/main/chart --namespace label-webhook --cert-file=./tls/label-webhook.pem --key-file=./tls/label-webhook-key.pem --ca-file=./tls/ca.pem --verbose
✚ generating HelmRepository source
► applying secret with repository credentials
✔ authentication configured
► applying HelmRepository source
✔ source created
◎ waiting for HelmRepository source reconciliation
✗ failed to fetch Helm repository index: failed to cache index to temporary file: Get "https://github.xxxxxx.com/user1/test-repo/tree/main/chart/index.yaml": x509: certificate signed by unknown authority
I am generating certs with the process below:
cat << EOF > ca-config.json
{
"signing": {
"default": {
"expiry": "43830h"
},
"profiles": {
"default": {
"usages": ["signing", "key encipherment", "server auth", "client auth"],
"expiry": "43830h"
}
}
}
}
EOF
cat << EOF > ca-csr.json
{
"hosts": [
"cluster.local"
],
"key": {
"algo": "rsa",
"size": 4096
},
"names": [
{
"C": "AU",
"L": "Melbourne",
"O": "xxxxxx",
"OU": "Container Team",
"ST": "aks1-dev"
}
]
}
EOF
docker run -it --rm -v ${PWD}:/work -w /work debian bash
apt-get update && apt-get install -y curl &&
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl_1.5.0_linux_amd64 -o /usr/local/bin/cfssl && \
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssljson_1.5.0_linux_amd64 -o /usr/local/bin/cfssljson && \
chmod +x /usr/local/bin/cfssl && \
chmod +x /usr/local/bin/cfssljson
cfssl gencert -initca ca-csr.json | cfssljson -bare /tmp/ca
cfssl gencert \
-ca=/tmp/ca.pem \
-ca-key=/tmp/ca-key.pem \
-config=ca-config.json \
-hostname="mutation-label-webhook,mutation-label-webhook.label-webhook.svc.cluster.local,mutation-label-webhook.label-webhook.svc,localhost,127.0.0.1" \
-profile=default \
ca-csr.json | cfssljson -bare /tmp/label-webhook
root#91bc7986cb94:/work# ls -alrth /tmp/
total 32K
drwxr-xr-x 1 root root 4.0K Jul 29 04:42 ..
-rw-r--r-- 1 root root 2.0K Jul 29 04:43 ca.pem
-rw-r--r-- 1 root root 1.8K Jul 29 04:43 ca.csr
-rw------- 1 root root 3.2K Jul 29 04:43 ca-key.pem
-rw-r--r-- 1 root root 2.2K Jul 29 04:43 label-webhook.pem
-rw-r--r-- 1 root root 1.9K Jul 29 04:43 label-webhook.csr
-rw------- 1 root root 3.2K Jul 29 04:43 label-webhook-key.pem
drwxrwxrwt 1 root root 4.0K Jul 29 04:43 .
root#91bc7986cb94:/work#
root#83faa77cd5f6:/work# cp -apvf /tmp/* .
'/tmp/ca-key.pem' -> './ca-key.pem'
'/tmp/ca.csr' -> './ca.csr'
'/tmp/ca.pem' -> './ca.pem'
'/tmp/label-webhook-key.pem' -> './label-webhook-key.pem'
'/tmp/label-webhook.csr' -> './label-webhook.csr'
'/tmp/label-webhook.pem' -> './label-webhook.pem'
root#83faa77cd5f6:/work# pwd
/work
chmod -R 777 tls/
helm upgrade --install mutation chart --namespace label-webhook --create-namespace --set secret.cert=$(cat tls/label-webhook.pem | base64 | tr -d '\n') --set secret.key=$(cat tls/label-webhook-key.pem | base64 | tr -d '\n') --set secret.cabundle=$(openssl base64 -A <"tls/ca.pem")
I am totally confused as to how to get flux working?

Answer 1

Flux doesn't trust the certificate presented by your git server github.xxxxxx.com
Quick workaround is to use --insecure-skip-tls-verify flag as described here: https://fluxcd.io/docs/cmd/flux_bootstrap_git/
Full command:
flux create source helm label-webhook --url https://github.xxxxxx.com/user1/test-repo/tree/main/chart --namespace label-webhook --cert-file=./tls/label-webhook.pem --key-file=./tls/label-webhook-key.pem --ca-file=./tls/ca.pem --verbose --insecure-skip-tls-verify
It's interesting there wasn't problem with flux bootstrap git step but it probably just create configuration for repository in this step and not establish connection to it.
Whatever certificates you are generating don't have anything to do with your GIT server TLS certificate. Seems you're doing some admission webhook magic but the certs you generate there have nothing in common with github.xxxxxx.com certificate so there is no need to specify if in --ca-file flag.
Permanent solution is to get the CA certificate that signed the github.xxxxxx.com so you need to ask the administrators of the GIT server to provide you CA file and specify that one in --ca-file flag. Not the one you created for your webhook experiments.