Deployments in a GKE cluster with Istio is working correctly via HTTP. But when i tried to secure it with cert-manager with following resources, HTTPS request fails state like so on curl
`Immediate connect fail for 64:ff9b::2247:fd8a: Network is unreachable
* connect to 34.71.253.138 port 443 failed: Connection refused`.
What should i do to make it work with HTTPS as well.
ClusterIssuer with following configuration
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
namespace: istio-system
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: iprocureservers#iprocu.re
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging
solvers:
# ACME DNS-01 provider configurations
- dns01:
# Google Cloud DNS
clouddns:
# Secret from the google service account key
serviceAccountSecretRef:
name: cert-manager-credentials
key: gcp-dns-admin.json
# The project in which to update the DNS zone
project: iprocure-server
Certificate configuration like so, which made a certifiate in a Ready:True state
apiVersion: cert-manager.io/v1alpha3
kind: Certificate
metadata:
name: letsencrypt-staging
namespace: istio-system
spec:
secretName: letsencrypt-staging
commonName: "*.iprocure.tk"
dnsNames:
- '*.iprocure.tk'
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuer
And lastly a Gateway
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: iprocure-gateway
namespace: default
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
tls:
httpsRedirect: false
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- "*"
tls:
mode: SIMPLE
credentialName: letsencrypt-staging
If i do, kubectl describe certificate -n istio-system
Name: letsencrypt-staging
Namespace: istio-system
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2020-10-13T13:32:37Z
Generation: 1
Resource Version: 28030994
Self Link: /apis/cert-manager.io/v1/namespaces/istio-system/certificates/letsencrypt-staging
UID: ad838d28-5349-4aaa-a618-cc3bfc316e6e
Spec:
Common Name: *.iprocure.tk
Dns Names:
*.iprocure.tk
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-staging-clusterissuer
Secret Name: letsencrypt-staging-cert-secret
Status:
Conditions:
Last Transition Time: 2020-10-13T13:35:05Z
Message: Certificate is up to date and has not expired
Reason: Ready
Status: True
Type: Ready
Not After: 2021-01-11T12:35:05Z
Not Before: 2020-10-13T12:35:05Z
Renewal Time: 2020-12-12T12:35:05Z
Revision: 1
Events: <none>
Running kubectl get certificates -o wide -n istio-system, yields
NAME READY SECRET ISSUER STATUS AGE
letsencrypt-staging True letsencrypt-staging-cert-secret letsencrypt-staging-clusterissuer Certificate is up to date and has not expired 17h
Issue
I assume that https wasn't working because of the requirements which have to be enabled if you want to use cert-menager with istio in older versions.
Solution
As #Yunus Einsteinium mentioned in comments
Thank you for guiding me in the right direction. Using the OSS Istio, not the GKE one, is the way to go! I managed to make HTTPS work!
So the solution here was to use OOS istio installed with istioctl instead of the older istio gke addon.
Related
I am trying to deploy a springboot application running on 8080 port. My target is to have https protocol for custom subdomain with google managed-certificates.
here are my yamls.
deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
namespace: my-namespace
spec:
replicas: 1
selector:
matchLabels:
app: my-deployment
namespace: my-namespace
template:
metadata:
labels:
app: my-deployment
namespace: my-namespace
spec:
containers:
- name: app
image: gcr.io/PROJECT_ID/IMAGE:TAG
imagePullPolicy: Always
ports:
- containerPort: 8080
resources:
requests:
memory: "256Mi"
ephemeral-storage: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
ephemeral-storage: "512Mi"
cpu: "250m"
2.service.yaml
apiVersion: v1
kind: Service
metadata:
name: my-service
namespace: my-namespace
annotations:
cloud.google.com/backend-config: '{"default": "my-http-health-check"}'
spec:
selector:
app: my-deployment
namespace: my-namespace
type: NodePort
ports:
- port: 80
name: http
targetPort: http
protocol: TCP
ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
namespace: my-name-space
annotations:
kubernetes.io/ingress.global-static-ip-name: my-ip
networking.gke.io/managed-certificates: my-cert
kubernetes.io/ingress.class: "gce"
labels:
app: my-ingress
spec:
rules:
- host: my-domain.com
http:
paths:
- pathType: ImplementationSpecific
backend:
service:
name: my-service
port:
name: http
I followed various documentation, most of them could help to make http work but, couldn't make https work and ends with error ERR_SSL_VERSION_OR_CIPHER_MISMATCH. Looks like there is issue with "Global forwarding rule". Ports shows 443-443. What is the correct way to terminate the HTTPS traffic at loadbalancer and route it to backend app with http?
From the information provided, I can see that the "ManagedCertificate" object is missing, you need to create a yaml file with the following structure:
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
name: my-cert
spec:
domains:
- <your-domain-name1>
- <your-domain-name2>
And then apply it with the command: kubectl apply -f file-name.yaml
Provisioning of the Google-managed certificate can take up to 60 minutes; you can check the status of the certificate using the following command: kubectl describe managedcertificate my-cert, wait for the status to be as "Active".
A few prerequisites you need to be aware, though:
You must own the domain name. The domain name must be no longer than
63 characters. You can use Google Domains or another registrar.
The cluster must have the HttpLoadBalancing add-on enabled.
Your "kubernetes.io/ingress.class" must be "gce".
You must apply Ingress and ManagedCertificate resources in the same
project and namespace.
Create a reserved (static) external IP address. Reserving a static IP
address guarantees that it remains yours, even if you delete the
Ingress. If you do not reserve an IP address, it might change,
requiring you to reconfigure your domain's DNS records.
Finally, you can take a look at the complete Google's guide on Creating an Ingress with a Google-managed certificate.
Following is my Kubernetes configuration. The API deployed using this config works as expected when SSL verification is disabled by the client or when HTTP is used instead of HTTPS. But on enabling, it throws SSL Error: Unable to verify the first certificate. The SSL certificate files are added as Kubernetes secret and the API is exposed on port 8080.
---
apiVersion: "v1"
kind: "ConfigMap"
metadata:
name: "test-config"
namespace: "default"
labels:
app: "test"
data:
ENV: "DEV"
---
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
name: "test"
namespace: "default"
labels:
app: "test"
spec:
replicas: 1
selector:
matchLabels:
app: "test"
template:
metadata:
labels:
app: "test"
spec:
containers:
- name: "test"
image: "gcr.io/test-project/test:latest"
env:
- name: "ENV"
valueFrom:
configMapKeyRef:
key: "ENV"
name: "test-config"
---
apiVersion: "extensions/v1beta1"
kind: "Ingress"
metadata:
name: "test-ingress"
annotations:
kubernetes.io/ingress.global-static-ip-name: "test-static-ip"
labels:
app: "test"
spec:
tls:
- hosts:
- "test.myhost.com"
secretName: "test-ssl-certificate"
backend:
serviceName: "test-service-nodeport"
servicePort: 8080
rules:
- host: "test.myhost.com"
http:
paths:
- path: "/*"
backend:
serviceName: "test-service-nodeport"
servicePort: 8080
---
kind: "Service"
apiVersion: "v1"
metadata:
name: "test-service-nodeport"
spec:
selector:
app: "test"
ports:
- protocol: "TCP"
port: 8080
targetPort: 8080
type: "NodePort"
Go server code
http.HandleFunc("/hello", HelloServer)
err := http.ListenAndServeTLS(":8080", "server.crt", "server.key", nil)
if err != nil {
log.Fatal("ListenAndServe: ", err)
}
A potential cause for this could be intermediate certificates were not installed on the server properly which caused a breakdown in the certificate chain. Even if it works in your browser, it may not be including all the public certificates in the chain needed for a cache-empty client to verify. Here are some preliminary troubleshooting steps:
Verify that your certificate chain is complete [https://certificatechain.io/ ]
Verify your server’s configuration [https://www.ssllabs.com/ssltest/ or https://www.sslshopper.com/ssl-checker.html ]
Look for this error:
This server's certificate chain is incomplete.
And this:
Chain issues.........Incomplete
If you encounter these issues, it means that the web server you are connecting to is misconfigured and did omit the intermediate certificate in the certificate chain it sent to you. Your server needs to serve not just the certificate for your domain, but also the intermediate certificates too.
Intermediate certificate should be installed on the server, along with the server certificate. Root certificates are embedded into the software applications, browsers and operating systems. The application serving the certificate has to send the complete chain, this means the server certificate itself and all the intermediates.
Refer stack post and combine the server certificate and intermediate certificate into a chained certificate for information.
I'm trying to create HTTPS ingress for my node.js authentication (auth) REST service in AKS, but I'm getting a 502 Bad Gateway response.
Here's my deployment and service definitions:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: auth
namespace: auth
labels:
app: auth
spec:
selector:
matchLabels:
app: auth
replicas: 1
template:
metadata:
labels:
app: auth
spec:
imagePullSecrets:
- name: docker-hub-creds
containers:
- name: auth
image: ***image***
ports:
- containerPort: 80
name: auth
---
apiVersion: v1
kind: Service
metadata:
name: auth
namespace: auth
spec:
selector:
app: auth
ports:
- protocol: TCP
port: 80
targetPort: 80
I think that's all pretty basic and it seems to work ok. I can see the service running and if I expose a node-port then I can access it with no problems. The service responds to well-formed POST requests on the /auth path with a JWT.
I have configured an Azure Application Gateway following Microsoft's instructions, and following the troubleshooting guide leads me to believe that the installation has worked ok. I have also checked through the web-ui and there appear to be no errors. Finally, I worked through the support options and the automated analysis of my cluster found no major configuration issues.
Next, I tried to create an HTTPS ingress route for my service, and this is where it goes wrong. This is made more complicated by the dynamic generation of certificates for TLS.
The ingress definition looks like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: auth-in
namespace: auth
annotations:
kubernetes.io/ingress.class: azure/application-gateway
cert-manager.io/cluster-issuer: letsencrypt-staging
cert-manager.io/acme-challenge-type: http01
ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
tls:
- hosts:
- ***hostname***
secretName: ***secret***
rules:
- host: ***same hostname***
http:
paths:
- backend:
serviceName: auth
servicePort: 80
path: /api/(auth/.*)
I have two rewrite-targets in there because I can't determine which one this ingress controller uses. All the example from the web use the nginx. prefix so I added it in desperation, despite thinking that it's probably not necessary.
Accessing the service through: ***hostname***/api/auth results in a Bad Gateway error.
I have checked through the portal and I can see the route is registered, listeners and rules are there, and my service is listed in the backend pools, but there is nothing in the 'rewrite' tabs. I expected to see something in the rewrite tabs.
I've tooled my service to log all access, and the logs show this, repeatedly:
{"level":30,"time":1611739355140,"pid":17,"hostname":"auth-6c7757bb89-d72td","msg":"Req-URL: /api/(auth/.*)"}
Describing the ingress gives me this:
Name: auth-in
Namespace: auth
Address: **redacted***
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
***redacted cert name** terminates **hostname***
Rules:
Host Path Backends
---- ---- --------
***hostname***
/api/(auth/.*) auth:80 10.0.0.69:80)
Annotations: cert-manager.io/acme-challenge-type: http01
cert-manager.io/cluster-issuer: letsencrypt-staging
ingress.kubernetes.io/rewrite-target: /$1
kubernetes.io/ingress.class: azure/application-gateway
nginx.ingress.kubernetes.io/rewrite-target: /$1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CreateCertificate 43m cert-manager Successfully created Certificate "***cert-name***"
Two things to note. 1st that the logs show that the URL isn't being rewritten -- it's being passed exactly as the path shows, including the regex part. 2nd, that the Default Backend entry in the ingress description shows an error. I'm not sure that the 2nd one matters, but the 1st is clearly wrong.
I am keen to discover how to diagnose the problem and then fix it.
Since you are using AGIC you can include Backend Path Prefix annotation appgw.ingress.kubernetes.io/backend-path-prefix: "/"
The Ingress will be like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: auth-in
namespace: auth
annotations:
kubernetes.io/ingress.class: azure/application-gateway
cert-manager.io/cluster-issuer: letsencrypt-staging
cert-manager.io/acme-challenge-type: http01
appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
tls:
- hosts:
- ***hostname***
secretName: ***secret***
rules:
- host: ***same hostname***
http:
paths:
- backend:
serviceName: auth
servicePort: 80
path: /api/auth/*
AGIC on Nov 12 '21 has also included a rewrite-rule-set as part of this PR. For rewrite-rule, you can use the rewrite-rule annotation.
appgw.ingress.kubernetes.io/rewrite-rule-set: <rewrite rule set>
I have an application running in Minikube that works with the ingress-gateway as expected. A spring boot app is called, the view is displayed and a protected resource is called via a link. The call is forwarded to Keycloak and is authorized via the login mask and the protected resource is displayed as expected.
With Istio the redirecting fails with the message: "Invalid parameter: redirect_uri".
My Istio Gateway config
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
namespace: istio-system
name: istio-bomc-app-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
My virtualservice config
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: istio-bomc-app-hrm-virtualservice
namespace: bomc-app
spec:
hosts:
- "*"
gateways:
- istio-bomc-app-gateway.istio-system.svc.cluster.local
http:
- match:
- uri:
prefix: /bomc-hrm
route:
- destination:
host: bomc-hrm-service.bomc-app.svc.cluster.local
port:
number: 80
After clicking the protected link, I get the following URI in the browser:
http://192.168.99.100:31380/auth/realms/bomc-hrm-realm/protocol/openid-connect/auth?response_type=code&client_id=bomc-hrm-app&redirect_uri=http%3A%2F%2F192.168.99.100%2Fbomc-hrm%2Fui%2Fcustomer%2Fcustomers&state=4739ab56-a8f3-4f78-bd29-c05e7ea7cdbe&login=true&scope=openid
I see the redirect_uri=http%3A%2F%2F192.168.99.100%2F is not complete. The port 31380 is missing.
How does Istio VirtualService need to be configured?
Have you checked the following command into Google Cloud
Maybe you will have clues using it
kubectl describe
Check Kube
I am working with cert-manager in my kubernetes cluster, in order to get certificates signed by let'sencrypt CA to my service application inside my cluster.
I am performing the following steps in the order presented. I've wanted provide the most details as a possible of my process in order to understand the behavior presented.
Install the CustomResourceDefinition resources separately
⟩ kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml
customresourcedefinition.apiextensions.k8s.io/certificates.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/challenges.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/issuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/orders.certmanager.k8s.io created
[I]
Label the cert-manager namespace to disable resource validation
⟩ kubectl label namespace kube-system certmanager.k8s.io/disable-validation=true
namespace/kube-system labeled
[I]
Install the cert-manager Helm chart
⟩ helm install \
--name cert-manager \
--namespace kube-system \
--version v0.7.0 \
jetstack/cert-manager
I've confirmed the steps in this guide, in order to avoid possible problems, and all steps are ok ...
Creating my ingress
I am using kong-ingress-controller to manage the ingress process.
⟩ kubectl get pod,svc,deploy,replicaset -n kong | grep kong-ingress-controller
pod/kong-ingress-controller-667b4748d4-ccj8z 2/2 Running 14 95m
service/kong-ingress-controller NodePort 10.0.48.131 <none> 8001:32257/TCP 3d19h
deployment.extensions/kong-ingress-controller 1 1 1 1 3d19h
replicaset.extensions/kong-ingress-controller-667b4748d4 1 1 1 3d19h
This means that my external IP addres is given by kong-proxy and is 52.166.60.158
⟩ kubectl get svc -n kong | grep kong-proxy
kong-proxy LoadBalancer 10.0.153.8 52.166.60.158 80:31577/TCP,443:32323/TCP 3d21h
I've created the ingress for first time of this way:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kong-ingress-zcrm365
# namespace: default
annotations:
# kubernetes.io/ingress.class: "nginx" # Don't include it in order to use kong-ingress-controller
# add an annotation indicating the issuer to use.
spec:
rules:
- host: test1kongletsencrypt.possibilit.nl
http:
paths:
- path: "/"
backend:
serviceName: zcrm365dev
servicePort: 80
#- backend:
# serviceName: zcrm365dev
# servicePort: 80
# path: /
tls: # < placing a host in the TLS config will indicate a cert should be created
- hosts:
- test1kongletsencrypt.possibilit.nl
secretName: cert-manager-webhook-webhook-tls
# for the moment I've included this secret which was created by cert-manager installation
Apply it.
⟩ kubectl apply -f 03-zcrm365-ingress.yaml
ingress.extensions/kong-ingress-zcrm365 created
[I]
And my ingress is taking the kong-ingress-controller
⟩ kubectl describe ingress kong-ingress-zcrm365
Name: kong-ingress-zcrm365
Namespace: default
Address: 52.166.60.158
Default backend: default-http-backend:80 (<none>)
TLS:
cert-manager-webhook-webhook-tls terminates test1kongletsencrypt.possibilit.nl
Rules:
Host Path Backends
---- ---- --------
test1kongletsencrypt.possibilit.nl
/ zcrm365dev:80 (<none>)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"kong-ingress-zcrm365","namespace":"default"},"spec":{"rules":[{"host":"test1kongletsencrypt.possibilit.nl","http":{"paths":[{"backend":{"serviceName":"zcrm365dev","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["test1kongletsencrypt.possibilit.nl"],"secretName":"cert-manager-webhook-webhook-tls"}]}}
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 3m30s kong-ingress-controller Ingress default/kong-ingress-zcrm365
Normal UPDATE 3m28s kong-ingress-controller Ingress default/kong-ingress-zcrm365
[I]
Creating a ClusterIssuer
I am going to create a ClusterIssuer, I'd can create a Issuer, but I've started with a ClusterIssuer. What is the best alternative? This depen of our deployment and requirements to future, mostly in the namespace situations
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: b.garcia#possibilit.nl
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging
# Enable the HTTP-01 challenge provider
http01: {}
Apply it
⟩ kubectl apply -f 01-lets-encrypt-issuer-staging.yaml
clusterissuer.certmanager.k8s.io/letsencrypt-staging created
[I]
This ClusterIssuer was registered on ACME letsencrypt server
⟩ kubectl describe clusterissuers letsencrypt-staging
Name: letsencrypt-staging
Namespace:
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"ClusterIssuer","metadata":{"annotations":{},"name":"letsencrypt-staging"},"spec":{"acm...
API Version: certmanager.k8s.io/v1alpha1
Kind: ClusterIssuer
Metadata:
Creation Timestamp: 2019-03-15T11:38:03Z
Generation: 1
Resource Version: 623999
Self Link: /apis/certmanager.k8s.io/v1alpha1/clusterissuers/letsencrypt-staging
UID: cb48b391-4716-11e9-a113-e27267a7d354
Spec:
Acme:
Email: b.garcia#possibilit.nl
Http 01:
Private Key Secret Ref:
Name: letsencrypt-staging
Server: https://acme-staging-v02.api.letsencrypt.org/directory
Status:
Acme:
Uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/8579841
Conditions:
Last Transition Time: 2019-03-15T11:38:05Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events: <none>
[I]
Modifying my ingress resource created previously
Now that I've created our Let's Encrypt staging ClusterIssuer, I am ready to modify the Ingress Resource we created above and enable TLS encryption for the test1kongletsencrypt.possibilit.nl paths adding the following
I am going to add certmanager.k8s.io/cluster-issuer: letsencrypt-staging annotation and use the secret created with the letsencrypt-staging ClusterIssuer named letsencrypt-staging
Our ingress has been stayed of this way:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kong-ingress-zcrm365
#namespace: default
annotations:
# kubernetes.io/ingress.class: "nginx" #new
# certmanager.k8s.io/acme-challenge-type: http01
# add an annotation indicating the issuer to use.
certmanager.k8s.io/cluster-issuer: letsencrypt-staging
spec:
rules:
- host: test1kongletsencrypt.possibilit.nl
http:
paths:
- path: "/"
backend:
serviceName: zcrm365dev
servicePort: 80
tls:
- hosts:
- test1kongletsencrypt.possibilit.nl
secretName: letsencrypt-staging # I've added this secret of letsencrypt cluster issuer
Apply it
⟩ kubectl apply -f 03-zcrm365-ingress.yaml
ingress.extensions/kong-ingress-zcrm365 configured
[I]
This process update on the ingress, create one ingres named cm-acme-http-solver-jr4fg
⟩ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
cm-acme-http-solver-jr4fg test1kongletsencrypt.possibilit.nl 80 33s
kong-ingress-zcrm365 test1kongletsencrypt.possibilit.nl 52.166.60.158 80, 443 56m
[I]
The detail of cm-acme-http-solver-jr4fg ingress is:
⟩ kubectl get ingress cm-acme-http-solver-jr4fg -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0
creationTimestamp: "2019-03-15T12:10:57Z"
generateName: cm-acme-http-solver-
generation: 1
labels:
certmanager.k8s.io/acme-http-domain: "4095675862"
certmanager.k8s.io/acme-http-token: "657526223"
name: cm-acme-http-solver-jr4fg
namespace: default
ownerReferences:
- apiVersion: certmanager.k8s.io/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Challenge
name: letsencrypt-staging-2613163196-0
uid: 638f1701-471b-11e9-a113-e27267a7d354
resourceVersion: "628284"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/cm-acme-http-solver-jr4fg
uid: 640ef483-471b-11e9-a113-e27267a7d354
spec:
rules:
- host: test1kongletsencrypt.possibilit.nl
http:
paths:
- backend:
serviceName: cm-acme-http-solver-svmvw
servicePort: 8089
path: /.well-known/acme-challenge/W7-9-KuPao_jg6EF5E2FXitFs8shOEsY5PlT9EEvNxE
status:
loadBalancer:
ingress:
- ip: 52.166.60.158
And the detail of our kong-ingress-zcrm365 resource ingress is:
⟩ kubectl describe ingress kong-ingress-zcrm365
Name: kong-ingress-zcrm365
Namespace: default
Address: 52.166.60.158
Default backend: default-http-backend:80 (<none>)
TLS:
letsencrypt-staging terminates test1kongletsencrypt.possibilit.nl
Rules:
Host Path Backends
---- ---- --------
test1kongletsencrypt.possibilit.nl
/ zcrm365dev:80 (<none>)
Annotations:
certmanager.k8s.io/cluster-issuer: letsencrypt-staging
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"certmanager.k8s.io/cluster-issuer":"letsencrypt-staging"},"name":"kong-ingress-zcrm365","namespace":"default"},"spec":{"rules":[{"host":"test1kongletsencrypt.possibilit.nl","http":{"paths":[{"backend":{"serviceName":"zcrm365dev","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["test1kongletsencrypt.possibilit.nl"],"secretName":"letsencrypt-staging"}]}}
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 60m kong-ingress-controller Ingress default/kong-ingress-zcrm365
Normal UPDATE 4m25s (x2 over 60m) kong-ingress-controller Ingress default/kong-ingress-zcrm365
Normal CreateCertificate 4m25s cert-manager Successfully created Certificate "letsencrypt-staging"
[I]
We can see that our ingress even using the kong-ingress-controller and the letsencrypt-staging certificate has been created in the default namespace:
⟩ kubectl get certificates
NAME
letsencrypt-staging
[I]
The letsencypt-staging certificate have the following detail
⟩ kubectl describe certificate letsencrypt-staging
Name: letsencrypt-staging
Namespace: default
Labels: <none>
Annotations: <none>
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Creation Timestamp: 2019-03-15T12:10:55Z
Generation: 1
Owner References:
API Version: extensions/v1beta1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: kong-ingress-zcrm365
UID: 8643558f-4713-11e9-a113-e27267a7d354
Resource Version: 628164
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/letsencrypt-staging
UID: 62b3a31e-471b-11e9-a113-e27267a7d354
Spec:
Acme:
Config:
Domains:
test1kongletsencrypt.possibilit.nl
Http 01:
Dns Names:
test1kongletsencrypt.possibilit.nl
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-staging
Secret Name: letsencrypt-staging
Status:
Conditions:
Last Transition Time: 2019-03-15T12:10:55Z
Message: Certificate issuance in progress. Temporary certificate issued.
Reason: TemporaryCertificate
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Generated 7m24s cert-manager Generated new private key
Normal GenerateSelfSigned 7m24s cert-manager Generated temporary self signed certificate
Normal OrderCreated 7m23s cert-manager Created Order resource "letsencrypt-staging-2613163196"
[I]
~/workspace/ZCRM365/Deployments/Kubernetes/cert-manager · (Deployments±)
I can see that my order issue is not completed, only was created in the OrderCreated event, and this order already have 7 minutes since I've created this certificate and the order was not completed and by that reason the certificate is not issued successfully
Another thing that happens to me, is that the letsencrypt-staging secret created by the letsencrypt-staging cluster Issuer and their respective certificate, only have the tls.key:
⟩ kubectl describe secrets letsencrypt-staging -n kube-system
Name: letsencrypt-staging
Namespace: kube-system
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
tls.key: 1675 bytes
[I]
According to I understand, is that if the letsencrypt certificate complete the order and the certificate is issued, in the letsencrypt-staging secret I would have one tls.crt key and maybe my letsencrypt-staging will be of tls type and not Opaque?
When I see the logs of my cert-manager pod I get the following output, I think that the http challenge is not executed:
I0315 12:10:57.833858 1 logger.go:103] Calling Discover
I0315 12:10:57.856136 1 pod.go:64] No existing HTTP01 challenge solver pod found for Certificate "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:57.923080 1 service.go:51] No existing HTTP01 challenge solver service found for Certificate "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:57.989596 1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=4095675862,certmanager.k8s.io/acme-http-token=657526223
I0315 12:10:57.989682 1 ingress.go:98] No existing HTTP01 challenge solver ingress found for Challenge "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:58.014803 1 controller.go:178] ingress-shim controller: syncing item 'default/cm-acme-http-solver-jr4fg'
I0315 12:10:58.014842 1 sync.go:64] Not syncing ingress default/cm-acme-http-solver-jr4fg as it does not contain necessary annotations
I0315 12:10:58.014846 1 controller.go:184] ingress-shim controller: Finished processing work item "default/cm-acme-http-solver-jr4fg"
I0315 12:10:58.015447 1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=4095675862,certmanager.k8s.io/acme-http-token=657526223
I0315 12:10:58.033431 1 sync.go:173] propagation check failed: wrong status code '404', expected '200'
I0315 12:10:58.079504 1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 12:10:58.079616 1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I0315 12:10:58.079569 1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
I get this message No existing HTTP01 challenge solver pod found for Certificate "default/letsencrypt-staging-2613163196-0"
According to this, I decide add the certmanager.k8s.io/acme-challenge-type: http01 annotation to my kong-ingress-zcrm365 ingress but nothing happened ... my ingress is updated, but nothing more.
All this process confirms that the TLS certificate was not successfully issued and HTTPS encryption is not active for my domains test1kongletsencrypt.possibilit.nl configured.
This make that my letsencrypt-staging certificate have a Status:False, and the order created event does not advance to completed to be issued.
Conditions:
Last Transition Time: 2019-03-15T12:10:55Z
Message: Certificate issuance in progress. Temporary certificate issued.
Reason: TemporaryCertificate
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Generated 51m cert-manager Generated new private key
Normal GenerateSelfSigned 51m cert-manager Generated temporary self signed certificate
Normal Cleanup 5m42s cert-manager Deleting old Order resource "letsencrypt-staging-2613163196"
Normal OrderCreated 5m42s cert-manager Created Order resource "letsencrypt-staging-2965106631"
Normal OrderCreated 39s (x2 over 51m) cert-manager Created Order resource "letsencrypt-staging-2613163196"
Normal Cleanup 39s cert-manager Deleting old Order resource "letsencrypt-staging-2965106631"
[I]
~/workspace/ZCRM365/Deployments/Kubernetes/cert-manager · (Deployments±)
How to can I to my certificate to be signed and succesfully issued by letsencrypt CA and active the https encryption active?
What is happening with these logs messages?
kubectl logs -n kube-system cert-manager-6f68b58796-q7txg
0315 13:06:11.027204 1 logger.go:103] Calling Discover
I0315 13:06:11.032299 1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=4095675862,certmanager.k8s.io/acme-http-token=657526223
I0315 13:06:11.046081 1 sync.go:173] propagation check failed: wrong status code '404', expected '200'
I0315 13:06:11.046109 1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 13:06:21.046242 1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I've heared that letsencrypt-staging environment only have test certificates and these are a kind of 'fake certificates' and maybe some clients like my chrome/firefox browser doesn’t trust certificate issuer ...
Is this a reason to I cannot enable https encryption on my domain?
In affirmative case, should I change from staging environment to production environment?
In this question some people talk about that but they emphasize:
that the staging environment should be used just to test that your client is working fine and can generate the challenges, certificates
In my case the http challenge is not generated still in staging environment. :(
here are the annotation I'm usually using for this:
"ingress.kubernetes.io/ssl-redirect": "true",
"certmanager.k8s.io/cluster-issuer": "letsencrypt-production",
# I'd suggest adding these 2 below
"kubernetes.io/tls-acme": "true",
"kubernetes.io/ingress.class": "nginx"
also, you didnt spot this error:
I0315 12:10:58.033431 1 sync.go:173] propagation check failed: wrong status code '404', expected '200'
I'm not sure what is wrong here exactly, your domain name should resolve to your ingress, you should be able to access yourdomain.name/.well-known/acme-challenge/W7-9-KuPao_jg6EF5E2FXitFs8shOEsY5PlT9EEvNxE (this is lets encrypt validation response url, according to your logs)