Following this
Setup:
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-19T22:12:47Z", GoVersion:"go1.12.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.7-gke.10", GitCommit:"8d9b8641e72cf7c96efa61421e87f96387242ba1", GitTreeState:"clean", BuildDate:"2019-04-12T22:59:24Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}
knative-serve & Istio is v0.5.2
cert-manager is v0.7 applyed with --validate=false as k8s is 1.12.7
Cert-manager ClusterIssuer status:
│ status: │
│ conditions: │
│ - lastTransitionTime: "2019-04-29T21:29:40Z" │
│ message: Certificate issuance in progress. Temporary certificate issued. │
│ reason: TemporaryCertificate │
│ status: "False" │
│ type: Ready
I have done as in the documentation, but setting up Google DNS not described
I have manually created a DNS in Google DNS consule.
My domain is pointing at the nameservers and I can ping the right server ip address,
When creating the DNS I added a record set:
*.mydomain.com. A 300 x.x.x.x
Note: also tried without " * "
I have seen here, that they talk abaout setting TXT?
Do you know how to make this(cert-manager & TLS) it work?
First, look at the logs being issued by the cert manager pod kubectl logs -n <namespace> pod/podname.
Cert manager will tell you why the challenge is failing.
One common reason is the rate limiting by Letsencrypt and you have to wait for 7 days.
You can also view this same issue on github https://github.com/jetstack/cert-manager/issues/1745
Related
I created a private Docker registry POD in my Kubernetes cluster.
Here are the relevant settings for the pod:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
private-repository-k8s-686564966d-8grr8 1/1 Running 2 (7h10m ago) 9d
$ kubectl describe pods private-repository-k8s-686564966d-8grr8
...
Containers:
private-repository-k8s:
Container ID: docker://faadba7513c6a1bae6ab96480fcc230ae94a1c8e27c20928f3f93bfd2e7b7714
Image: registry:2
Image ID: docker-pullable://registry#sha256:265d4a5ed8bf0df27d1107edb00b70e658ee9aa5acb3f37336c5a17db634481e
Port: 5000/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 05 Oct 2021 14:49:07 -0700
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Sun, 26 Sep 2021 16:36:48 -0700
Finished: Tue, 05 Oct 2021 14:48:43 -0700
Ready: True
Restart Count: 2
Environment:
REGISTRY_HTTP_TLS_CERTIFICATE: /certs/registry.crt
REGISTRY_HTTP_TLS_KEY: /certs/registry.key
Mounts:
/certs from certs-vol (rw)
/var/lib/registry from registry-vol (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mqknd (ro)
...
Volumes:
certs-vol:
Type: HostPath (bare host directory volume)
Path: /opt/certs
HostPathType: Directory
registry-vol:
Type: HostPath (bare host directory volume)
Path: /opt/registry
HostPathType: Directory
kube-api-access-mqknd:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
I generated the certs on the master as follows:
$ sudo openssl req -newkey rsa:4096 -nodes -sha256 -keyout \
/opt/certs/registry.key -x509 -days 365 -out /opt/certs/registry.crt
Then the folder with the crt and key files are shared via NFS mount across all of the workers.
When I try to push an image from outside the k8s cluster and I get the following error:
$ docker push k8s-master:31320/nginx:1.17 The push refers to repository [k8s-master:31320/nginx]
Get "https://k8s-master:31320/v2/": x509: certificate is not valid for any names, but wanted to match k8s-master
The logs from the POD show this:
$ kubectl logs private-repository-k8s-686564966d-8grr8 -f
...
2021/10/06 05:06:02 http: TLS handshake error from 10.108.82.192:28058: remote error: tls: bad certificate
This proves to me that the request is hitting the POD, but TLS certs weren't setup properly.
I'm doing trying to push the Docker image from my MacOS client to this private Docker registry on a k8s server (each node in the server running Ubuntu).
I'm a bit shaky on the TLS stuff, but my understanding is that I'm using a self-signed cert (which should be fine as I'm only accessing this from my internal network). But I assume I need to do something from my Mac client to setup the TLS certs in order to access the registry. I have already tried adding the crt and key files to my Keychain and that didn't work. I cannot figure out what to do here.
I'm using these instructions:
https://www.linuxtechi.com/setup-private-docker-registry-kubernetes/
I'm running k8s v1.22.0. I have 4 VMs running Ubuntu 20.04.2 LTS inside a single rack server using VMware ESXi: 1 master, 3 worker VMs. I'm trying to push the docker image from my MacBook.
First, I found the CN (Common Name) was not setup property in the certificate (reference: https://github.com/docker/for-linux/issues/248). Once I regenerated the certificate I hit this issue:
$ docker push k8s-master:31320/nginx:1.17 The push refers to repository [k8s-master:31320/nginx]
Get "https://k8s-master:31320/v2/": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
Then I found I needed to add SAN (subjectAltName) to the certificate. I did this as follows:
$ sudo openssl req -newkey rsa:4096 -nodes -sha256 -keyout /opt/certs/registry.key -x509 -days 365 -out /opt/certs/registry.crt -addext "subjectAltName = DNS:k8s-master, DNS:k8s-master.local"
I restarted the registry pod and then I ran into this error:
$ docker push k8s-master:31320/nginx:1.17
The push refers to repository [k8s-master:31320/nginx]
Get "https://k8s-master:31320/v2/": x509: certificate signed by unknown authority
At this point, I realized MacOS client needed the certificate installed into the Keychain. I downloaded the registry.crt file and install it in Keychain (drag and drop). I also had to go into the Keychain, double-clicked on the certificate, opened the "Trust" drop down and selected "Always Trust". Then I restarted Docker on my MacOS.
At this point push started to work:
$ docker push k8s-master:31320/nginx:1.17
The push refers to repository [k8s-master:31320/nginx]
65e1ea1dc98c: Pushed
88891187bdd7: Pushed
6e109f6c2f99: Pushed
0772cb25d5ca: Pushed
525950111558: Pushed
476baebdfbf7: Pushed
1.17: digest: sha256:39065444eb1acb2cfdea6373ca620c921e702b0f447641af5d0e0ea1e48e5e04 size: 1570
We are installing Anthos on VMWare platform and now we have an error in the Admin Cluster deployment procedure of the Seesaw Loadbalancer in HA.
The Deploy of two Seesaw VMs has been created with success, but when checking the health check we get the following error 403:
ubuntu#anth-mgt-wksadmin:~$ gkectl create loadbalancer --config admin-cluster.yaml -v5
Reading config with version "v1"
- Validation Category: OS Images
- [SUCCESS] Admin cluster OS images exist
- Validation Category: Admin Cluster VCenter
- [SUCCESS] Credentials
- [SUCCESS] DRS enabled
- [SUCCESS] Hosts for AntiAffinityGroups
- [SUCCESS] vCenter Version
- [SUCCESS] ESXi Version
- [SUCCESS] Datacenter
- [SUCCESS] Datastore
- [SUCCESS] Resource pool
- [SUCCESS] Folder
- [SUCCESS] Network
- Validation Category: Bundled LB
- [FAILURE] Seesaw validation: admin cluster lb health check failed: LB "10.25.94.229" is not healthy: received 403 Forbidden
- Validation Category: Network Configuration
- [SUCCESS] CIDR, VIP and static IP (availability and overlapping)
- Validation Category: GCP
- [SUCCESS] GCP service
- [SUCCESS] GCP service account
Some validation results were FAILURE or UNKNOWN. Check report above.
Preflight check failed with preflight check failed
Exit with error:
also this simple test give the same result
root#jump-mgm-wks:~# wget http://10.25.94.229
--2021-07-27 13:56:04-- http://10.25.94.229/
Connecting to 10.173.119.123:8080... connected.
Proxy request sent, awaiting response... 403 Forbidden
2021-07-27 13:56:04 ERROR 403: Forbidden.
We get also this error on log:
ubuntu#anth-mgt-bigip1:/var/log/seesaw$ cat seesaw_ha.anth-mgt-bigip1.root.log.ERROR.20210727-123208.1738
Log file created at: 2021/07/27 12:32:08
Running on machine: anth-mgt-bigip1
Binary: Built with gc go1.15.11 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0727 12:32:08.331013 1738 main.go:86] config: Failed to retrieve Config: HAConfig: Dial failed: dial unix /var/run/seesaw/engine/engine.sock: connect: no such file or directory
Solved after the recreation of the admin workstation with the following parameter.
gkectl delete loadbalancer --config admin-cluster.yaml --seesaw-group-file seesaw-for-gke-admin.yaml
now save the following files from ubuntu home director of the admin workstation to the jump-mgm-wks in /backup
amin-cluster.yaml
admin-cluster-ipblock.yaml
admin-seesaw-ipblock.yaml
gkeadm delete admin-workstation
gkeadm create admin-workstation --auto-create-service-accounts
gkectl create loadbalancer --config admin-cluster.yaml
I'm working on a repo which is serving a create-react-app from a node endpoint. So, the react app is nested as a child directory:
.
├── Procfile
├── frontend
│ ├── README.md
│ ├── build
│ ├── package.json <---- "proxy": "http://localhost:$PORT"
│ ├── public
│ ├── src
│ │ ├── App.css
│ │ ├── App.js
│ │ └── // etc...
│ └── .env <----- frontend env file, removed PORT value from here
├── package.json
├── src
│ ├── app.js
│ ├── server.js
│ └── // etc...
├── .env <--- backend env file, PORT=9000 for node
├── static.json
└── yarn.lock
With port value removed from the .env file, CRA runs on port 3000. If I hardcode port 9000 instead of $PORT, then the proxy works properly in development.
However, when deploying to production, I want the frontend to proxy Heroku's dynamic port number, this is one example:
Heroku seems to ignore the port value even if I intentionally define it in the env in their website, with a value of 9000.
My question is how do I define the proxy on the frontend without having CRA to instance at that port number, e.g. apply PORT=9000 in the frontend .env but have CRA load at port 3000.
I've tried defining the port number in the script, while making sure that I've defined PORT=9000 in the frontend env:
"scripts": {
"start": "export PORT=3000 && react-scripts start",
CRA will load at 3000, but I get a proxy error:
Heroku doesn't let you chose your port, but rather allocates a port for your app to use as an environment variable. Read more here
Each web process simply binds to a port, and listens for requests coming in on that port. The port to bind to is assigned by Heroku as the PORT environment variable.
Remove all hardcoded PORT variables
It's not ideal to use $PORT in your package.json file as you cannot add logic to it. In your nodejs app read the port variable like so:
const PORT = process.env.PORT || 3000
This will set the port variable to whatever is in the environment variable PORT and if it is not set, will default to 3000
It is not efficient to serve a production app with CRA
Don't run two servers for react and nodejs, instead use your nodejs app to serve a production built react app
const express = require('express')
const path = require('path')
const app = express()
// All your other routes go here
app.use('/', express.static(path.join(__dirname,'client/build'))) // this must be the last one
NOTE: This is assuming your react app is built inside client/build relative to your project root
The proxy setting is only for development convenience and will not work if the app is not served by CRA
Make heroku build your react app during buildtime with:
npm --prefix client run build # or if you use yarn
yarn --cwd client build
in your outer package.json file's build script
You start script is going to run your nodejs server:
"scripts": {
"start": "node src/server.js",
"build": "npm --prefix client run build"
}
Don't commit your .env files to heroku, instead set environment variables directly using heroku config:set KEY=VALUE if you have heroku cli or use the dashboard settings
NOTE: Do this before pushing your code to have these variables accessible during buildtime of the react app
I'm trying to get elasticsearch running on Kubernetes 1.16 with Helm 3 on GKE. I'm aware that both 1.16 and 3 aren't supported yet. I want to prepare a PR to make it compatible. I'm using the helm charts from https://github.com/elastic/helm-charts.
If I use the original chart 7.6.1 the pod creation fails due to create Pod elasticsearch-master-0 in StatefulSet elasticsearch-master failed error: pods "elasticsearch-master-0" is forbidden: unable to validate against any pod security policy: [spec.volumes[1]: Invalid value: "projected": projected volumes are not allowed to be used]. Therefore I created the following patch:
diff --git a/elasticsearch/values.yaml b/elasticsearch/values.yaml
index 053c020..fd9c37b 100755
--- a/elasticsearch/values.yaml
+++ b/elasticsearch/values.yaml
## -107,6 +107,7 ## podSecurityPolicy:
- secret
- configMap
- persistentVolumeClaim
+ - projected
persistence:
enabled: true
With this patch on master/d9ccb5a and tag 7.6.1 (tried both) the pods quickly get into unhealthy state due to failed to resolve host [elasticsearch-master-headless] caused by a java.net.UnknownHostException: elasticsearch-master-headless.
I don't understand why the name resolution doesn't work as there's no change introduced in 1.16 which changes name resolution with Kubernetes names afaik. If I try to ping elasticsearch-master-headless from a shell in the pod started with kubectl exec, I can't reach it neither.
I tried to contact the nameserver in /etc/resolv.conf with telnet because it allows specifying a specific port:
[elasticsearch#elasticsearch-master-1 ~]$ cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local us-central1-a.c.myproject.internal c.myproject.internal google.internal
nameserver 10.23.240.10
options ndots:5
[elasticsearch#elasticsearch-master-1 ~]$ telnet 10.23.240.10
Trying 10.23.240.10...
^C
[elasticsearch#elasticsearch-master-1 ~]$ telnet 10.23.240.10 53
Trying 10.23.240.10...
telnet: connect to address 10.23.240.10: Connection refused
I obfuscated the project ID with myproject.
The patch is already proposed to be merged upstream together with other changes at https://github.com/elastic/helm-charts/pull/496.
This is caused by the pod kube-dns crashing due to
F0315 20:01:02.464839 1 server.go:61] Failed to create a kubernetes client: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied
Since Kubernetes 1.16 is only available in the rapid channel of GKE and it's a system pod, I consider this a bug.
I'll update this answer if I find the energy to file a bug.
Chances are that there is a firewall(firewalld) blocking 53/udp,tcp or an issue with CoreDNS pod in the cluster where you are performing the test.
I have installed Filebeat deb package in Client-server(Linux Wind-River) and ELK in Elk-server(Ubuntu-16.04-server). The problem is, I can't receive logs from Client-server. I checked the network statistics and it seems 5044 port(Listening port) in ELK server is LISTENING. I can ping from both sides. I also have ssh connection in both directions.
This is the link which I used to install these packages on servers.
My Filebeat configurations:
filebeat.prospectors:
- type: log
# Change to true to enable this prospector configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths. paths:
- /var/log/filebeat/*
- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*
document_type: log
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["192.168.10.3:5044"]
proxy_url: socks5://wwproxy.seln.ete.ericsson.se:808
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
# Certificate for SSL client authentication
ssl.certificate: "/etc/pki/tls/certs/logstash-forwarder.crt"
# Client Certificate Key
ssl.key: "/etc/pki/tls/private/logstash-forwarder.key"
I figured out the Error! The problem is the server IP in openssl.cnf should be the IP address of bridged Interface. And the certificate generated with this openssl.cnf should be used in both the servers. Further, I also shared the .key generated in ELK server to Client-server to be more secured/authenticate.