I see this error when trying to use Gitea with microk8s on Ubuntu 21.10:
$ k logs gitea-0 -c configure-gitea
Wait for database to become avialable...
gitea-postgresql (10.152.183.227:5432) open
...
2021/11/20 05:49:40 ...om/urfave/cli/app.go:277:Run() [I] PING DATABASE postgres
2021/11/20 05:49:45 cmd/migrate.go:38:runMigrate() [F] Failed to initialize ORM engine: dial tcp: lookup gitea-postgresql.default.svc.cluster.local: Try again
I am looking for some clues as to how to debug this please.
The other pods seem to be running as expected:
$ k get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system hostpath-provisioner-5c65fbdb4f-nfx7d 1/1 Running 0 11h
kube-system calico-node-h8tpk 1/1 Running 0 11h
kube-system calico-kube-controllers-f7868dd95-dpp8n 1/1 Running 0 11h
kube-system coredns-7f9c69c78c-cnpkj 1/1 Running 0 11h
default gitea-memcached-584956987c-zb8kp 1/1 Running 0 20s
default gitea-postgresql-0 1/1 Running 0 20s
default gitea-0 0/1 Init:1/2 1 20s
The services are not as expected, since gitea-0 is not starting:
$ k get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 11h
kube-system kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 11h
default gitea-postgresql-headless ClusterIP None <none> 5432/TCP 3m25s
default gitea-ssh ClusterIP None <none> 22/TCP 3m25s
default gitea-http ClusterIP None <none> 3000/TCP 3m25s
default gitea-memcached ClusterIP 10.152.183.15 <none> 11211/TCP 3m25s
default gitea-postgresql ClusterIP 10.152.183.227 <none> 5432/TCP 3m25s
Also see:
https://github.com/ubuntu/microk8s/issues/2741
https://gitea.com/gitea/helm-chart/issues/249
I worked through to the point where I had the logs below, specifically:
cmd/migrate.go:38:runMigrate() [F] Failed to initialize ORM engine: dial tcp: lookup gitea-postgresql.default.svc.cluster.local: Try again
Using k cluster-info dump I saw:
[ERROR] plugin/errors: 2 gitea-postgresql.default.svc.cluster.local.cisco.com. A: read udp 10.1.147.194:56647->8.8.8.8:53: i/o timeout
That led me to test the DNS with dig and 8.8.8.8. That test didn't reveal any errors, in that DNS seemed to work. Even so, DNS seemed suspect.
So then I tried microk8s enable storage dns:<IP address of DNS in lab>, whereas I was previously only using microk8s storage dns. The storage part enables the persistent volumes that the database needs.
The key piece here is the lab DNS server IP address argument when enabling DNS with microk8s.
Related
I am trying to deploy a two node Elastic cluster on my 3 node Kubernetes cluster using the ECK operator (version 2.4.0). I have deployed the operator by applying the CRDS and manifest. I then followed the quickstart steps to get started and planned to add persistant storage after.
I have applied following Elasticsearch definition:
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
namespace: elasticsearch
spec:
version: 8.4.3
nodeSets:
- name: default
count: 2
config:
node.store.allow_mmap: false
A get all on the namespace:
kube#k8s-master-01:~$ kubectl get all -n elasticsearch
NAME READY STATUS RESTARTS AGE
pod/quickstart-es-default-0 0/1 Pending 0 5m47s
pod/quickstart-es-default-1 0/1 Pending 0 5m47s
pod/quickstart-kb-679bf8c9f7-psvn6 1/1 Running 0 5m45s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/quickstart-es-default ClusterIP None <none> 9200/TCP 5m47s
service/quickstart-es-http ClusterIP 10.101.171.44 <none> 9200/TCP 5m47s
service/quickstart-es-internal-http ClusterIP 10.105.134.217 <none> 9200/TCP 5m47s
service/quickstart-es-transport ClusterIP None <none> 9300/TCP 5m47s
service/quickstart-kb-http ClusterIP 10.111.195.203 <none> 5601/TCP 5m46s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/quickstart-kb 1/1 1 1 5m45s
NAME DESIRED CURRENT READY AGE
replicaset.apps/quickstart-kb-679bf8c9f7 1 1 1 5m45s
NAME READY AGE
statefulset.apps/quickstart-es-default 0/2 5m47s
Events output of a describe pod:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m8s (x3 over 7m18s) default-scheduler 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
Logs of the ECK operator:
{"log.level":"info","#timestamp":"2022-10-25T15:18:49.749Z","log.logger":"elasticsearch-controller","message":"Starting reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteration":"10557","namespace":"elasticsearch","es_name":"quickstart"}
{"log.level":"info","#timestamp":"2022-10-25T15:18:49.982Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","namespace":"elasticsearch","pod_name":"quickstart-es-default-0"}
{"log.level":"info","#timestamp":"2022-10-25T15:18:49.982Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","namespace":"elasticsearch","pod_name":"quickstart-es-default-1"}
{"log.level":"info","#timestamp":"2022-10-25T15:18:49.984Z","log.logger":"driver","message":"Elasticsearch cannot be reached yet, re-queuing","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","namespace":"elasticsearch","es_name":"quickstart"}
{"log.level":"info","#timestamp":"2022-10-25T15:18:49.984Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"2.4.0+96282ca9","service.type":"eck","ecs.version":"1.4.0","iteration":"10557","namespace":"elasticsearch","es_name":"quickstart","took":0.23531655}
What am I misunderstanding from the Elastic docs? If relevant I use Traefik as Ingress, Calico as CNI and Metallb as LB.
I'm trying to install Kafka with Strimzy on a local munikube cluster running on Windows 10, to test the impact of different parameters (especially the TLS configuration). Before moving to TLS, i'd simply like to connect to my cluster :)
Here is my yaml configuration :
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 2.3.0
replicas: 1
listeners:
external:
type: nodeport
tls: false
config:
offsets.topic.replication.factor: 1
transaction.state.log.replication.factor: 1
transaction.state.log.min.isr: 1
log.message.format.version: "2.3"
storage:
type: persistent-claim
size: 1Gi
zookeeper:
replicas: 1
storage:
type: persistent-claim
size: 2Gi
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}
For the listener, I firstly started with plain: {} but this only gives me services of type ClusterIP, not accessible from outside minikube (i really need to connect from outside).
I then moved to a listener of kind external.
You can fin below the configuration of the cluster:
kubectl get all -n kafka
NAME READY STATUS RESTARTS AGE
pod/my-cluster-entity-operator-9657c9d79-8hknc 3/3 Running 0 17m
pod/my-cluster-kafka-0 2/2 Running 0 18m
pod/my-cluster-zookeeper-0 2/2 Running 0 18m
pod/strimzi-cluster-operator-f77b7d544-hq5pq 1/1 Running 0 5h22m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/my-cluster-kafka-0 NodePort 10.99.3.204 <none> 9094:30117/TCP 18m
service/my-cluster-kafka-bootstrap ClusterIP 10.106.176.111 <none> 9091/TCP 18m
service/my-cluster-kafka-brokers ClusterIP None <none> 9091/TCP 18m
service/my-cluster-kafka-external-bootstrap NodePort 10.109.235.156 <none> 9094:32372/TCP 18m
service/my-cluster-zookeeper-client ClusterIP 10.97.2.69 <none> 2181/TCP 18m
service/my-cluster-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 18m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/my-cluster-entity-operator 1/1 1 1 17m
deployment.apps/strimzi-cluster-operator 1/1 1 1 5h22m
The IP address of the minikube cluster is 192.168.49.2 (given by minikube ip)
For the while, is everything correct on my configuration ? I cannot connect on the cluster with a producer (i get a timeout error when i try to publish data).
I tried to connect to 192.168.49.2:32372 & 192.168.49.2:30117 and I always get the same timeout error. I also tryed to run
minikube service -n kafka my-cluster-kafka-external-bootstrap
and
minikube service -n kafka my-cluster-kafka-0
and i still get the same error.
What is wrong in what i'm trying to do?
Thanks!
Ok, I got the answser.
I changed the type of the service to LoadBalancer and started minikube tunnel
One other point, as I'm running this on windows, I noticed that if I run everything using powershell it works, if I used an other command line tool (like Moba) it does not work, I don't explain this.
I am trying to make this basic example work on docker desktop on windows, I am not using minikube.
I managed to reach the service using NodePort with:
http://localhost:31429
But when I try http://hello-world.info (made sure to add it in hosts) - 404 not found.
kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 20m
default web NodePort 10.111.220.81 <none> 8080:31429/TCP 6m47s
ingress-nginx ingress-nginx-controller LoadBalancer 10.107.29.182 localhost 80:30266/TCP,443:32426/TCP 19m
ingress-nginx ingress-nginx-controller-admission ClusterIP 10.101.138.244 <none> 443/TCP 19m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 20m
kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
example-ingress <none> hello-world.info 80 21m
I am lost, can someone please help ?
I also noticed that ADDRESS is empty.
Many thanks.
Reproduced this case on Docker Desktop 4.1.1, Windows 10 Pro
Install Ingress Controller for Docker Desktop:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/cloud/deploy.yaml
As I understand it, #dev1334 used an example from Set up Ingress on Minikube with the NGINX Ingress Controller article. I also tried it with some modifications to the original example.
In the example for the example-ingress.yaml file in the spec.rules section, the host hello-world.info is specified. Since Docker Desktop for Windows adds to a hosts file in C:\Windows\System32\drivers\etc\hosts during installation the following entry: 127.0.0.1 kubernetes.docker.internal I changed the host in the example-ingress.yaml from hello-world.info to kubernetes.docker.internal
But Ingress still didn't work as expected due to the following error:
"Ignoring ingress because of error while validating ingress class" ingress="default/example-ingress" error="ingress does not contain a valid IngressClass"
I added this line kubernetes.io/ingress.class: "nginx" to the annotations section in example-ingress.yaml
So, the final version of the example-ingress.yaml file is below.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: kubernetes.docker.internal
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: web
port:
number: 8080
- path: /v2
pathType: Prefix
backend:
service:
name: web2
port:
number: 8080
Test results
C:\Users\Andrew_Skorkin>kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default web-79d88c97d6-c8xnf 1/1 Running 0 112m
default web2-5d47994f45-cxtzm 1/1 Running 0 94m
ingress-nginx ingress-nginx-admission-create-sjdcq 0/1 Completed 0 114m
ingress-nginx ingress-nginx-admission-patch-wccc9 0/1 Completed 1 114m
ingress-nginx ingress-nginx-controller-5c8d66c76d-jb4w9 1/1 Running 0 114m
...
C:\Users\Andrew_Skorkin>kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7d15h
default web NodePort 10.101.43.157 <none> 8080:32651/TCP 114m
default web2 NodePort 10.100.4.84 <none> 8080:30081/TCP 96m
ingress-nginx ingress-nginx-controller LoadBalancer 10.106.138.217 localhost 80:30287/TCP,443:32664/TCP 116m
ingress-nginx ingress-nginx-controller-admission ClusterIP 10.111.208.242 <none> 443/TCP 116m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 7d15h
C:\Users\Andrew_Skorkin>curl kubernetes.docker.internal
Hello, world!
Version: 1.0.0
Hostname: web-79d88c97d6-c8xnf
C:\Users\Andrew_Skorkin>curl kubernetes.docker.internal/v2
Hello, world!
Version: 2.0.0
Hostname: web2-5d47994f45-cxtzm
How can I call a web action from my browser, when my OpenWhisk deployment is on a company sandbox server?
I have an OpenWhisk deployment setup on a company sandbox server using this guide: Deploying OpenWhisk on kind. The server is running CentOS 7. I created a web action and I am able to call with curl -k https://apiHostName:apiHostPort/api/v1/web/guest/demo/hello?name=myName where apiHostName and apiHostPort are the values defined in mycluster.yaml. However, trying to access the above url from my browser returns ERR_CONNECTION_TIMED_OUT.
[root[DEV]#vx3a27 wskcluster]# kubectl -n openwhisk get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
owdev-apigateway ClusterIP 10.96.9.60 A.B.C.D 8080/TCP,9000/TCP 6d1h
owdev-controller ClusterIP 10.96.104.180 <none> 8080/TCP 6d1h
owdev-couchdb ClusterIP 10.96.40.104 <none> 5984/TCP 6d1h
owdev-kafka ClusterIP None <none> 9092/TCP 6d1h
owdev-nginx NodePort 10.96.84.3 A.B.C.D 80:31486/TCP,443:31001/TCP 6d1h
owdev-redis ClusterIP 10.96.84.177 <none> 6379/TCP 6d1h
owdev-zookeeper ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 6d1h
I have tried setting the external API of both my nginx and apigateway services as seen above, where A.B.C.D is the IP of my sandbox obtained with ifconfig. Running both curl -k and the browser with A.B.C.D yields ERR_CONNECTION_REFUSED.
What else can I try to get this to work?
ElasticSearch pod is up and running , Kibana is working but cannot connect to the ElasticSearch ( connection refused in logs )
Dears,
I have 3 nodes Kubernetes cluster with 1 master and 2 workers . i am trying to deploy ElasticSearch (Testing environment only ) for logging.
I followed the details here.
I only set the replicas of the ElasticSearch to 1 instead of two because i do not have enough memory.
- I run Kibana as Nodeport and can open it but it gives error ( Cannot connect to ElasticSearch)
kubectl get svc --namespace=kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch-logging ClusterIP 10.106.231.74 <none> 9200/TCP
kibana-logging NodePort 10.111.199.12 <none> 5601:30461/TCP 5h44m ClusterIP 10.96.0.10 <none>
I tried to ping from the Kibana pod to ElasticSearch pod and it is pingable
I tried to curl http://10.106.231.74:9200 ( service IP ) from all hosts and from the Kibana pod and i get everytime Failed connect to 10.106.231.74:9200; Connection refused
I tried to curl 127.0.0.1:9200 from inside the ElasticSearch pod and also connection refused.
All Yaml files are exactly as provided in above link
I expect any response at least from inside the ElasticSearch pod but all the trials returned connection refused.
The Yamls are all in link and all RBAC rules are ok.