could anyone help me out how to deploy kibana on Kubernetes cluster and to connect with pre-existing elasticsearch ?? I couldn't find any appropriate doc on google
Here's a bare minimal to help you get started, just changed elasticsearch below to your own elasticsearch service name in your cluster.
apiVersion: v1
kind: Service
metadata:
name: kibana
spec:
ports:
- port: 5601
selector:
run: kibana
---
apiVersion: v1
kind: Pod
metadata:
labels:
run: kibana
name: kibana
spec:
containers:
- env:
- name: ELASTICSEARCH_HOSTS
value: http://elasticsearch:9200 # <-- change to your own es service url
image: docker.elastic.co/kibana/kibana:7.16.3
imagePullPolicy: IfNotPresent
name: kibana
ports:
- containerPort: 5601
restartPolicy: OnFailure
Related
the following is my elasticsearch deployment yaml file for version 7.9 :
apiVersion: apps/v1
kind: Deployment
metadata:
name: elasticsearch
labels:
app: elasticsearch
spec:
selector:
matchLabels:
app: elasticsearch
replicas: 1
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- name: elasticsearch
image: elasticsearch:7.9.0
resources:
requests:
memory: 4Gi
limits:
memory: 5Gi
ports:
- containerPort: 9200
- containerPort: 9300
env:
- name: discovery.type
value: single-node
its quit working fine.But when I try to upgrade the elasticsearch version to 8.0.0 in the same above yaml image: elasticsearch:8.0.0 , I cannot access my elasticsearch with my URL. Or is there any changes to be made for e.s version 8.0.0.
Below is my service.yaml:
apiVersion: v1
kind: Service
metadata:
name: esservice
spec:
selector:
app: elasticsearch
type: NodePort
ports:
- port: 9200
targetPort: 9200
nodePort: 31200
I have deployed elastic APM server into kubernetes and was trying to expose it through nginx ingress controller. Following is my configuration:
---
apiVersion: v1
kind: ConfigMap
metadata:
namespace: elastic
name: apm-server-config
labels:
k8s-app: apm-server
data:
apm-server.yml: |-
apm-server:
host: "0.0.0.0:8200"
setup.kibana:
enabled: "true"
host: "kibana:5601"
output.elasticsearch:
hosts: ["elastic:9200"]
---
#Deployment Configuration
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
name: apm-server
env: msprod
state: common
name: apm-server
namespace: elastic
spec:
replicas: 1
minReadySeconds: 10
selector:
matchLabels:
app: apm-server
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: apm-server
spec:
containers:
- image: docker.elastic.co/apm/apm-server:7.12.1
imagePullPolicy: Always
env:
- name: output.elasticsearch.hosts
value: "http://elastic:9200"
name: apm-server
ports:
- name: liveness-port
containerPort: 8200
volumeMounts:
- name: apm-server-config
mountPath: /usr/share/apm-server/apm-server.yml
readOnly: true
subPath: apm-server.yml
resources:
limits:
cpu: 250m
memory: 1024Mi
requests:
cpu: 100m
memory: 250Mi
volumes:
- name: apm-server-config
configMap:
name: apm-server-config
nodeSelector:
env: prod
restartPolicy: Always
terminationGracePeriodSeconds: 30
---
#Service Configuration
apiVersion: v1
kind: Service
metadata:
labels:
app: apm-server
name: apm-server
namespace: elastic
spec:
ports:
- port: 8200
targetPort: 8200
name: http
nodePort: 31000
selector:
app: apm-server
sessionAffinity: None
type: NodePort
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
namespace: elastic
name: gateway-ingress-apm
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: my.domain.com
http:
paths:
- path: /apm
backend:
serviceName: apm-server
servicePort: 8200
The pod is running and I am able to hit APM server using kubectl port-forward.
But when I am accessing the apm server with https://my.domain.com/apm then I am getting page not found error in browser and following error in APM pod:
{"log.level":"error","#timestamp":"2021-10-21T06:22:00.198Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":60},"message":"404 page not found","url.original":"/apm","http.request.method":"GET","user_agent.original":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36","source.address":"10.148.7.7","http.request.body.bytes":0,"http.request.id":"9294124a-5356-4b2c-ba8e-c0a589b23571","event.duration":110881,"http.response.status_code":404,"error.message":"404 page not found","ecs.version":"1.6.0"}
The error is coming because there is no context path configured in APM. I have gone through the APM documentation and couldn't find a way to configure context path in the apm server. Please help.
Posting this as answer out of comments.
Initial ingress rule passes the same path /apm to the APM service, which is confirmed by error in APM pod's logs - "message":"404 page not found","url.original":"/apm"
To fix it, nginx ingress has rewrite annotation. The way it works is described in the link with example.
Final ingress.yaml should look like:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
namespace: elastic
name: gateway-ingress-apm
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$2 # adding captured group
spec:
rules:
- host: my.domain.com
http:
paths:
- path: /apm(/|$)(.*) # to have captured group works correctly
backend:
serviceName: apm-server
servicePort: 8200
What happens here is requests sent to my.domain.com/apm goes to the service on / path.
Captured group allows to preserve correct paths, for instance if the request goes to my.domain.com/apm/something, ingress will translate it to /something which will be passed to the service.
I'm using ECK 1.5.0 and I have to use Ingress to expose Elasticsearch. But I'm getting a 502 gateway when I go to the url (http://my-db-url.com). I have confirmed the database is running fine and able to collect / display data.
I was only able to find solutions to exposing Kibana with Ingress on the web but those were not working for me.
Heres my elasticsearch.yaml (contains Elasticsearch object and Ingress object):
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: my-db
spec:
version: 7.12.0
volumeClaimDeletePolicy: DeleteOnScaledownOnly
nodeSets:
- name: default
count: 3
config:
node.store.allow_mmap: false
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: longhorn
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-db-ingress
namespace: my-namespace
annotations:
nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
nginx.ingress.kubernetes.io/upstream-vhost: "$http_host"
spec:
rules:
- host: my-db-url.com
http:
paths:
- backend:
serviceName: my-db-es-http
servicePort: 9200
Turns out the same question was asked here: https://discuss.elastic.co/t/received-plaintext-http-traffic-on-an-https-channel-closing-connection/271380
and the solution is to force HTTPS using annotations, which for the nginx ingress controller can be found here: https://github.com/elastic/helm-charts/issues/779#issuecomment-781431675
I'm new in kubernetes and docker world :)
I try to deploy our application in docker in kubernetes, but i can't connect to external mysql database..
my steps:
1, Install kubernetes with kubeadm in our new server.
2, Create a docker image from our application with mvn spring-boot:build-image
3, I create a deployment and service yaml to use image.
Deployment YAML:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
labels:
app: demo-app
name: demo-app
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: demo-app
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: demo-app
spec:
containers:
- image: demo/demo-app:0.1.05-SNAPSHOT
imagePullPolicy: IfNotPresent
name: demo-app-service
env:
- name: SPRING_DATASOURCE_URL
value: jdbc:mysql://mysqldatabase/DBDEV?serverTimezone=Europe/Budapest&useLegacyDatetimeCode=false
ports:
- containerPort: 4000
volumeMounts:
- name: uploads
mountPath: /uploads
- name: ssl-dir
mountPath: /ssl
volumes:
- name: ssl-dir
hostPath:
path: /var/www/dev.hu/backend/ssl
- name: uploads
hostPath:
path: /var/www/dev.hu/backend/uploads
restartPolicy: Always
Service YAML:
apiVersion: v1
kind: Service
metadata:
labels:
app: demo-app
name: demo-app
namespace: default
spec:
ports:
- port: 4000
name: spring
protocol: TCP
targetPort: 4000
selector:
app: demo-app
sessionAffinity: None
type: LoadBalancer
4, Create an endpoints and Service YAML, to communicate to outside:
kind: Endpoints
apiVersion: v1
metadata:
name: mysqldatabase
subsets:
- addresses:
- ip: 10.10.0.42
ports:
- port: 3306
---
kind: Service
apiVersion: v1
metadata:
name: mysqldatabase
spec:
type: ClusterIP
ports:
- port: 3306
targetPort: 3306
But it's not working, when i going to see logs i see spring cant connect to database.
Caused by: java.net.UnknownHostException: mysqldatabase
at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at com.mysql.cj.protocol.StandardSocketFactory.connect(StandardSocketFactory.java:132)
at com.mysql.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:63)
thanks for any helps
hold on. you don't create endpoints yourself. endpoints are registered by kubernetes when a service has matching pods. right now, you have deployed your application and exposed it via a service.
if you want to connect to your mysql database via service it needs to be deployed and kubernetes as well. if it is not hosted on kubernetes you will need a hostname or the ip address of the database and adapt your SPRING_DATASOURCE_URL accordingly!
I am trying to run a kubernetes closer locally using minikube. This is my first try with kubernetes. Therefore
I am not familiar with all aspects of it.
I am trying to deploy a spring boot app which connects to elastic search server.
springboot deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
labels:
app: myapp
spec:
replicas: 1
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: myapp1:latest
imagePullPolicy: Never
Elastic search sever deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: elasticsearch
spec:
selector:
matchLabels:
run: elasticsearch
replicas: 1
template:
metadata:
labels:
run: elasticsearch
spec:
containers:
- image: docker.elastic.co/elasticsearch/elasticsearch:6.6.1
name: elasticsearch
imagePullPolicy: IfNotPresent
env:
- name: discovery.type
value: single-node
- name: cluster.name
value: elasticsearch
ports:
- containerPort: 9300
name: nodes
- containerPort: 9200
name: client
Exposed elastic search service as follows
apiVersion: v1
kind: Service
metadata:
name: elasticsearch
labels:
service: elasticsearch
spec:
ports:
- name: client
port: 9200
protocol: TCP
targetPort: 9200
- name: nodes
port: 9300
protocol: TCP
targetPort: 9300
type: NodePort
selector:
run: elasticsearch
Similarly, I exposed service of springboot app also.
Now I am wondering how can I connect from springboot services to elastic search service.
When springbbot and elastic search was normal deployment on the same machine ( not in kubernetes), I connected using as
RestClient.builder(new HttpHost("localhost", 9200))
.build();
What's the best way to connect to the elastic search from springboot in kubernetes?
Save the ip of the elastic search service in an environment variable and use it in springboot or use the service name of the elastic search service?
Please advice
You should be able to get to the service, from within the cluster, using:
http://servicename.servicenamespace:serviceport
Kubernetes dns internal to the cluster will resolve the service name as a host name. If they are in the same namespace you probably don't need the serivcenamespace
Given the yaml above and if you used the default namespace for both elasticsearch and your myapp, then myapp process can connect via:
http://elasticsearch:9200
Now, I am able to connect to the elastic search from my springboot app.
Somehow springboot is not able to connect it using http://elasticsearch:9200.
Instead, I pass the ip and port of the exposed elastic search service (9200 port's equivalent output of minikube service elasticsearch --url) (ip of the node:exposed Nodeport of 9200)to every springboot request which connects to the elastic search service and now I am able to connect it.
I know that it's not the ideal solution and I do not know why it can not resolve the servicename to ip. But atleast I am able to proceed.
It will be helpful, if somebody can suggest someways to fix/diagnose the issue
******* UPDATE ******
Finally springboot is able to connect with elastic search using http://elasticsearch:9200. I do not know which change done by me fixed that. I changed my elasticsearch from a Deployment to Statefulset as shown in the following yaml but that change was not done to fix this issue.
Another change which I did is in the label. I changed it from "run":"elasticsearch" to "app":"elastcisearch" but I do not know whether this helped in that. (I am going to read more labels change and will see whether this has any effect).
Please see the final elasticsearch.yaml file ( more explanation of the file can be seen at Minikube - Not able to get any result from elastic search to if it uses existing indices)
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: elasticsearch
spec:
serviceName: "elasticsearch"
replicas: 1
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
initContainers:
- name: set-permissions
image: registry.hub.docker.com/library/busybox:latest
command: ['sh', '-c', 'mkdir -p /usr/share/elasticsearch/data && chown 1000:1000 /usr/share/elasticsearch/data' ]
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:6.6.1
env:
- name: discovery.type
value: single-node
ports:
- containerPort: 9200
name: client
- containerPort: 9300
name: nodes
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
volumes:
- name: data
hostPath:
path: /indexdata
---
apiVersion: v1
kind: Service
metadata:
name: elasticsearch
labels:
service: elasticsearch
spec:
ports:
- port: 9200
name: client
- port: 9300
name: nodes
type: NodePort
selector:
app: elasticsearch