Good afternoon everyone, I have a question about adding monitoring of the application itself to prometheus.
I am using spring boot actuator and see the values for prometheus accordingly: https://example.com/actuator/prometheus
I have raised prometheus via the default helm chart ( helm -n monitor upgrade -f values.yaml pg prometheus-community/kube-prometheus-stack )by adding default values for it:
additionalScrapeConfigs:
job_name: prometheus
scrape_interval: 40s
scrape_timeout: 40s
metrics_path: /actuator/prometheus
scheme: https
Prometheus itself can be found at http://ex.com/prometheus
The deployment.yaml file of my springboot application is as follows:
apiVersion : apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
annotations:
prometheus.io/path: /actuator/prometheus
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
spec:
containers:
- env:
- name: DATABASE_PASSWORD
value: {{ .Values.DATABASE_PASSWORD }}
- name: DATASOURCE_USERNAME
value: {{ .Values.DATASOURCE_USERNAME }}
- name: DATASOURCE_URL
value: jdbc:postgresql://database-postgresql:5432/dev-school
name : {{ .Release.Name }}
image: {{ .Values.container.image }}
ports:
- containerPort : 8080
However, after that prometheus still can't see my values.
Can you tell me what the error could be?
In prometheus-operator,
additionalScrapeConfigs is not used in this way.
According to documentation Additional Scrape Configuration:
AdditionalScrapeConfigs allows specifying a key of a Secret containing additional Prometheus scrape configurations.
The easiest way to add new scrape config is to use a servicemonitor, like the example below:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: example-app
labels:
team: frontend
spec:
selector:
matchLabels:
app: backend
endpoints:
- port: web
Related
I'm deploying my config server and application which uses it to initalize properties.
The following problem I encounter now is the URI to specify where the config-server is located.
Before it was easy like this:
spring.config.import=optional:configserver:https://localhost:8888/
Now in the cluster I try to specify it like this
spring.config.import=optional:configserver:centralconfig-service:8888/
This the service name of the config-server (deployment name is 'centralconfig').
Whenever I try this URL i get an Invalid Url exception.
Invalid URL: centralconfig-service:8888/
I've set up an internal network so they should be able to communicate with eachother.
Anyone knows how to specify the correct URI in the kubernetes-environment?
centralconfig-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: central-config
labels:
app: central-config
spec:
replicas: 1
selector:
matchLabels:
app: central-config
strategy: {}
template:
metadata:
labels:
app: central-config
spec:
containers:
- name: central-config
image: "central-config:latest"
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8888
resources: {}
restartPolicy: Always
serviceAccountName: ""
volumes: null
status: {}
centralconfig.service.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: central-config
labels:
app: central-config
spec:
replicas: 1
selector:
matchLabels:
app: central-config
strategy: {}
template:
metadata:
labels:
app: central-config
spec:
containers:
- name: central-config
image: "central-config:latest"
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8888
resources: {}
restartPolicy: Always
serviceAccountName: ""
volumes: null
status: {}
endpoints:
Thanks in advance
I suspect that missing the "http://" or "https://" prefix is the actual cause of the problem.
So give this one a try:
spring.config.import=optional:configserver:https://centralconfig-service:8888/
I am trying to use "hostNetwork" flag to make my POD communicating over the node's network interface. But after adding this flag the config is not injected from the configmap but the default values from "application.properties" are used instead. Is that expected behavior ?
Note: when I remove the "hostNetwok" flag the app values are injected from the configmap.
I am working on kubernetes V 1.19.
deployment.yaml :
apiVersion: apps/v1
kind: Deployment
metadata:
name: dummy-service-deployment
labels:
app: dummy-service
spec:
replicas: 1
selector:
matchLabels:
app: dummy-service
template:
metadata:
labels:
app: dummy-service
spec:
hostNetwork: true
containers:
- name: dummy-service
image: dummyService:2.10.0
ports:
- containerPort: 8081
bootstrap.yaml:
spring:
application:
name: dummy-service
cloud:
kubernetes:
reload:
enabled: true
mode: polling
period: 10000
config:
sources:
- name: dummy-service-cm
Many thanks in advance ...
I have tried to deploy the producer-service app with MySQL database in the Kubernetes cluster. When i try to deploy producer app then the following validation error has thrown.
error: error validating "producer-deployment.yml": error validating data: apiVersion not set; if you choose to ignore these errors, turn validation off with --validate=false
producer-deployment.yml
apiVerion: v1
kind: Service
metadata:
name: producer-app
labels:
name: producer-app
spec:
ports:
-nodePort: 30163
port: 9090
targetPort: 9090
protocol: TCP
selector:
app: producer-app
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: producer-app
spec:
selector:
matchLabels:
app: producer-app
replicas: 3
template:
metadata:
labels:
app: producer-app
spec:
containers:
- name: producer
image: producer:1.0
ports:
- containerPort: 9090
env:
- name: DB_HOST
valueFrom:
configMapKeyRef:
name: db-config
key: host
- name: DB_NAME
valueFrom:
configMapKeyRef:
name: db-config
key: name
- name: DB_USERNAME
valueFrom:
secretKeyRef:
name: db-user
key: username
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-user
key: password
i have tried to find the error or typo within the config file but still, couldn't. What is wrong with the producer-deployment.yml file
Multiple issues:
It would be apiVersion: v1 not apiVerion: v1 in the Service
wrong Spec.ports formation of Service. As nodePort, port, targetPort and protocol are under the ports as a list but your did wrong formation.
your service yaml should be like below:
apiVersion: v1
kind: Service
metadata:
name: producer-app
labels:
name: producer-app
spec:
ports:
- nodePort: 30163
port: 9090
targetPort: 9090
protocol: TCP
selector:
app: producer-app
type: NodePort
So your overall yaml should be:
apiVersion: v1
kind: Service
metadata:
name: producer-app
labels:
name: producer-app
spec:
ports:
- nodePort: 30163
port: 9090
targetPort: 9090
protocol: TCP
selector:
app: producer-app
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: producer-app
spec:
selector:
matchLabels:
app: producer-app
replicas: 3
template:
metadata:
labels:
app: producer-app
spec:
containers:
- name: producer
image: producer:1.0
ports:
- containerPort: 9090
env:
- name: DB_HOST
valueFrom:
configMapKeyRef:
name: db-config
key: host
- name: DB_NAME
valueFrom:
configMapKeyRef:
name: db-config
key: name
- name: DB_USERNAME
valueFrom:
secretKeyRef:
name: db-user
key: username
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-user
key: password
Please change the first line in producer-deployment.yml. Letter s is missing.
From
apiVerion: v1
To
apiVersion: v1
There is a typo in the first line: "apiVerion" should be "apiVersion".
Your first error(there are more than 1) just point you to the place where you should start your investigation from..
error validating data: apiVersion not set;
As you know, each object in kubernetes has its own apiVersion.
Check Understanding Kubernetes Objects, especially Required Fields part:
In the .yaml file for the Kubernetes object you want to create, you'll
need to set values for the following fields:
apiVersion - Which version of the Kubernetes API you're using to
create this object
kind - What kind of object you want to create
metadata - Data that helps uniquely identify the object, including a
name string, UID, and optional namespace
spec - What state you desire
for the object The precise format of the object spec is different for
every Kubernetes object, and contains nested fields specific to that
object.
The Kubernetes API Reference can help you find the spec format
for all of the objects you can create using Kubernetes.
You can check Latest 1.20 API here
These values are mandatory and you wont be able to create object without them. So please, next time read more carefully errors you receive.
I am using spring-cloud-starter-kubernetes-all dependency for reading config map from my spring boot microservices and its working fine.
After modifying the config map i am using refresh endpoint
minikube servie list # to get the servive url
curl http://192.168.99.100:30824/actuator/refresh -d {} -H "Content-Type: application/json"
it working as expected and application loads configmap changes.
Issue
The above working fine if i have only 1 pod of my application but when i do use more that 1 pods only 1 pods picks the changes not all.
In below example only i pod picks the changes
[message-producer-5dc4b8b456-tbbjn message-producer] Say Hello to the World12431
[message-producer-5dc4b8b456-qzmgb message-producer] Say Hello to the World
minkube deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: message-producer
labels:
app: message-producer
spec:
replicas: 2
selector:
matchLabels:
app: message-producer
template:
metadata:
labels:
app: message-producer
spec:
containers:
- name: message-producer
image: sandeepbhardwaj/message-producer
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: message-producer
spec:
selector:
app: message-producer
ports:
- protocol: TCP
port: 80
targetPort: 8080
type: LoadBalancer
configmap.yml
kind: ConfigMap
apiVersion: v1
metadata:
name: message-producer
data:
application.yml: |-
message: Say Hello to the World
bootstrap.yml
spring:
cloud:
kubernetes:
config:
enabled: true
name: message-producer
namespace: default
reload:
enabled: true
mode: EVENT
strategy: REFRESH
period: 3000
configuration
#ConfigurationProperties(prefix = "")
#Configuration
#Getter
#Setter
public class MessageConfiguration {
private String message = "Default message";
}
rbac
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
namespace: default # "namespace" can be omitted since ClusterRoles are not namespaced
name: service-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["services"]
verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
# This cluster role binding allows anyone in the "manager" group to read secrets in any namespace.
kind: ClusterRoleBinding
metadata:
name: service-reader
subjects:
- kind: User
name: default # Name is case sensitive
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: service-reader
apiGroup: rbac.authorization.k8s.io
This is happening because when you hit curl http://192.168.99.100:30824/actuator/refresh -d {} -H "Content-Type: application/json" kubernetes will send that request to one of the pods behind the service via round robin load balancing.
You should use the property source reload feature by setting spring.cloud.kubernetes.reload.enabled=true. This will reload the property whenever there is a change in the config map hence you don't need to use the refresh endpoint.
I am trying to my spring boot micro service in GKE Cluster with istio 1.1.5 latest version as of now. It throws error and pod never spins up. If I run it as a separate service in Kubernetes engine it works perfectly but with isito, it does not work. The purpose for using istio is to host multiple microservices and to use the feature istio provides. Here is my yaml file:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: revenue
spec:
replicas: 1
template:
metadata:
labels:
app: revenue-serv
tier: backend
track: stable
spec:
containers:
- name: backend
image: "gcr.io/finacials/revenue-serv:latest"
imagePullPolicy: Always
ports:
- containerPort: 8081
livenessProbe:
httpGet:
path: /
port: 8081
initialDelaySeconds: 15
timeoutSeconds: 30
readinessProbe:
httpGet:
path: /
port: 8081
initialDelaySeconds: 15
timeoutSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
name: revenue-serv
spec:
ports:
- port: 8081
#targetPort: 8081
#protocol: TCP
name: http
selector:
app: revenue-serv
tier: backend
type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gateway
annotations:
kubernetes.io/ingress.class: "istio"
spec:
rules:
- http:
paths:
- path: /revenue/.*
backend:
serviceName: revenue-serv
servicePort: 8081
Thanks for your valuable feedback.
I have found the issue. I removed readynessProbe and livenessProbe and created ingressgateway and virtual service. It worked.
deployment & service:
#########################################################################################
# This is for deployment - Service & Deployment in Kubernetes ################
# Author: Arindam Banerjee ################
#########################################################################################
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: revenue-serv
namespace: dev
spec:
replicas: 1
template:
metadata:
labels:
app: revenue-serv
version: v1
spec:
containers:
- name: revenue-serv
image: "eu.gcr.io/rcup-mza-dev/revenue-serv:latest"
imagePullPolicy: Always
ports:
- containerPort: 8081
---
apiVersion: v1
kind: Service
metadata:
name: revenue-serv
namespace: dev
spec:
ports:
- port: 8081
name: http
selector:
app: revenue-serv
gateway.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: worldcup-serv-gateway
namespace: dev
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
virtual-service.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: revenue-serv-virtualservice
namespace: dev
spec:
hosts:
- "*"
gateways:
- revenue-serv-gateway
http:
- route:
- destination:
host: revenue-serv