I have spring boot application and deployed in kubernetes with 4 replica pods configurations. Now I have spring profiles: api and scheduler. When api spring profile is active then only pods will run those apis and when scheduler profile is active the pods will run the scheduler code.
Now I want 2 of my pods to run with api and other 2 pods with scheduler spring profile. How can I achieve that in kubernetes?
One of the option is to define two pod yaml and override the docker file CMD and entrypoint. Yaml file 'command and args' will override whatever mention in docker's Entrypoint.
For api container
spec:
containers:
- name: api-container
image: nameOfImage
command: ["java"]
args: ["-jar", "application.jar", "--spring.profiles.active=api"]
Similarly you can define for scheduler container.
Related
I want to mark the pod ready only when there are enough connections created and the pod is ready to handle requests. The connections are created at the startup of my Springboot application. How can I make sure that the pod is ready only after the connections are created?
You can write a small python (or other) script which checks if the connections are created and ready to receive requests.
Then, add it to the pod deployment yaml as an initContainers:
https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
initContainers:
- name: my-connection-validator
image: path-to-your-image
env:
- name: POD_HOST
value: "localhost-or-ip"
- name: POD_PORT
value: "12345"
I deploy a Elasticsearch cluster to EKS, below is the spec
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elk
spec:
version: 7.15.2
serviceAccountName: docker-sa
http:
tls:
selfSignedCertificate:
disabled: true
nodeSets:
- name: node
count: 3
config:
...
I can see it has been deployed correctly and all pods are running.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
elk-es-node-0 1/1 Running 0 19h
elk-es-node-1 1/1 Running 0 19h
elk-es-node-2 1/1 Running 0 11h
But I can't restart the deployment Elasticsearch,
$ kubectl rollout restart Elasticsearch elk-es-node
Error from server (NotFound): elasticsearches.elasticsearch.k8s.elastic.co "elk-es-node" not found
The Elasticsearch is using statefulset so I tried to restart statefulset,
$ kubectl rollout restart statefulset elk-es-node
statefulset.apps/elk-es-node restarted
the above command says restarted, but the actual pods are not restarting.
what is the right way to restart a custom kind in K8S?
Use - kubectl get all
To identify if the resource created is a deployment or a statefulset -
use -n <namespace"> along with the above command, if you are working in a specific namespace.
Assuming, you are using a statefulset, the issue below command to understand the properties in which it is configured.
kubectl get statefulset <statefulset-name"> -o yaml > statefulsetContent.yaml
this will create a yaml file names statefulsetContent.yaml in same directory.
you can use it to explore different options configured in the statefulset.
Check for .spec.updateStrategy in the yaml file. Based on this we can identify its update strategy.
Below is from the official documentation
There are two possible values:
OnDelete
When a StatefulSet's .spec.updateStrategy.type is set to OnDelete, the StatefulSet controller will not automatically update the Pods in a StatefulSet. Users must manually delete Pods to cause the controller to create new Pods that reflect modifications made to a StatefulSet's .spec.template.
RollingUpdate
The RollingUpdate update strategy implements automated, rolling update for the Pods in a StatefulSet. This is the default update strategy.
As a work around, you can try to scale down/up the statefulset.
kubectl scale sts <statefulset-name"> --replicas=<count">
With ECK as the operator, you do not need to use rollout restart. Apply your updated Elasticsearch spec and the operator will perform rolling update for you. If for any reason you need to restart a pod, you use kubectl delete pod <es pod> -n <your es namespace> to remove the pod and the operator will spin up new one for you.
Hello I'm starting with docker and docker compose and I have the following problem:
I'm working in a spring micro services architecture where I have one configuration service, one discovery service, one gateway service and multiple resource services.
To run these services, I build jar files, which I place in separated folder per service with their config files (application.yml and bootstrap.yml):
e.g:
config-service/
config-service.jar
application.yml
discovery-service/
discovery-service.jar
bootstrap.yml
gateway-service/
gateway-service.jar
bootstrap.yml
crm-service/
crm-service.jar
bootstrap.yml
This works so far on my server.
Now I want to deploy my services in different environments as docker images (created with mvn build image and buildpack) using docker compose, where the configuration files vary depending on the environment. How can I deploy a service as a container using an existing image but with a different configuration file?
Thank you in advance!
There are a few possibilities when handling configuration in a containerized environment.
One of the options is that Spring boot allows you to use environment variables for each application property. For example, let's say you have a spring.datasource.url property, in that case you could also define that property by setting a SPRING_DATASOURCE_URL environment variable:
version: '3.8'
services:
my-spring-boot-app:
image: my-image:0.0.1
environment:
- SPRING_DATASOURCE_URL=jdbc:my-database-url
Alternatively, you could use volumes to put an external file on a specific location within a container:
version: '3.8'
services:
my-spring-boot-app:
image: my-image:0.0.1
volumes:
./my-app/bootstrap.yml:/etc/my-app/bootstrap.yml
In this example I'm copyingbootstrap.yml from a relative folder on my host machine, to /etc/my-app within the container. If you put these files within the same folder as your JAR file, you can override the configuration.
I was looking at step-by-step tutorial on how to run my spring boot, mysql-backed app using AWS EKS (Elastic Container service for Kubernetes) using the existing SSL wildcard certificate and wasn't able to find a complete solution.
The app is a standard Spring boot self-contained application backed by MySQL database, running on port 8080. I need to run it with high availability, high redundancy including MySQL db that needs to handle large number of writes as well as reads.
I decided to go with the EKS-hosted cluster, saving a custom Docker image to AWS-own ECR private Docker repo going against EKS-hosted MySQL cluster. And using AWS issued SSL certificate to communicate over HTTPS. Below is my solution but I'll be very curious to see how it can be done differently
This a step-by-step tutorial. Please don't proceed forward until the previous step is complete.
CREATE EKS CLUSTER
Follow the standard tutorial to create EKS cluster. Don't do step 4. When you done you should have a working EKS cluster and you must be able to use kubectl utility to communicate with the cluster. When executed from the command line you should see the working nodes and other cluster elements using
kubectl get all --all-namespaces command
INSTALL MYSQL CLUSTER
I used helm to install MySQL cluster following steps from this tutorial. Here are the steps
Install helm
Since I'm using Macbook Pro with homebrew I used brew install kubernetes-helm command
Deploy MySQL cluster
Note that in MySQL cluster and Kubernetes (EKS) cluster, word "cluster" refers to 2 different things. Basically you are installing cluster into cluster, just like a Russian Matryoshka doll so your MySQL cluster ends up running on EKS cluster nodes.
I used a 2nd part of this tutorial (ignore kops part) to prepare the helm chart and install MySQL cluster. Quoting helm configuration:
$ kubectl create serviceaccount -n kube-system tiller
serviceaccount "tiller" created
$ kubectl create clusterrolebinding tiller-crule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io "tiller-crule" created
$ helm init --service-account tiller --wait
$HELM_HOME has been configured at /home/presslabs/.helm.
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
$ helm repo add presslabs https://presslabs.github.io/charts
"presslabs" has been added to your repositories
$ helm install presslabs/mysql-operator --name mysql-operator
NAME: mysql-operator
LAST DEPLOYED: Tue Aug 14 15:50:42 2018
NAMESPACE: default
STATUS: DEPLOYED
I run all commands exactly as quoted above.
Before creating a cluster, you need a secret that contains the ROOT_PASSWORD key.
Create a file named example-cluster-secret.yaml and copy into it the following YAML code
apiVersion: v1
kind: Secret
metadata:
name: my-secret
type: Opaque
data:
# root password is required to be specified
ROOT_PASSWORD: Zm9vYmFy
But what is that ROOT_PASSWORD? Turns out this is base64 encoded password that you planning to use with your MySQL root user. Say you want root/foobar (please don't actually use foobar). The easiest way to encode the password is to use one of the websites such as https://www.base64encode.org/ which encodes foobar into Zm9vYmFy
When ready execute kubectl apply -f example-cluster-secret.yaml which will create a new secret
Then you need to create a file named example-cluster.yaml and copy into it the following YAML code:
apiVersion: mysql.presslabs.org/v1alpha1
kind: MysqlCluster
metadata:
name: my-cluster
spec:
replicas: 2
secretName: my-secret
Note how the secretName matches the secret name you just created. You can change it to something more meaningful as long as it matches in both files. Now run kubectl apply -f example-cluster.yaml to finally create a MySQL cluster. Test it with
$ kubectl get mysql
NAME AGE
my-cluster 1m
Note that I did not configure a backup as described in the rest of the article. You don't need to do it for the database to operate. But how to access your db? At this point the mysql service is there but it doesn't have external IP. In my case I don't even want that as long as my app that will run on the same EKS cluster can access it.
However you can use kubectl port forwarding to access the db from your dev box that runs kubectl. Type in this command: kubectl port-forward services/my-cluster-mysql 8806:3306. Now you can access your db from 127.0.0.1:8806 using user root and the non-encoded password (foobar). Type this into separate command prompt: mysql -u root -h 127.0.0.1 -P 8806 -p. With this you can also use MySQL Workbench to manage your database just don't forget to run port-forward. And of course you can change 8806 to other port of your choosing
PACKAGE YOUR APP AS A DOCKER IMAGE AND DEPLOY
To deploy your Spring boot app into EKS cluster you need to package it into a Docker image and deploy it into the Docker repo. Let's start with a Docker image. There are plenty tutorials on this like this one but the steps are simple:
Put your generated, self-contained, spring boot jar file into a directory and create a text file with this exact name: Dockerfile in the same directory and add the following content to it:
FROM openjdk:8-jdk-alpine
MAINTAINER me#mydomain.com
LABEL name="My Awesome Docker Image"
# Add spring boot jar
VOLUME /tmp
ADD myapp-0.1.8.jar app.jar
EXPOSE 8080
# Database settings (maybe different in your app)
ENV RDS_USERNAME="my_user"
ENV RDS_PASSWORD="foobar"
# Other options
ENV JAVA_OPTS="-Dverknow.pypath=/"
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar" ]
Now simply run a Docker command from the same folder to create an image. Of course that requires Docker client installed on your dev box.
$ docker build -t myapp:0.1.8 --force-rm=true --no-cache=true .
If all goes well you should see your image listed with docker ps command
Deploy to the private ECR repo
Deploying your new image to ECR repo is easy and ECR works with EKS right out of the box. Log into AWS console and navigate to the ECR section. I found it confusing that apparently you need to have one repository per image but when you click "Create repository" button put your image name (e.g. myapp) into the text field. Now you need to copy the ugly URL for your image and go back to the command prompt
Tag and push your image. I'm using a fake URL as example: 901237695701.dkr.ecr.us-west-2.amazonaws.com you need to copy your own from the previous step
$ docker tag myapp:0.1.8 901237695701.dkr.ecr.us-west-2.amazonaws.com/myapp:latest
$ docker push 901237695701.dkr.ecr.us-west-2.amazonaws.com/myapp:latest
At this point the image should show up at ECR repository you created
Deploy your app to EKS cluster
Now you need to create a Kubernetes deployment for your app's Docker image. Create a myapp-deployment.yaml file with the following content
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: myapp-deployment
spec:
selector:
matchLabels:
app: myapp
replicas: 2
template:
metadata:
labels:
app: myapp
spec:
containers:
- image: 901237695701.dkr.ecr.us-west-2.amazonaws.com/myapp:latest
name: myapp
ports:
- containerPort: 8080
name: server
env:
# optional
- name: RDS_HOSTNAME
value: "10.100.98.196"
- name: RDS_PORT
value: "3306"
- name: RDS_DB_NAME
value: "mydb"
restartPolicy: Always
status: {}
Note how I'm using a full URL for the image parameter. I'm also using a private CLUSTER-IP of mysql cluster that you can get with kubectl get svc my-cluster-mysql command. This will differ for your app including any env names but you do have to provide this info to your app somehow. Then in your app you can set something like this in the application.properties file:
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://${RDS_HOSTNAME}:${RDS_PORT}/${RDS_DB_NAME}?autoReconnect=true&zeroDateTimeBehavior=convertToNull
spring.datasource.username=${RDS_USERNAME}
spring.datasource.password=${RDS_PASSWORD}
Once you save the myapp-deployment.yaml you need to run this command
kubectl apply -f myapp-deployment.yaml
Which will deploy your app into EKS cluster. This will create 2 pods in the cluster that you can see with kubectl get pods command
And rather than try to access one of the pods directly we can create a service to front the app pods. Create a myapp-service.yaml with this content:
apiVersion: v1
kind: Service
metadata:
name: myapp-service
spec:
ports:
- port: 443
targetPort: 8080
protocol: TCP
name: http
selector:
app: myapp
type: LoadBalancer
That's where the magic happens! Just by setting the port to 443 and type to LoadBalancer the system will create a Classic Load Balancer to front your app.
BTW if you don't need to run your app over HTTPS you can set port to 80 and you will be pretty much done!
After you run kubectl apply -f myapp-service.yaml the service in the cluster will be created and if you go to to the Load Balancers section in the EC2 section of AWS console you will see that a new balancer is created for you. You can also run kubectl get svc myapp-service command which will give you EXTERNAL-IP value, something like bl3a3e072346011e98cac0a1468f945b-8158249.us-west-2.elb.amazonaws.com. Copy that because we need to use it next.
It is worth to mention that if you are using port 80 then simply pasting that URL into the browser should display your app
Access your app over HTTPS
The following section assumes that you have AWS-issued SSL certificate. If you don't then go to AWS console "Certificate Manager" and create a wildcard certificate for your domain
Before your load balancer can work you need to access AWS console -> EC2 -> Load Balancers -> My new balancer -> Listeners and click on "Change" link in SSL Certificate column. Then in the pop up select the AWS-issued SSL certificate and save.
Go to Route-53 section in AWS console and select a hosted zone for your domain, say myapp.com.. Then click "Create Record Set" and create a CNAME - Canonical name record with Name set to whatever alias you want, say cluster.myapp.com and Value set to the EXTERNAL-IP from above. After you "Save Record Set" go to your browser and type in https://cluster.myapp.com. You should see your app running
I am trying to get some custom application metrics captured in golang using the prometheus client library to show up in Prometheus.
I have the following working:
I have a go application which is exposing metrics on localhost:8080/metrics as described in this article:
https://godoc.org/github.com/prometheus/client_golang/prometheus
I have a kubernates minikube running which has Prometheus, Grafana and AlertManager running using the operator from this article:
https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus
I created a docker image for my go app, when I run it and go to localhost:8080/metrics I can see the prometheus metrics showing up in a browser.
I use the following pod.yaml to deploy my docker image to a pod in k8s
apiVersion: v1
kind: Pod
metadata:
name: my-app-pod
labels:
zone: prod
version: v1
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '8080'
spec:
containers:
- name: my-container
image: name/my-app:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
If I connect to my pod using:
kubectl exec -it my-app-pod -- /bin/bash
then do wget on "localhost:8080/metrics", I can see my metrics
So far so good, here is where I am hitting a wall. I could have multiple pods running this same image. I want to expose all the images to prometheus as targets. How do I configure my pods so that they show up in prometheus so I can report on my custom metrics?
Thanks for any help offered!
The kubernetes_sd_config directive can be used to discover all pods with a given tag. Your Prometheus.yml config file should have something like so:
- job_name: 'some-app'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
regex: python-app
action: keep
The source label [__meta_kubernetes_pod_label_app] is basically using the Kubernetes api to look at pods that have a label of 'app' and whose value is captured by the regex expression, given on the line below (in this case, matching 'python-app').
Once you've done this Prometheus will automatically discover the pods you want and start scraping the metrics from your app.
Hope that helps. You can follow blog post here for more detail.
Note: it is worth mentioning that at the time of writing, kubernetes_sd_config is still in beta. Thus breaking changes to configuration may occur in future releases.
You need 2 things:
a ServiceMonitor for the Prometheus Operator, which specifies which services will be scraped for metrics
a Service which matches the ServiceMonitor and points to your pods
There is an example in the docs over here: https://coreos.com/operators/prometheus/docs/latest/user-guides/running-exporters.html
Can you share the prometheus config that you are using to scrape the metrics. The config will control what all sources to scrape the metrics from. Here are a few links that you can refer to : https://groups.google.com/forum/#!searchin/prometheus-users/Application$20metrics$20monitoring$20of$20Kubernetes$20Pods%7Csort:relevance/prometheus-users/uNPl4nJX9yk/cSKEBqJlBwAJ