I have a Spring Boot application that exposes multiple APIs and uses swagger for documentation. This service is then deployed to AKS using Helm through Azure DevOps.
When running locally, the swagger documentation looks updated but however, when I deploy it; the documentation goes back to the outdated version. I'm not really sure what is happening during deployment and I am unable to find any help on the forums.
As far as I know; I do not think there is any sort of caching taking place but again I'm not sure.
It sounds like you suspect an incorrect version of your application is running in the cluster following a build and deployment.
Assuming things like local browser caching have been eliminated from the equation, review the state of deployments and/or pods in your cluster using CLI tools.
Run kubectl describe deployment <deployment-name>, the pod template will be displayed which defines which image tag the pods should use. This should correlate with the tag your AzDO pipeline is publishing.
List the pods and describe them to see if the expected image tag is what is running in the cluster after a deployment. If not, check the pods for failures - when describing the pod, pay attention to the lastState object if it exists. Use kubectl logs <podname> to troubleshoot in the application layer.
It can take a few minutes for the new pods to become available depending on configuration.
Related
Most software tech has a "Hello World" type example to get started on. With Kubernetes this seems to be lacking.
My scenario cannot be simpler. I have a simple hello world app made with Spring-Boot with one Rest controller that just returns: "Hello Hello!"
After I create my docker file, I build an image like this :
docker build -t helloworld:1.0 .
Then I run it in a container like this :
docker run -p 8080:8080 helloworld:1.0
If I open up a browser now, I can access my application here :
http://localhost:8080/hello/
and it returns :
"Hello Hello!"
Great! So far so good.
Next I tag it (my docker-hub is called ollyw123, and the ID of my image is 776...)
docker tag 7769f3792278 ollyw123/helloworld:firsttry
and push :
docker push ollyw123/helloworld
If I log into Docker-Hub I will see
Now I want to connect this to Kubernetes. This is where I have plunged deep into the a state of confusion.
My thinking is, I need to create a cluster. Somehow I need to connect this cluster to my image, and as I understand, I just need to use the URL of the image to connect to (ie.
https://hub.docker.com/repository/docker/ollyw123/helloworld)
Next I would have to create a service. This service would then be able to expose my "Hello World!" rest call through some port. This is my logical thinking, and for me this would seem like a very simple thing to do, but the tutorials and documentation on Kubernetes is a mine field of confusion and dead ends.
Following on from the spring-boot kubernetes tutorial (https://spring.io/guides/gs/spring-boot-kubernetes/) I have to create a deployment object, and then a service object, and then I have to "apply" it :
kubectl create deployment hello-world-dep --image=ollyw123/helloworld --dry-run -o=yaml > deployment.yaml
kubectl create service clusterip hello-world-dep --tcp=8080:8080 --dry-run -o=yaml >> deployment.yaml
kubectl apply -f deployment.yaml
OK. Now I see a service :
But now what???
How do I push this to the cloud? (eg. gcloud) Do I need to create a cluster first, or is this already a cluster?
What should my next step be?
There are a couple of concepts that we need to go through regarding your question.
The first would be about the "Hello World" app in Kubernetes. Even this existing (as mentioned by Limido in the comments [link]), the app itself is not a Kubernetes app, but an app created in the language of your choice, which was containerized and it is deployed in Kubernetes.
So I would call it (in your case) a Dockerized SpringBoot HelloWorld app.
Okay, now that we have a container we could simply deploy it running docker, but what if your container dies, or you need to scale it up and down, manage volumes, network traffic and a bunch of other things, this starts to become complicated (imagine a real life scenario, with hundreds or even thousands of containers running at the same time). That's exactly where the Container Orchestration comes into place.
Kubernetes helps you managing this complexity, in a single place.
The third concept that I'd like to talk, is the create and apply commands. You can definitely find a more detailed explanation in here, but both of then can be used to create the resource in Kubernetes.
In your case, the create command is not creating the resources, because you are using the --dry-run and adding the output to your deployment file, which you apply later on, but the following command would also create your resource:
kubectl create deployment hello-world-dep --image=ollyw123/helloworld
kubectl create service clusterip hello-world-dep --tcp=8080:8080
Note that even this working, if you need to share this deployment, or commit it in a repository you would need to get it:
kubectl get deployment hello-world-dep -o yaml > your-file.yaml
So having the definition file is really helpful and recommended.
Great... Going further...
When you have a deployment you will also have a number of replicas that is expected to be running (even when you don't define it - the default value is 1). In your case your deployment is managing one pod.
If you run:
kubectl get pods -o wide
You will get your pod hello-world-dep-hash and an IP address. This IP is the IP of your container and you can access your application using it, but as pods are ephemeral, if your pod dies, Kubernetes will create a new one for you (automatically) with a new IP address, so if you have for instance a backend and its IP is constantly changing, you would need to manage this change in the frontend every time you have a new backend pod.
To solve that, Kubernetes has the Service, which will expose the deployment in a persistent way. So if your pod dies and a new one comes back, the address of your service will continue the same, and all the traffic will be automatically routed to your new pod.
When you have more than one replica of your deployment, the service also load balance the load across all the available pods.
Last but not least, your question!
You have asked, now what?
So basically, once you have your application containerized, you can deploy it almost anywhere. There are N different places you can get it. In your case you are running it locally, but you could get your deployment.yaml file and deploy your application in GKE, AKS, EKS, just to quote the biggest ones, but all cloud providers have some type of Kubernetes service available, where you can spin up a cluster and start playing around.
Actually, to play around I'd recommend Katakoda, as they have scenarios for free, and you can use the cluster to play around.
Wow... That was a long answer...
Just to finish, I'd recommend the Network Introduction in Katakoda, as there are different types of Services, depending on your scenario or what you need, and the tutorial is goes through the different types in a hands-on approach.
In the context of Kubernetes, Cluster is the environment where your PODS and Services are running. Think of it like a VM environment where you setup your Web Server and etc.. (although I don't like my own analogy)
If you want to run the same thing in GCloud, then you create a Kubernetes cluster there and all you need to do is to apply your YAML files that contains the Service and Deployment there via the CLI that Google Cloud provides to interact with your Cluster.
In order to interact with GCloud GKS Cluster via your local command prompt, you need to get the credentials for that cluster. This official GCloud document explain how to retrieve your cluster credential. once done, you can start interacting with the Kubernetes instance running in GCloud via kubectl command using your command prompt.
The service that you have is of type clusterIP which is only accessible from within the kubernetes cluster. You need to either use NodePort or LoadBalanacer type service or ingress to expose the application outside the remote kubernetes cluster(a set of VMs or bare metal servers in public or private cloud environment with kubernetes deployed on them) or local minikube/docker desktop. Once you do that you should be able to access it using a browser or curl
We are using Gitlab's CI/CD Auto DevOps on gitlab.com with a connected Kubernetes cluster. Until recently, we ran on Azure, but have now decided to switch over to digitalocean. The build / deploy pipeline used to run fine on Azure, but when I run it now on our fresh cluster, I’m getting this error during the "production" job:
$ auto-deploy deploy
secret/production-secret replaced
Deploying new release...
Release "production" does not exist. Installing it now.
Error: validation failed: unable to recognize "": no matches for kind
"Deployment" in version "extensions/v1beta1"
ERROR: Job failed: exit code 1
After doing some googling, I found this release announcement for Kubernetes 1.16, which states that the Deployment resource has been moved up from extensions/v1beta1 to (eventually) apps/v1, and - more importantly - has been dropped from extensions/*:
https://kubernetes.io/blog/2019/09/18/kubernetes-1-16-release-announcement/
The Kubernetes version used on digitalocean is indeed 1.16.2. I do not recall the version we used on Azure, but judging from the article’s date, the 1.16 release is somewhat recent (September 2019).
As far as I can tell, the deployment algorithm is implemented inside Gitlab's "auto-deploy" image, specifically this script, but I fail to see where I can adapt the specific kubectl commands being executed.
My question is this: Am I right in assuming that this issue is caused by Gitlab’s CI/CD using a pre-1.16 notation to automatically create Deployments on Kubernetes clusters? If so, how can I adapt the deployment script to use the apps/v1 scope?
I was getting the same error while doing it with circleCI, then found that the kubectl version was 1.13 against the kubernetes version of 1.15.4 , so refer to this stackoverflow post and try placing the kubectl version in gitlab, same as the one your current kubernetes cluster shows.
you can do
kubectl version
to get the client and server versions, just match the gitlab kubectl with the updated one
Use 'apps/v1' for 'apiVersion', 'extensions/v1beta1' has deprecated Deployment for a few versions now.
As it turns out, Gitlab Auto DevOps indeed does not support Kubernetes 1.16 as of yet. They're working on it.
See also the issue I opened on gitlab.
I assume it's possible to fork the helm chart project and build your own version, but I'm not willing to go to those lengths.
As it turns out, Gitlab Auto DevOps indeed does not support Kubernetes 1.16 as of yet.
That might change with GitLab 13.5 (October 2020)
Incremental rollout via AutoDevOps is compatible with Kubernetes 1.16
Clusters in GKE were automatically upgraded to Kubernetes v1.16 on October 6th, 2020. We
updated Auto DevOps to support this version
for incremental rollouts to continue working as expected. This upgrade affects users who
continuously deploy to production using timed incremental rollout, and those who automatically
deploy to staging but manually deploy to production.
See Documentation and Issue.
And that comes with, still with GitLab 13.5:
Upgrade Auto DevOps to Helm 3
Auto DevOps aims to bring outstanding ease of use and security best practices to its users, out of the box. Until now, Auto DevOps in Kubernetes environments required Helm v2 to be installed on the cluster. This presented a security risk given Tiller’s root access rights. With the introduction of Helm v3, Tiller is no longer a requirement.
The current GitLab version finally supports Helm v3 so you can rest assured you’re getting the latest and great functionality and security updates. In the case of a GitLab Managed Cluster, you can upgrade your Helm installation by following our documentation. Note that Helm 2 support is expected to end around November 2020.
See Documentation and Issue.
As I mention here (May 2021), Helm V2 will be not supported starting GitLab 14.0 (June 2021).
I have created the hello world application from the SAP Cloud SDK archetypes and pushed this to the cloud foundry environment, binding it to an application logging service instance. My understanding is that this should already provide me with the ability to analyze all logs in the Kibana dashboard of the cloud platform and previously it also worked this way.
However, this time the Kibana dashboard remains empty, so I am wondering if I missed a step or configuration. Looking at the documentation of the service and the respective tutorial blog, I was not able to identify any additional required steps. In the Logs view on the SCP cockpit I can definitely see the entries, but they are not replicated to the ELK stack in the background.
Problem was not SDK related, but seems to have been an incident on the SCP - now works correctly without any changes.
I have been trying to spin up a Kubernetes/Fabric8 installation on AWS using Stackpoint as described in this video: https://www.youtube.com/watch?v=lNRpGJTSMKA
My problem is that three of the apps wont start becuase no volumes are available and I cannot see how to resolve those PV requests. For example Gogs is reporting the following error:
Unable to mount volumes for pod "gogs-2568819805-bcw8e_default(03d618b9-7477-11e6-8c6b-0a945216fb91)": timeout expired waiting for volumes to attach/mount for pod "gogs-2568819805-bcw8e"/"default". list of unattached/unmounted volumes=[gogs-data]
Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "gogs-2568819805-bcw8e"/"default". list of unattached/unmounted volumes=[gogs-data]
I am pretty sure this is very simple but cannot see how to connect the dots here from the various K8, Fabric8 docs. I can create a new EBS volume in AWS easily enough but cannot see how to then update this running stack to attach it to these services. Any help would be greatly appreciated!
Sorry about that, what version of gofabric8 are you using? We're currently adding persistent volume support for the core platform apps although the integration our stackpoint isn't there quite yet. Hopefully soon though.
For now you should be able to disable the PV claims using --pv=false during the deploy. So gofabric8 deploy --pv=false. We'll look at using this as the default until the integration is there and we can leverage AWS persistent volumes
We just shipped functionality that allows you to create and manage AWS volumes for Kubernetes. You get a volume, PV, and claim - just name the claim to be what is required by Fabric8. Eventually, you'll be able to use dynamic volume creation.
I have historically used a lot of manual chaining to get a CI pipeline in place for microservice development so am excited to try Fabric8 as it seems that it will make life a lot easier. Running into some early issues though.
I did manage to get Fabric8 running locally but want to get things running on AWS so I can present a more real world flow to stakeholders. Following the notes on this page Fabric8 on AWS I was able to get a 3 server cluster running using Stackpoint. But, I cannot connect to that cluster to be able to start administering the services. The page references this link (http://fabric8.default.replace.me.io) but it is not working for me. Tried hitting each of the AWS instances by public IP but that failed also. What would be my next steps here?
yeah the getting started guides don't really explain this in great deal. There's a similar issue on the fabric8 issue tracker that we've tried to help answer how to access the console
TL;DR using the AWS loadbalancer can add expense so we deploy an NGINX reverse proxy so you can set up a wildcard DNS. We use and recommend cloudflare for that as its free for this type of use and fast to setup.
We also created a blog to explain the different options how to access apps on kubernetes
Hope that helps!