I have been trying to spin up a Kubernetes/Fabric8 installation on AWS using Stackpoint as described in this video: https://www.youtube.com/watch?v=lNRpGJTSMKA
My problem is that three of the apps wont start becuase no volumes are available and I cannot see how to resolve those PV requests. For example Gogs is reporting the following error:
Unable to mount volumes for pod "gogs-2568819805-bcw8e_default(03d618b9-7477-11e6-8c6b-0a945216fb91)": timeout expired waiting for volumes to attach/mount for pod "gogs-2568819805-bcw8e"/"default". list of unattached/unmounted volumes=[gogs-data]
Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "gogs-2568819805-bcw8e"/"default". list of unattached/unmounted volumes=[gogs-data]
I am pretty sure this is very simple but cannot see how to connect the dots here from the various K8, Fabric8 docs. I can create a new EBS volume in AWS easily enough but cannot see how to then update this running stack to attach it to these services. Any help would be greatly appreciated!
Sorry about that, what version of gofabric8 are you using? We're currently adding persistent volume support for the core platform apps although the integration our stackpoint isn't there quite yet. Hopefully soon though.
For now you should be able to disable the PV claims using --pv=false during the deploy. So gofabric8 deploy --pv=false. We'll look at using this as the default until the integration is there and we can leverage AWS persistent volumes
We just shipped functionality that allows you to create and manage AWS volumes for Kubernetes. You get a volume, PV, and claim - just name the claim to be what is required by Fabric8. Eventually, you'll be able to use dynamic volume creation.
Related
I'm trying to run only the Elastic agent as a deployment in a Kubernetes cluster. The reason I'm doing this is maybe an atypical usage of the Elastic agent: I only want to deploy the HTTP log endpoint integration and have other pods send logs to this Elastic agent. I'm not using it to collect cluster metrics (so the manifest they supply is not relevant to me).
I'm using the image docker.elastic.co/beats/elastic-agent:8.4.2. Apparently, this image needs to write files and directories to /usr/share/elastic-agent/, which at first was leading to errors along the lines of failed: mkdir /usr/share/elastic-agent/state: read-only file system. So, I created an emptyDir volume and mounted it at /usr/share/elastic-agent. Now, that error disappears, but is replaced with a new error:
/usr/local/bin/docker-entrypoint: line 14: exec: elastic-agent: not found
The entrypoint of the image is
ENTRYPOINT ["/usr/bin/tini" "--" "/usr/local/bin/docker-entrypoint"]
and it is apparently unable to find /usr/local/bin/docker-entrypoint.
A couple questions:
Why is it not finding the elastic-agent executable? It is definitely at that path.
More broadly: I am new to Elasticsearch -- this is only to set up a QA environment meant to test a product feature where we forward data from certain of our services to customers' Elastic Cloud deployments. I thought deploying the agent as a service in the same cluster where these services run would be the least painful way to do this. Is this not a good way to achieve what I describe in the first paragraph?
Assuming I can get the deployment to actually work, is this the way the next steps would go?
Create the "Custom HTTP Endpoint Logs" integration on the agent policy, listening on a given port and on all interfaces.
Map that port to an external port for the pod.
Send data to the pod at that external port.
The issue is that mounting the emptyDir volume to /usr/share overwrites the elastic-agent binary. Remove this volume and set readOnlyRootFilesystem: false.
I have a Spring Boot application that exposes multiple APIs and uses swagger for documentation. This service is then deployed to AKS using Helm through Azure DevOps.
When running locally, the swagger documentation looks updated but however, when I deploy it; the documentation goes back to the outdated version. I'm not really sure what is happening during deployment and I am unable to find any help on the forums.
As far as I know; I do not think there is any sort of caching taking place but again I'm not sure.
It sounds like you suspect an incorrect version of your application is running in the cluster following a build and deployment.
Assuming things like local browser caching have been eliminated from the equation, review the state of deployments and/or pods in your cluster using CLI tools.
Run kubectl describe deployment <deployment-name>, the pod template will be displayed which defines which image tag the pods should use. This should correlate with the tag your AzDO pipeline is publishing.
List the pods and describe them to see if the expected image tag is what is running in the cluster after a deployment. If not, check the pods for failures - when describing the pod, pay attention to the lastState object if it exists. Use kubectl logs <podname> to troubleshoot in the application layer.
It can take a few minutes for the new pods to become available depending on configuration.
I just ported my application to OpenShift Online 3 (from version 2), and now I'm struggling to understand how to manage persistent, "shared" data, that is not wiped after each build.
After reading the documentation about Persistent Volume Claims, I created a new PVC inside my project, of type RWO, using the Web dashboard. At this point I tried to understand how to access this storage from inside each pod, or if I needed to do something to mount it, and I ended up doing this:
$ oc volume dc/myapp --add --type=persistentVolumeClaim --claim-name=pvcname --mount-path=/usr/share/data
After this, it looks like the new configuration was successfully registered:
$ oc volume dc --all
deploymentconfigs/myapp
pvc/pvcname (allocated 1GiB) as volume-jh1jf
mounted at /usr/share/data
I could also see the new /usr/share/data directory from inside the pods created by the new builds.
However, after making this change, all deployments started failing with this error:
Failed to attach volume "pvc-0b747c80-a687-11e7-9eb0-122631632f42" on node "ip-172-31-48-134.ec2.internal" with: Error attaching EBS volume "vol-0008c8127ff0f4617" to instance "i-00195cc4e1d31f8ce": VolumeInUse: vol-0008c8127ff0f4617 is already attached to an instance status code: 400, request id: 722f3797-f486-4739-ab4e-fe1826ae53af. The volume is currently attached to instance "i-089e2a60e525f447c"
from which it looks like my latest change had the effect of attaching the volume to a specific instance. But then how can I mount the volume to my pods so that it survives each build and deploy?
Because you are using an EBS volume type, you must set the deployment strategy on the deployment config to Recreate instead of Rolling. This is because an EBS volume can only be mounted on a single node in the cluster at a time. This means you cannot using a rolling deployment, nor scale your application above 1 replica, as both result in more than one instance and there is no guarantee they will be deployed to the same node.
I successfully deployed Kubernetes on AWS using "getting started on AWS ec2 guide" (http://kubernetes.io/v1.0/docs/getting-started-guides/aws.html), but the disk size of all the minions (kubernetes hosts) is 8gb. I would like to increase the disk size, but I haven't found a way to do it.
I can change the VM size by setting MINION_SIZE (e.g. export MINION_SIZE=m3.medium) prior to installing, but the disk size is still 8gb.
From the Kubernetes install instructions for other cloud providers there's an option to set MINION_DISK_SIZE to set the disk size. I tried that with AWS ec2 installation, and the variable is ignored.
I also poked around the config files, but I didn't see anything obvious.
Any suggestions on how to set the disk size for minions when installing Kubernetes on AWS ec2?
I recently stumbled upon the same issue. Have a look at BLOCK_DEVICE_MAPPINGS in kubernetes/cluster/aws/util.sh. You can modify it to have something more appropriate for a EBS-only minion.
For example:
[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":80}}]
AWS docs: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html
I have faced this very issue and tried the current accepted answer but it looks like Kubernetes is changing quite fast what may make this answer also outdated soon.
To this date, I've tested the solution below that might become or not a definitive solution in the future:
There is this PR on Kubernetes' github project that implements an easy way to ignore the SSD storage by setting KUBE_AWS_STORAGE=ebs before running kubernetes/cluster/kube-up.sh.
Hope it is helpful!
My organization's website is a Django app running on front end webservers + a few background processing servers in AWS.
We're currently using Ansible for both :
system configuration (from a bare OS image)
frequent manually-triggered code deployments.
The same Ansible playbook is able to provision either a local Vagrant dev VM, or a production EC2 instance from scratch.
We now want to implement autoscaling in EC2, and that requires some changes towards a "treat servers as cattle, not pets" philosophy.
The first prerequisite was to move from a statically managed Ansible inventory to a dynamic, EC2 API-based one, done.
The next big question is how to deploy in this new world where throwaway instances come up & down in the middle of the night. The options I can think of are :
Bake a new fully-deployed AMI for each deploy, create a new AS Launch config and update the AS group with that. Sounds very, very cumbersome, but also very reliable because of the clean slate approach, and will ensure that any system changes the code requires will be here. Also, no additional steps needed on instance bootup, so up & running more quickly.
Use a base AMI that doesn't change very often, automatically get the latest app code from git upon bootup, start webserver. Once it's up just do manual deploys as needed, like before. But what if the new code depends on a change in the system config (new package, permissions, etc) ? Looks like you have to start taking care of dependencies between code versions and system/AMI versions, whereas the "just do a full ansible run" approach was more integrated and more reliable. Is it more than just a potential headache in practice ?
Use Docker ? I have a strong hunch it can be useful, but I'm not sure yet how it would fit our picture. We're a relatively self-contained Django front-end app with just RabbitMQ + memcache as services, which we're never going to run on the same host anyway. So what benefits are there in building a Docker image using Ansible that contains system packages + latest code, rather than having Ansible just do it directly on an EC2 instance ?
How do you do it ? Any insights / best practices ?
Thanks !
This question is very opinion based. But just to give you my take, I would just go with prebaking the AMIs with Ansible and then use CloudFormation to deploy your stacks with Autoscaling, Monitoring and your pre-baked AMIs. The advantage of this is that if you have most of the application stack pre-baked into the AMI autoscaling UP will happen faster.
Docker is another approach but in my opinion it adds an extra layer in your application that you may not need if you are already using EC2. Docker can be really useful if you say want to containerize in a single server. Maybe you have some extra capacity in a server and Docker will allow you to run that extra application on the same server without interfering with existing ones.
Having said that some people find Docker useful not in the sort of way to optimize the resources in a single server but rather in a sort of way that it allows you to pre-bake your applications in containers. So when you do deploy a new version or new code all you have to do is copy/replicate these docker containers across your servers, then stop the old container versions and start the new container versions.
My two cents.
A hybrid solution may give you the desired result. Store the head docker image in S3, prebake the AMI with a simple fetch and run script on start (or pass it into a stock AMI with user-data). Version control by moving the head image to your latest stable version, you could probably also implement test stacks of new versions by making the fetch script smart enough to identify which docker version to fetch based on instance tags which are configurable at instance launch.
You can also use AWS CodeDeploy with AutoScaling and your build server. We use CodeDeploy plugin for Jenkins.
This setup allows you to:
perform your build in Jenkins
upload to S3 bucket
deploy to all the EC2s one by one which are part of the assigned AWS Auto-Scaling group.
All that with a push of a button!
Here is the AWS tutorial: Deploy an Application to an Auto Scaling Group Using AWS CodeDeploy