We have had image pull issues at one time or another with all of our possible docker registries including Artifactory, AWS ECR, and GitLab. Even DockerHub occasionally has issues.
Is there a way in a Kubernetes deployment to specify that a pod can get an image from multiple different repositories so it can fall back if one is down?
If not, what other solutions are there to maintain stability? I've seen things like Harbor and Trow, but it seems like a heavy handed solution to a simple problem.
Is there a way in a Kubernetes deployment to specify that a pod can get an image from multiple different repositories so it can fall back if one is down?
Not really, not natively 😔. You could probably trick a K8s node to pull images from different image registries (one at a time) if you place them behind something like a TCP load balancer that directs traffic to multiple registries. But this might take a lot of testing and work.
If not, what other solutions are there to maintain stability? I've seen things like Harbor and Trow, but it seems like a heavy handed solution to a simple problem.
I'd say either Harbor, Quay, and Trow is the way to go if you want something more redundant.
Kubernetes has the ability to set ImagePullPolicy and you can set it for example to Never if you'd like to pre-pull all your critical images on all the K8s nodes. You can tie this to some automation to pre-pull your images across your clusters and nodes.
I've actually opened a K8s feature request to see 👀 if this idea gains traction.
Update:
If you're using containerd or cri-o (or even Docker has registry mirrors). You have the ability to configure mirror registries:
containerd.toml example
...
[plugins.cri.registry]
[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins.cri.registry.mirrors."local.insecure-registry.io"]
endpoint = ["http://localhost:32000"]
[plugins.cri.registry.mirrors."gcr.io"]
endpoint = ["https://gcr.io"]
[plugins.cri.registry.configs]
[plugins.cri.registry.configs.auths]
[plugins.cri.registry.configs.auths."https://gcr.io"]
auth = "xxxxx...."
...
cri-o.conf example
...
# registries is used to specify a comma separated list of registries to be used
# when pulling an unqualified image (e.g. fedora:rawhide).
registries = [
“registry.example.xyz”,
“registry.fedoraproject.org”
]
...
✌️
Related
We have build a few Microservices (MS) which have been deployed to our company's K8s clusters.
For current deployment, any one of our MSs will be built as a Docker image and they deployed manually using the following steps; and it works fine:
Create Configmap
Installing a Service.yaml
Installing a Deployment.yaml
Installing an Ingress.yaml
I'm now looking at Helm v3 to simplify and encapsulate these deployments. I've read a lot of the Helm v3 documentation, but I still haven't found the answer to some simple questions and I hope to get an answer here before absorbing the entire doc along with Go and SPRIG and then finding out it won't fit our needs.
Our Spring MS has 5 separate application.properties files that are specific to each of our 5 environments. These properties files are simple multi-line key=value format with some comments preceded by #.
# environment based values
key1=value1
key2=value2
Using helm create, I installed a chart called ./deploy in the root directory which auto-created ./templates and a values.yaml.
The problem is that I need to access the application.properties files outside of the Chart's ./deploy directory.
From helm, I'd like to reference these 2 files from within my configmap.yaml's Data: section.
./src/main/resource/dev/application.properties
./src/main/resources/logback.xml
And I want to keep these files in their current format, not rewrite them to JSON/YAML format.
Does Helm v3 allow this?
Putting this as answer as there's no enough space on the comments!
Check the 12 factor app link I shared above, in particular the section on configuration... The explanation there is not great but the idea is behind is to build one container and deploy that container in any environment without having to modify it plus to have the ability to change the configuration without the need to create a new release (the latter cannot be done if the config is baked in the container). This allows, for example, to change a DB connection pool size without a release (or any other config parameter). It's also good from a security point of view as you might not want the container running in your lower environments (dev/test/whatnot) having production configuration (passwords, api keys, etc). This approach is similar to the Continuous Delivery principle of build once, deploy anywhere.
I assume that when you run the app locally, you only need access to one set of configuration, so you can keep that in a separate file (e.g. application.dev.properties), and have the parameters that change between environments in helm environment variables. I know you mentioned you don't want to do this, but this is considered a good practice nowadays (might be considered otherwise in the future...).
I also think it's important to be pragmatic, if in your case you don't feel the need to have the configuration outside of the container, then don't do it, and probably using the suggestion I gave to change a command line parameter to pick the config file works well. At the same time, keep in mind the 12 factor-app approach in case you find out you do need it in the future.
According to Jelastic documentation it is possible to export the Environment configuration and download it so it can be restored in another provider
However I have tried with 2 Jelastic providers and they both have disabled the option for exporting private data.
So exporting/download/upload/import of environment is not possible.
i.e. I was expecting to have a process similar to CPanel backup/restore tool
In fact, another view for the deployment process gives a possibility to get rid of the model of handling the data or configuration on the platform. Try to think a bit differently and using CI/CD approach. The Jelastic provides a platform where something you created, locate on somewhere you're elaborating(VCS or GIT as an example) and based on or depends on the specific stack, already pre-configured like a layer and can be installed(copied) over Jelastic. Don't need to handle the data somewhere in the cloud because you have it locally(means within any VCS) and doing the changes there. Then just do a 'pull' procedure(manually or automatically) on that deployment(test, production, staging) environment you're expecting.
Moreover, you can expect any environments type like a code and perform it creating before deploying the data.
Please, find the articles being described each case:
Deployment Guide
Jelastic Packaging Standard for CI/CD Automation
In case you would like to handle the databases' backups, check this article:
Scheduling Database Backups
Additional FTP add-on can make the copies more easily for each instance:
FTP/FTPS Support in Jelastic
I am dealing an infrastructure and trying to figure it out how to deploy just single lambda from CI/CD pipeline.
Let's say in a repo you have 20 lambdas, and you made change for one single lambda, instead of deploying all of them i just want to deploy the changed one so cut out the deployment time.
I've got an idea like checking difference from git and figure it out which ones are changed, and do deployment only that part of functionality, but it surely doesn't seem right way to do it. Believing there is more proper way to do it.
I am using terraform for now (moving to serverless framework) i know that terraform and serverless framework holds a state on s3 bucket. However on my case when i run it through pipelines, eventhogh there is a terraform state and there is no change on the state, it still deploys the whole thing as far as realised (i might be wrong). I just want to get clear my mind to see how people does this with their pipline.
Since you seem to be asking about both Terraform and Serverless Framework here, I'm assuming you're looking for a general answer rather than specifically how this would be solved with a particular tool.
One way to solve this problem is to decouple your build process from your deploy process by adding a version selection mechanism in between. This just means that somewhere in your system you have a value that can be written by your build process and read by your deploy process which indicates what is the "current" artifact for each of your Lambda functions.
When your build process completes successfully, it can write the information about the artifact it built into the appropriate location, and then trigger your deployment process. Your deployment process will then read the artifact information and use it to decide what to deploy.
If you have made no changes to the current artifact metadata for a particular function then the deploy process can see that and not do anything. If a particular artifact is flawed in some way and you only notice once it's deployed, you can potentially set the artifact metadata back to the previous one and re-run the deployment process to roll back. If you choose a data store that retains historical versions, you'll also have a log of changes to the current artifact which might be useful to understand circumstances that lead to an incident.
Without getting into specifics it's hard to say more about this. For Terraform in particular, the artifact metadata store ought to be something that Terraform can read using a data source. To show a real example I'm going to just arbitrarily choose AWS SSM Parameter Store as a location for that artifact metadata store:
data "aws_ssm_parameter" "foo" {
name = "FooFunctionArtifact"
}
locals {
# For this example, we'll assume that the stored parameter is a JSON
# string shaped like this:
# {
# "s3_bucket": "awesomecorp-app-artifacts"
# "s3_key": "/awesomeapp/v1.2.0/function.zip"
# }
foo_artifact = jsondecode(data.aws_ssm_parameter.foo)
}
resource "aws_lambda_function" "foo" {
function_name = "foo"
s3_bucket = local.foo_artifact.s3_bucket
s3_key = local.foo_artifact.s3_key
# etc, etc
}
The technical details of this will vary a lot depending on your technology choices. If you don't use Terraform then you'll either use a feature similar to data sources in your other tool or you'd write some wrapper glue code that can itself retrieve the necessary information and pass it into the tool as an argument.
The main thing, regardless of technology choices, is that there is an explicit record somewhere of what is the latest artifact for each function, which is updated by your build step and read by your deploy step. This pattern can apply to other artifact types too, such as AMIs for EC2, docker images, etc.
Seems you have added label of terraform, serverless-framework (I called it sls), and aws-lambda. So all of them work for you.
terraform - Terraform itself will care of the differences which lambda need be updated. But it is not lambda friendly if you need install related packages.
serverless framework (sls) - it is good to use to manage lambda functions, but as side effect, it has to be managed with api gateway together. I am not sure if sls team has fix this issue or not. Need some confirmations.
SLS will take care of installing related packages.
The bad part is, sls can't diff the resources to be deployed and to be planned.
cloudformation - that's AWS owned Infrastructure as Code (IaC) tool to manage aws resources, you should be fine to use it to manage the lambda resource. you will get same issues as Terraform that you have to install the related packages before deploy the stack.
Bad part is, cfn (cloudformation) doesn't have diff feature as well, furtherly, it doesn't have proper tools to manage its aws cli commands, you have to use others, such as shell scriping, Ansible or even Terraform to manage coudformation templates updates.
aws cdk - The newest way is using aws-cdk, it does have the diff feature cdk diff which is mostly suitable for your current job, but it is very new project, a lot of features are still waiting to be developed.
You can take these and think with your team's skill sets. Always choice the tools, which you and your team are most confident.
I have the following
docker registry :http://myPrivateRegistry:5000
repository : myRepo
Image : myImage
I pushed this image to the remote repo by the following
docker push http://myPrivateRegistry:5000/myRepo/myImage
How do I delete this image from the 'remote repo' not just locally??
docker rmi http://myPrivateRegistry:5000/myRepo/myImage untags the image but does not remove it from teh remote repo
After some time googling I've found that you could use Curl command to delete images, e.g:
curl -X DELETE registry-url/v1/repositories/repository-name/
As far as I can see, this is still being debated in issue 422
While deletes are part of the API, they cannot be safely implemented on top of an eventually consistent backend (read: s3).
The main blocker comes from the complexity of reference counting on top of an eventually consistent storage system.
We need to consider whether it is worth facing that complexity or to adopt a hybrid storage model, where references are stored consistently.
As long as the registry supports varied backends, using an eventually consistent VFS model, safe deletions are not really possible without more infrastructure.
Issue 210 does mention
Soft delete have been implemented as part of the API, and more specialized issues have been opened for garbage collection.
https://github.com/docker/distribution/issues/422#issuecomment-114963170
I need to know if it is possible to keep multiple C5 servers in sync, while using local disks to contain the DocumentRoot for each instance. I cannot find any documentation on the subject of basic web clustering with C5.
Currently, we have a shared MySQL server, handling all DB services. (that we don't intend to change) We also use NFS to host the DocumentRoot repository, also being used by all of our hosts, to hold the data.
We want to break away from the NFS model, and use local drives on each web server instead. However, I don't know if C5 will have problems with this scenario, or what pitfalls are waiting for me.
I understand I will need some kind of mechanism to trigger the data propagation across local disks. That should be simple enough to accomplish. However C5 and its functionality may not like my plans, therefore I am asking for help.
How do "you" set up multiple C5 hosts, containing the same web sites, and keep them all in sync? Let me know!
Thanks!
You can use storage locations to store your files in a common location, and you can use database sessions to store your sessions in the common database.
Beyond that, all you need to do is make sure that you deploy any changes to file configuration in /application/config. Generally teams do that by ensuring that they don't make any configuration changes on the production site and instead deploy configuration changes from their staging environment.
If configuration becomes an issue, you can swap out the existing file configuration with a database based model pretty easily.