We currently use TeamCity to build a deployment artifact, then a further TeamCity task takes that artifact and deploys it to our development and testing servers on demand.
We can store the passwords and other secret data in properties files that we can check into source control, as these are all internal servers and the developers have full access to them.
However for release to Production (and our final test layer) there are secret passwords and configuration that we don't want checked into the normal source control, or to have development be able to discover the passwords. So to do 'real' deployments we have to hand the artifact over to another team and they maintain a properties file with the production values.
What methods exist to store these secrets and allow TeamCity to run a deploy without ever leaking the secrets out?
(note I am one of the devs and it is not a trust issue... I don't want to have the ability to find out prod passwords so I can never accidently know them and do some horrific damage!)
Probably what you need here, is to create a separate project with narrower scope of permissions (for example, allow only certain people to edit build configurations). In this project create a build configuration, responsible for deployment. In this configuration, you can define a Typed Parameter of type 'password' to store the password to the production environment.
Another option is to use Deployer Plugin, especially its ability to deploy over ssh with private key authentication
If you are OK to use a third party solution, consider using a solution like CloudMunch which can help you to perform release management functions with these secure parameters collected at deploy time and encrypted post deployment.
Disclaimer: I work with CloudMunch
You can do 2 things.
Use a teamcity project to deploy artefacts for production only. This will only be accessible to ops members.
Teamcity also supports running agents with different user ids. You can create a new user id which can have access to the production "secrets" (passwords and configuration). Use this id to run the targets in the 1st step.
Related
I have access to a private Nexus Repository and would like to speed up my CI builds and thought that I could use the private repository to store and access my build cache. Is this a possibility or a dead end?
It works like a breeze.
Just create a "Raw" repository and give a user write permission for it.
This user then is used to fill the cache and you can use another user or anonymous access to read from the cache.
I just tried it minutes ago.
Any web server that supports PUT for storing files and GET for retrieving the same files should be fine with the default HttpBuildCache implementation.
You can even provide an own client-side implementation to use any remote service you want as build cache.
No.
Gradle's remote build cache is one of the selling points of Gradle Enterprise. So it's not something you can just "plugin" to another piece of software like Nexus.
There is however a Docker image that is designed to work with Gradle Enterprise. Maybe you could make use of that somehow.
But again, the remote build cache is a selling point of Gradle enterprise and as a result is designed to work with Gradle enterprise.
https://gradle.com/build-cache/
I am planning to build an enterprise application using aws lambda and serverless framework.
I want to separate the dev, test and prod environments and I am planning to use AWS Parameter store for it.
I don't want my production environment configuration be exposed to developers. If the developer runs the command serverless offline -s production start then the production configuration should not be obtained.
It should be obtained only when the serverless function has been successfully deployed to aws lambda.
Here are few considerations based on your question:
To have different environments on Serverless framework you have to set up the stage. This value can be passed as a parameter when executing sls commands.
If you are keeping your code in a repo, the developers will have access to all the configurations. If this is really important, you could keep the production configuration in a diff repo where only very specific people will have access to it, and then you make a reference to in in your serverless.yml. Ex:
custom: ${file(./config/${opt:stage, 'dev'}.json)} and then in your config folder you create the prod.json file, but pointing to the real one of the new repo you created. Note: this would make your project harder to maintain.
Considering you don't want your developers to execute your production environment locally. You can use the global variable of serverless offline to block the execution. You could also inform then to not do so.
Here is what should be a good practice and solution based on your problem:
Considering you have a production environment you want to isolate from a given group in your company, you should create VPC's and configure their resources access, accordingly.
Then you create users to have diff access. When your developer try to execute the code accessing a resource (dynamoDB for example) in a VPC they don't have access, they will be blocked.
AWS configure to define which user will execute the SLS command.
Your development team will still have access to your configuration file.
Note: In this case the person/group with access to the production VPC will have to do the deploy.
If the answer does not suffice, could you please reinforce which type of resource(s) are sensitive across your Serverless project? I am taking for granted it is the DB as it is the most common scenario.
I do have a requirement in which I need to deploy my database into different environments with application-specific data on the target environment.
For an instance, we are integrating with a 3rd party services and they do have different credentials and other parameters specific to the target environment. Such as credential and endpoints and other parameters for test and prod environments.
Currently, we are using post-deploy DB scripts targetting those environments and stored them on our git repository.
I read and heard we should not commit such sensitive information in repositories due to the risk of hacking and stealing.
Can someone suggest a better way (best practices) of handling this requirement in your CI/CD pipeline?
We are using TeamCity and Octopus tools in our CI/CD process.
Appreciate your valuable advice and feedback in this regard.
Also feel free to share the best CI/CD practices to overcome such concerns in your development process.
Thanks,
RSF
I would have approached it by keeping data like usr/psw as a sensitive variables per environment in Octopus.
To deploy I'd use sqlpackage.exe passing in the dacpac and the variables (/v: variable1=value1).
I'm used to having a single entity checkout, build, test, and deploy code, on every commit change (whether it be for a staging server or a production server). Now that we have started looking into Ansible, I'm beginning to think that there are isolated roles with these tools.
Basically I'm asking is it Ansible's responsibility to handle compiling and testing the code before deployment, or should it grab artifacts from a CI server such as Bamboo and trust that artifact is ready for deployment?
I'm not sure about the idea of using ansible to do the compiling, I rather just do that inside of CI as they have facilities done just for that. As for testing it depends on type of tests - if those are unit tests then they should be ran right after build (preferably inside of CI again) and either fail or pass a build.
But if those tests are of integration/functional nature (where they verify whether service actually works in the environment as we expect) then they for sure should be a part of post_tasks of the playbook, and if they don't pass you should mark the deployment as failed and act accordingly. This of course gives an idea of having a safe way to do that, before the service is exposed to production traffic, so if the tests do not pass, you can safely unroll the thing.
Nope, Ansible's responsibility is not to handle compiling and testing the code before deployment.
Yes it should grab artifacts from a CI server such as Bamboo and trust that artifact is ready for deployment.
Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.
https://www.ansible.com/how-ansible-works
I need to configure our Nexus repo to have multiple procured groups and have different Jenkins jobs reference different groups depending on the type of job.
For example, for CI jobs I need to be able to proxy a bunch of repositories. For deploy or release jobs I need to use a proxy group that is procured and controlled. The CI may allow the users to get to artifacts that aren't yet approved but for releases the artifacts are vetted and approved first.
It seems simple to create the groups and do procurement in Nexus. However, I can't figure out how to get the Jenkins jobs to use the different groups. I can only have a single mirrorOf tag in the maven .settings.xml and mirrors can't be set in profiles.
Has anyone tried this?
You have to have a different settings.xml file for the different jobs that access different groups. The Maven 3 integration of Hudson can do that within the server and automatically use the right one. For Jenkins you have to manually manage it a different way. E.g. by having different settings.xml file with different names on the CI server and then using the -s options of the mvn invocation.
"The CI may allow the users to get to artifacts that aren't yet approved but for releases the artifacts are vetted and approved first."
I wonder what are you using the CI for. Normally The CI is simply used to build continuously your projects but not used to be arbitrary between your developers and your repos manager server.
Maybe you could configure your proxies in multiple profile in your setting.xml and activate them as needed using properties/environment variables.