How can I access the variables I define in Google Secret Manager from my Google Cloud Build Pipeline ?
You can access to secret from Cloud Build by using the standard Cloud Builder gcloud
But, there is 2 issues:
If you want to use the secret value in another Cloud Build step, you have to store your secret in a file, the only way to reuse a previous value from one step to another one
The current Cloud Builder gcloud isn't up to date (today, 03 feb 2020). You have to add a gcloud component update for using the correct version. I opened an issue for this.
steps:
- name: gcr.io/cloud-builders/gcloud
entrypoint: "bash"
args:
- "-c"
- |
gcloud components update
# Store the secret is a temporary file
gcloud beta secrets versions access --secret=MySecretName latest > my-secret-file.txt
- name: AnotherCloudBuildStepImage
entrypoint: "bash"
args:
- "-c"
- |
# For getting the secret and pass it to a command/script
./my-script.sh $(cat my-secret-file.txt)
Think to grant the role Secret Manager Secret Accessor roles/secretmanager.secretAccessor to the Cloud Build default service account <PROJECT_ID>#cloudbuild.gserviceaccount.com
EDIT
You can access to the secret from anywhere, either with the gcloud CLI installed (and initialized with a service account authorized to access secrets) or via API call
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
https://secretmanager.googleapis.com/v1beta1/projects/PROJECT_ID/secrets/MySecretName/versions/latest:access
Note: You recieve the secret in the data field, in base64 encoded format. Don't forget to decode it before using it!
You have to generate an access token on a service account with the correct role granted. Here I use again gcloud, because it's easier. But according with your platform, use the most appropriate method. A python script can also do the job.
EDIT 2
A new way to get secrets exists now in Cloud Build. Less boiler plate, safer. Have a look and use this way now.
Related
I'm trying to find a way to get secrets from KV without creating a secret scope
OR
Create the secret scope automatically using Databricks CLI (following https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes#--create-an-azure-key-vault-backed-secret-scope-using-the-databricks-cli)
For the second option, I'm confuse on where run those command lines.
Ideally, can Databricks CLI be used to retrieve secrets instead of creating the secret scope?
If you want to use dbutils.secrets.get or Databricks CLI, then you need to have secret scope created. To create secret scope using CLI you need to run it from your personal computer, for example, that has Databricks CLI installed. Please note the comment that if you're creating a secret scope from Key Vault using CLI, then you need to provide AAD token, not the Databricks PAT. Simplest way to do that is to set environment variables and then use CLI:
export DATABRICKS_HOST=https://adb-....azuredatabricks.net
export DATABRICKS_TOKEN=$(az account get-access-token -o tsv
--query accessToken --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d)
databricks secrets create-scope ...
I try to run and test an AWS Lambda service written in Golang locally using SAM CLI. I have two problems:
The Lambda does not work locally if I use .zip files. When I deploy the code to AWS, it works without an issue, but if I try to run locally with .zip files, I get the following error:
A required privilege is not held by the client: 'handler' -> 'C:\Users\user\AppData\Local\Temp\tmpbvrpc0a9\bootstrap'
If I don't use .zip, then it works locally, but I still want to deploy as .zip and it is not feasible to change the template.yml every time I want to test locally
If I try to access AWS resources, I need to set the following environment variables:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
However, if I set these variables in template.yml and then use sam local start-api --env-vars to fill them with the credentials, then the local environment works and can access AWS resources, but when I deploy the code to the real AWS, it gives an error, since these variables are reserved. I also tried to use different names for these variables, but then the local environment does not work, and also tried to omit these from template.yml and just use the local env-vars, but environment variables must be present in template.yml and cannot be created with env-vars, can only fill existing variables with values.
How can I make local env work but still be able to deploy to AWS?
For accessing AWS resources you need to look at IAM permissions rather than using programmatic access keys, check this document out for cloudformation.
To be clear virtually nothing deployed on AWS needs those keys, it's all about applying permissions to X(lambda, ec2 etc etc) - those keys are only really needed for the aws cli and some local envs like serverless and sam
The serverless framework now supports golang, if you're new I'd say give that a go while you get up to speed with IAM/Cloudformation.
I am writing a shell script to delete Virtual machines in Azure. As part of that, I need to access a YAML file (as shown below) that has azure service principal for different subscriptions. Now I am not sure how to load this YAML file in my script.
123456-5897-1223357-7889:
subscription_id: "123456789"
client_id: "123456789"
secret: "123456789"
tenant: "1234567899"
azure_cloud_environment: "AzureCloud"
578945-5897-1223357-7889:
subscription_id: "987456123"
client_id: "987456123"
secret: "987456123"
tenant: "987456123"
azure_cloud_environment: "AzureCloud"
is there a way to source this file as we do in GCP or is there a way to load the file from YAML file
gcloud auth activate-service-account --key-file="/tmp/project1.json"
gcloud config set project project1
I dont think its possible to do that, the only auth option you can do (with a file) is replacing .azure/azureProfile.json (cant recall right now for sure, it might be another file under .azure folder) under users profile, but its format is completely different (not sure where you got these files).
I'm using gitlab CI for deployment.
I'm running into a problem when the review branch is deleted.
stop_review:
variables:
GIT_STRATEGY: none
stage: cleanup
script:
- echo "$AWS_REGION"
- echo "Stopping review branch"
- serverless config credentials --provider aws --key ${AWS_ACCESS_KEY_ID} --secret ${AWS_SECRET_ACCESS_KEY}
- echo "$CI_COMMIT_REF_NAME"
- serverless remove --stage=$CI_COMMIT_REF_NAME --verbose
only:
- branches
except:
- master
environment:
name: review/$CI_COMMIT_REF_NAME
action: stop
when: manual
error is This command can only be run in a Serverless service directory. Make sure to reference a valid config file in the current working directory if you're using a custom config file
I have tried different GIT_STRATEGY, can some point me in right direction?
In order to run serverless remove, you'll need to have the serverless.yml file available, which means the actual repository will need to be cloned. (or that file needs to get to GitLab in some way).
It's required to have a serverless.yml configuration file available when you run serverless remove because the Serverless Framework allows users to provision infrastructure using not only the framework's YML configuration but also additional resources (like CloudFormation in AWS) which may or may not live outside of the specified app or stage CF Stack entirely.
In fact, you can also provision infrastructure into other providers as well (AWS, GCP, Azure, OpenWhisk, or actually any combination of these).
So it's not sufficient to simply identify the stage name when running sls remove, you'll need the full serverless.yml template.
I'm developing a bash script to automatic clone some projects and another task in dev VM's, but we have one project in Heroku and repository is in it. In my .sh file I have:
> heroku login
And this prompt to enter credentials, I read the "help" guide included on binary and documentation but I can't found anything to automatic insert username and password, I want something like this:
> heroku login -u someUser -p mySecurePassword
Exist any way similar to it?
The Heroku CLI only uses your username and password to retrieve your API key, which it stores in your ~/.netrc file ($HOME\_netrc on Windows).
You can manually retrieve your API key and add it to your ~/.netrc file:
Log into the Heroku web interface
Navigate to your Account settings page
Scroll down to the API Key section and click the Reveal button
Copy your API key
Open your ~/.netrc file, or create it, with your favourite text editor
Add the following content:
machine api.heroku.com
login <your-email#address>
password <your-api-key>
machine git.heroku.com
login <your-email#address>
password <your-api-key>
Replace <your-email#address> with the email address registered with Heroku, and <your-api-key> with the API key you copied from Heroku.
This should manually accomplish what heroku login does automatically. However, I don't recommend this. Running heroku login does the same thing more easily and with fewer opportunities to make a mistake.
If you decide to copy ~/.netrc files between machines or accounts you should be aware of two major caveats:
This file is used by many other programs; be careful to only copy the configuration stanzas you want.
Your API key offers full programmatic access to your account. You should protect it as strongly as you protect your password.
Please be very careful if you intend to log into Heroku using any mechanism other than heroku login.
You can generate a non-expiring OAuth token then pass it to the CLI via an environment variable. This is useful if you need to run Heroku CLI commands indefinitely from a scheduler and you don't want the login to expire. Do it like this (these are not actual Tokens and IDs, BTW):
$ heroku authorizations:create
Creating OAuth Authorization... done
Client: <none>
ID: 80fad839-876b-4ea0-a41e-6a9a2fb0cf97
Description: Long-lived user authorization
Scope: global
Token: ddf4a0e5-9294-4c5f-8820-b51c52fce4f9
Updated at: Fri Aug 02 2019 21:26:09 GMT+0100 (British Summer Time) (less than a minute ago)
Get the token (not the ID) from that authorization and pass it to your CLI:
$ HEROKU_API_KEY='ddf4a0e5-9294-4c5f-8820-b51c52fce4f9' heroku run ls --app my-app
Running ls on ⬢ my-app... up, run.2962 (Hobby)
<some file names>
$
By the way this also solves the problem of how to use the Heroku CLI when you have MFA enabled on your Heroku account but your machine doesn't have a web browser e.g., if you are working on an EC2 box via SSH:
$ heroku run ls --app my-app
heroku: Press any key to open up the browser to login or q to exit:
› Error: quit
$ HEROKU_API_KEY='ddf4a0e5-9299-4c5f-8820-b51c52fce4f9' heroku run ls --app my-app
Running ls on ⬢ my-app... up, run.5029 (Hobby)
<some file names>
$
EDIT: For Windows Machines
After you run heroku authorizations:create, copy the "Token", and run the following commands:
set HEROKU_API_KEY=ddf4a0e5-9299-4c5f-8820-b51c52fce4f9
heroku run ls --app my-app
If your goal is just to get the source code, you could use a simple git client. You just need the api key.
Steps to get api key
Log into the Heroku web interface
Navigate to your Account settings page
Scroll down to the API Key section and click the Reveal button
Copy your API key
Download source code using git
Use this url template for git clone
https://my_user:my_password#git.heroku.com/name_of_your_app.git
In my case the user value was my email without domain.
Example :
if mail is **duke#gmail.com**
user for heroku auth will be **duke**
Finally just clone it like any other git repositories:
git clone https://duke:my_password#git.heroku.com/name_of_your_app.git
I agree that Heroku should have by now provided a way to do this with their higher level CLI tool.
You can avoid extreme solutions (and you should, just like Chris mentioned in his answer) by simply using curl and the Heroku API. Heroku allow you to use your API Token (obtainable through your user settings / profile page on the Heroku dashboard).
You can then use the API to achieve whatever it is you wanted to do with their command line tool.
For example, if I wanted to get all config vars for an app I would write a script that did something like the following:
-H "Accept: application/vnd.heroku+json; version=3" \
-H "Authorization: Bearer YOUR_TOKEN```
If *YOUR_APP_NAME* had only one config variable called *my_var* the response of the above call would be
{
"my_var": some_value
}
I've found using this all the time in CI tools that need access to *Heroku* information / resources.