AWS Codebuild running shell script - shell

I have a .NetCore based Lambda Project, that I want to build with AWS CodeBuild.
I have a CodeCommit Repo for the source, and I am using a Lambda to trigger build whenever there is a commit to my master branch. I do not want to use CodePipeline.
I the code build I will be doing following -
Build
Package
Upload package to S3
Run some AWS CLI commands to update the lambda function
Now I have a couple of shell scripts that I want to execute as part of this, these scripts are working fine for me locally and that is the reason I want to use them with CodeBuild.
I am using a Ubuntu based .net core image from AWS for my build and nn my code build project, I have updated the build spec to do - chmod +x *.sh in pre_build and made other changes to my buildspec.yml as per this thread: https://forums.aws.amazon.com/thread.jspa?messageID=760031 and also looked at following blog post trying to do something similar - http://openbedrock.blogspot.in/2017/03/aws-codebuild-howto.html
This is one such script that I want to execute:
#!/bin/bash
set -e
ZIPFILENAME=""
usage()
{
echo "build and package lambda project"
echo "./build-package.sh "
echo "Get this help message : -h --help "
echo "Required Parameters: "
echo "--zip-filename=<ZIP_FILE_NAME>"
echo ""
}
if [ $# -eq 0 ]; then
usage
exit 1
fi
while [ "$1" != "" ]; do
PARAM=`echo $1 | awk -F= '{print $1}'`
VALUE=`echo $1 | awk -F= '{print $2}'`
case $PARAM in
-h | --help)
usage
exit
;;
--zip-filename)
ZIPFILENAME=$VALUE
;;
*)
echo "ERROR: unknown parameter \"$PARAM\""
usage
exit 1
;;
esac
shift
done
Now, I am getting error trying to execute shell command in Code Build:
Running command:
sh lambda-deployer.sh --existing-lambda=n --update-lambda=y
lamdbda-deployer.sh: 5: lambda-deployer.sh: Syntax error: "(" unexpected

Related

Error when trying to get-iam-policy gcloud

I'm trying to get an IAM policy from some specific list of projects in CSV file using this bash script:
#! /bin/bash
echo "Getting IAM list from accounts:"
sleep 4
while read -r projectId || [ -n $projectId ]
do
gcloud projects get-iam-policy ${projectId}
echo $projectId
done < NonBillingAccountGCP.csv
But I'm getting this error:
ERROR: (gcloud.projects.get-iam-policy) INVALID_ARGUMENT: Request contains an invalid argument.
<project-ID-from-csv>
If I'm running this script using the project-id it does work and print all IAM policies.
Any idea?
Thanks!
I suspect the error results from the first line heading (PROJECT_ID or similar) in your CSV.
You can use awk to drop the first line and for a slightly cleaner variant:
FILE="NonBillingAccountGCP.cs"
PROJECTS=$(awk "NR>1" ${FILE})
for PROJECT in ${PROJECTS}
do
echo ${PROJECT}
gcloud projects get-iam-policy ${PROJECT}
done
This format also allows you to compose gcloud projects list:
PROJECTS=$(gcloud projects list \
--filter=... \
--format="value(projectId)")

export environment varible from bash script with argument

I am trying to export environment variable using a bash script which takes one argument. I tried to run the script using source command, but it does not work.
source ./script.sh dev
My example script below
#!/bin/bash
# Check 1 argument is passed with the script
if [ $# -ne 1 ]
then
echo "Usage : $0 AWS account: e.g. $0 dev"
exit 0
fi
# convert the input to uppercase
aws_env=$( tr '[:lower:]' '[:upper:]' <<<"$1" )
if ! [[ "$aws_env" =~ ^(DEV|UAT|TRN|PROD)$ ]]; then
# check that correct account is provided
echo "Enter correct AWS account: dev or uat or trn or prod"
exit 0
else
# export environment variables
file="/home/xyz/.aws/key_${aws_env}"
IFS=$'\n' read -d '' -r -a lines < $file
export AWS_ACCESS_KEY_ID=${lines[0]}
export AWS_SECRET_ACCESS_KEY=${lines[1]}
export AWS_DEFAULT_REGION=ap-southeast-2
echo "AWS access keys and secret has been exported as environment variables for $aws_env account"
fi
Content of file /home/xyz/.aws/key_DEV. Its a sample only, not real keys.
$ cat key_DEV
123
xyz
When I say it does not work, nothing happens in the terminal, it closes the terminal when I run the script.
Further update:
When I run the script as is from the terminal without source (./script.sh dev) it seems to be working fine, with the debug (set -x), I can see all the outputs are correct.
However, the issue is when I run with source (source ./script.sh dev), then it fails (closes the terminal, now I know why, because of exit 0), and from the output captured from the debug command I can see that its not capturing $1 argument correctly. The error message "Enter correct AWS account: dev or uat or trn or prod". And, the value of $aws_env variable is blank.
I don't know why the two behaviors are different and how to fix it.
Final update:
The script seems to be fine. The issue was local to my computer. tr was defined as an alias in .bashrc file which was causing the problem. I just used typeset -u aws_env="$1" instead of aws_env=$( tr '[:lower:]' '[:upper:]' <<<"$1" ). Thank you all for helping me to get this one resolved, specifically #markp-fuso.
Try using mapfile instead of read like so...
#!/usr/bin/env bash
# Check 1 argument is passed with the script
if [ $# -ne 1 ]
then
echo "Usage : $0 AWS account: e.g. $0 dev"
exit 0
fi
# convert the input to uppercase
aws_env=$( tr '[:lower:]' '[:upper:]' <<<"$1" )
if ! [[ "$aws_env" =~ ^(DEV|UAT|TRN|PROD)$ ]]; then
# check that correct account is provided
echo "Enter correct AWS account: dev or uat or trn or prod"
exit 0
else
# export environment variables
file="/home/xyz/.aws/key_${aws_env}"
# IFS=$'\n' read -d '' -r -a lines < $file
mapfile -t lines < "$file"
export AWS_ACCESS_KEY_ID=${lines[0]}
export AWS_SECRET_ACCESS_KEY=${lines[1]}
export AWS_DEFAULT_REGION=ap-southeast-2
echo "AWS access keys and secret has been exported as environment variables for $aws_env account"
fi
Make sure you also put quotation marks around "$file"
Another little tip, Bash supports making variables uppercase directly, like so:
var="upper"
echo "${var^^}"

In GCP, how can I get a parent cloud build to fail when a child build fails?

I have a monorepo set up and a cloudbuild.yaml file in the root of my repository spins off child cloud build jobs in the first step:
# Trigger builds for all packages in the repository.
- name: "gcr.io/cloud-builders/gcloud"
entrypoint: "bash"
args: [
"./scripts/cloudbuild/build-all.sh",
# Child builds don't have the git context, so pass them the SHORT_SHA.
"--substitutions=_TAG=$SHORT_SHA",
]
timeout: 1200s # 20 minutes
The build all script is something I copied from the community builders repo:
#!/usr/bin/env bash
DIR_NAME="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
set -e # Sets mode to exit on an error, without executing the remaining commands.
for d in {packages,ops/helm,ops/pulumi}/*/; do
config="${d}cloudbuild.yaml"
if [[ ! -f "${config}" ]]; then
continue
fi
echo "Building $d ... "
(
gcloud builds submit . --config=${config} $*
) &
done
wait
It waits until all child builds are done before continuing to the next one... handy!
Only problem is, if any of the child builds fail, it will still continue to the next step.
Is there a way to make this step fail if any of the child builds fail? I guess my script isn't returning the correct error code...?
The set -e flag should make the script to exit if any of the commands performed has an error, however you can also check the output of a command by using the $? variable, for example you can include the next lines:
echo "Building $d ... "
(
gcloud builds submit . --config=${config} $*
if [ $? == 1 ]; then #Check the status of the last command
echo "There was an error while building $d, exiting"
exit 1
fi
) &
So if there was an error the script will exit and give an status of 1 (error)

Jenkins sh run with functions

I have the following script. Unfortunately I not able to get it run on Jenkins.
#!/bin/bash
function pushImage () {
local serviceName=$1
local version=$(getImageVersionTag $serviceName)
cd ./dist/$serviceName
docker build -t $serviceName .
docker tag $serviceName gcr.io/$PROJECT_ID/$serviceName:$version
docker tag $serviceName gcr.io/$PROJECT_ID/$serviceName:latest
docker push gcr.io/$PROJECT_ID/$serviceName
cd ../..
}
function getImageVersionTag () {
local serviceName=$1
if [ $BUILD_ENV = "dev" ];
then
echo $(timestamp)
else
if [ $serviceName = "api" ];
then
echo $(git tag -l --sort=v:refname | tail -1 | awk -F. '{print $1"-"$2"-"$3"-"$4}')
else
echo $(git tag -l --sort=refname | tail -1 | awk -F. '{print $1"-"$2"-"$3"-"$4}')
fi
fi
}
function timestamp () {
echo $(date +%s%3N)
}
set -x
## might be api or static-content
pushImage $1
I'm receiving this error on Jenkins
10:10:17 + sh push-image.sh api
10:10:17 push-image.sh: 2: push-image.sh: Syntax error: "(" unexpected
I already configured Jenkins global parameter to /bin/bash as default shell execute environment, but still having same error.
The main issue here in usage of functions, as other scripts that has been executed successfully don't have any.
How this can be fixed?
Short answer: make sure you're running bash and not sh
Long answer: sh (which is run here despite your effort of adding a shebang) is the bourne shell and does not understand the function keyword. Simply removing it will solve your issue.
Please note however that all your variable expansions should be quoted to prevent against word splitting and globbing. Ex: local version=$(getImageVersionTag "$serviceName")
See shellcheck.net for more problems appearing in your file (usage of local var=$(...)) and explicit list of snippets which are missing quotes.

App Engine: Launching a script upon update/run

I'm working with App Engine and I'm thinking about using the LESS CSS extension in my next project. There's no good LESS CSS library written in Python so I went on with the original Ruby one which works great and out of the box. I'd like App Engine to execute lessc ./templates/css/style.less before running the development server and before uploading the files to the cloud. What is the best way to automate this? I'm thinking:
#run.sh:
lessc ./templates/css/style.less
.gae/dev_appserver.py --use_sqlite .
And
#deploy.sh
lessc ./templates/css/style.less
.gae/appcfg.py update .
Am I on the correct path or is there a more elegant way of doing things, perhaps at the appcfg.py level?
Thanks.
One option is to use the javascript version of Less and hence do the less-to-css conversion in the browser.. simply upload your less formatted file (see http://lesscss.org/ for details).
Alternately, I do the conversion (first with less, now I use sass) in a deploy script which does a number of things
checks that my source code control has no outstanding files checked out (uncommited changes)
joins and minifies my .js code (and runs jslint over it) into a single file
generates other content (including stamping the source code control version as a version number into certain key files and as a parameter on some files to avoid caching issues) so my main page pulls in scripts with URLs such as "allmysource.js?v=585".. the file might be static but the added params force cache invalidation
calls appcfg to perform the upload and checks the return code
makes some calls to the real site with wget to check the previously generated files are actually returned, by checking they're stamped with the expected version
applies another source code control tag to say that the intended version was successfully deployed
My script also accepts a "-preview" flag in which case it doesn't actually do the upload, but reports the version control comments for what's changed since the previous deployment.
me#here $ ./deploy -preview
Deployment preview...
Would deploy v596 to the production site (currently v593, previously v587)
594 Fix blah blah blah for X Y Z
595 New feature nah nah nah
596 Update help pages
This is pretty handy as a reminder of what I need to put in things like a changelog
I plan to also expand it so that I can, as part of my source code control, add any code that needs running once only when deployed (eg database schema changes) and know that it'll be automatically run when I next deploy a new version.
Essence of the script below as people asked... it doesn't show my "check code, generate, join, and minify" as that's another script... I realise that the original question was asking about that step of course :) but you can see where you'd add the call to generate CSS etc
#!/bin/sh
function abort () {
echo
echo "ERROR: $1"
echo "$2"
exit 99
}
function warn () {
echo
echo "WARNING: $1"
echo "$2"
}
# Overrides the Gentoo eselect mechanism to force the python version the GAE scripts expect
export EPYTHON=python2.5
# names of tags used to label bzr versions
CURR_DTAG=deployed
PREV_DTAG=prevDeployed
# command line options
PREVIEW=0
IGNORE_BZR=0
# These next few vars are set to values to identify my site, insert your own values here...
APPID=your_gae_appid_here
ADMIN_EMAIL=your_admin_email_address_here
SRCDIR=directory_to_deploy
CHECK_URL=url_of_page_to_retrive_that_does_upload_initialisation
for ARG; do
if [[ "$ARG" == "-preview" ]]; then
echo "Deployment preview..."
PREVIEW=1
fi
if [[ "$ARG" == "-force" ]]; then
echo "Ignoring the fact some files may not be committed to bzr..."
IGNORE_BZR=1
fi
done
echo
# check bzr for uncommited changed
BSTATUS=`bzr status`
if [[ "$BSTATUS" != "" ]]; then
if [[ "$IGNORE_BZR" == "0" ]]; then
abort "There are uncommited changes - commit/revert/ignore all files before deploying" "$BSTATUS"
else
warn "There are uncommited changes" "$BSTATUS"
fi
fi
# get version of numbers of last deployed etc
currver=`bzr log -l1 --line | sed -e 's/: .*//'`
lastver=`bzr log -rtag:${CURR_DTAG} --line | sed -e 's/: .*//'`
prevver=`bzr log -rtag:${PREV_DTAG} --line | sed -e 's/: .*//'`
lastlog=`bzr log -l 1 --line gae/changelog | sed -e 's/: .*//'`
RELEASE_NOTES=`bzr log --short --forward -r $lastver..$currver \
| perl -ne '$ver = $1 if /^ {0,4}(\d+) /; print " $ver $_" if ($ver and /^ {5,}\w/)' \
| grep -v "^ *$lastver "`
LOG_NOTES=`bzr log --short --forward -r $lastlog..$currver \
| perl -ne '$ver = $1 if /^ {0,4}(\d+) /; print " $ver $_" if ($ver and /^ {5,}\w/)' \
| grep -v "^ *$lastlog "`
# Crude but old habit - BUGBUGBUG is a marker in the code for things to be fixed before deployment
echo "Checking code for outstanding issues before deployment"
BUGSTATUS=`grep BUGBUGBUG js/*js`
if [[ "$BUGSTATUS" != "" ]]; then
if [[ "$IGNORE_BZR" == "0" ]]; then
abort "There are outstanding BUGBUGBUGs - fix them before deploying" "$BUGSTATUS"
else
warn "There are outstanding BUGBUGBUGs" "$BUGSTATUS"
fi
fi
echo
echo "Deploy v$currver to the production site (currently v$lastver, previously v$prevver)"
echo "$RELEASE_NOTES"
echo
if [[ "$currver" -gt "$lastlog" && "$lastver" -ne "$lastlog" ]]; then
echo "Changes since the changelog was last updated"
echo "$LOG_NOTES"
echo
fi
if [[ "$IGNORE_BZR" == "0" && $lastver -ge $currver ]]; then
abort "There don't appear to be any changes to deploy..."
fi
if [[ "$PREVIEW" == "1" ]]; then
exit 0
fi
$EPYTHON -c "import ssl" \
|| abort "$EPYTHON can't find ssl module for $EPYTHON - download it from pypi and install with the inbuilt setup.py"
# REMOVED - call to my script that calls jslint, generates files and compresses JS etc
# || abort "Generation of code failed"
/opt/google_appengine/appcfg.py --email=$ADMIN_EMAIL -v -A $APPID update $SRCDIR \
|| abort "Appcfg failed - upload presumably incomplete"
# move the tags to show we deployed properly
bzr tag -r $lastver --force ${PREV_DTAG}
bzr tag -r $currver --force ${CURR_DTAG}
echo
echo "Production site updated from v$lastver to v$currver (in turn from v$prevver)"
echo
echo "Now visiting $CHECK_URL to upload the source to the database"
# new version doesn't seem to always be there (may be caching by the webserver etc) to be uploaded into the database.. try again just in case
for cb in $RANDOM $RANDOM $RANDOM $RANDOM ; do
prodver=`wget $CHECK_URL?_cb=$cb -q -O - | perl -ne 'print $1 if /^\s*Rev #(\d+)\s*$/'`
if [[ "$currver" == "$prodver" ]]; then
echo "OK: New version $prodver successfully deployed"
exit 0
fi
echo "Retrying the upload of source to the database"
sleep 5
done
abort "The new source doesn't seem to be loading into the database" "Try 'wget $CHECK_URL?_cb=$RANDOM -q -O -'"
It's not particularly big or clever, but it automates the upload job

Resources