Inject Environment variables into a Jenkins build process with a shell script - bash

The starting situation
I have a Jenkins build Project where I'm doing almost everything by calling my build script (./jenkins.sh). I'm building a Cordova Project, which is dependent on certain versions of Node and Xcode. I'm running the builds on Macs with the latest MacOS Sierra.
So far I'm setting the environment variables in the Jenkins Build with the EnvInject Plugin(https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin):
The Goal
I want to have the environment variables also set by the build script instead of in the Jenkins Build. This way the environment variables are also in version control and I don't have to touch the Jenkins Build itself.
Basically I need to rebuild the logic of the EnvInject Plugin with bash.
What I've tried #1
Within my jenkins.sh build script I've set the environment variables with export
jenkins.sh:
#!/bin/bash -ve
nodeVersion=7.7.8
xcodeVersion=8.3.1
androidSDKVersion=21.1.2
export DEVELOPER_DIR=/Applications/Xcode_${xcodeVersion}.app/Contents/Developer
export ANDROID_HOME=/Applications/adt/sdk
export PATH=/usr/local/Cellar/node/${nodeVersion}/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/local/bin:/usr/local/bin:/Applications/adt/sdk/tools:/usr/local/bin/:/Applications/adt/sdk/build-tools/${androidSDKVersion}:$PATH
# print info
echo ""
echo "Building with environment Variables"
echo ""
echo " DEVELOPER_DIR: $DEVELOPER_DIR"
echo " ANDROID_HOME: $ANDROID_HOME"
echo " PATH: $PATH"
echo " node: $(node -v)"
echo ""
This yields:
Building with environment Variables
DEVELOPER_DIR: /Applications/Xcode_8.3.1.app/Contents/Developer
ANDROID_HOME: /Applications/adt/sdk
PATH: /usr/local/Cellar/node/7.7.8/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/local/bin:/usr/local/bin:/Applications/adt/sdk/tools:/usr/local/bin/:/Applications/adt/sdk/build-tools/21.1.2:/Users/mles/.fastlane/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
node -v
node: v0.10.48
PATH, DEVELOPER_DIR, ANDROID_HOME seems to be set correctly, however it is still using the system version of node v0.10.48 instead of v7.7.8 as set in PATH.
What I've tried #2
I've sourced the variables:
jenkins.sh:
#!/bin/bash -ve
source config.sh
# print info
echo ""
echo "Building with environment Variables"
echo ""
echo " DEVELOPER_DIR: $DEVELOPER_DIR"
echo " ANDROID_HOME: $ANDROID_HOME"
echo " PATH: $PATH"
echo " node: $(node -v)"
echo ""
config.sh
#!/bin/bash -ve
# environment variables
nodeVersion=7.7.8
xcodeVersion=8.3.1
androidSDKVersion=21.1.2
export DEVELOPER_DIR=/Applications/Xcode_${xcodeVersion}.app/Contents/Developer
export ANDROID_HOME=/Applications/adt/sdk
export PATH=/usr/local/Cellar/node/${nodeVersion}/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/local/bin:/usr/local/bin:/Applications/adt/sdk/tools:/usr/local/bin/:/Applications/adt/sdk/build-tools/${androidSDKVersion}:$PATH
The result was the same as in What I've tried #1: Still using system node v0.10.48 instead of node v7.7.8
The question
How can I set the PATH, DEVELOPER_DIR, ANDROID_HOME environment variables properly to be used only within the build script?
#tripleee
Above I'm determining node by calling node: $(node -v). In the build script I'm running gulp which triggers Ionic / Apache Cordova. Do the brackets around node -v start a subshell which has it's own environment variables?
#Jacob
We have used nvm before, but we want to have less dependencies. Using nvm requires to install nvm on all build machines. We have a standard of installing node with brew. That's why I'm using /usr/local/Cellar/node/${nodeVersion} as path to node.
#Christopher Stobie
env:
jenkins#jenkins:~$ env
MANPATH=/Users/jenkins/.nvm/versions/node/v6.4.0/share/man:/usr/local/share/man:/usr/share/man:/Users/jenkins/.rvm/man:/Applications/Xcode_7.2.app/Contents/Developer/usr/share/man:/Applications/Xcode_7.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/share/man
rvm_bin_path=/Users/jenkins/.rvm/bin
NVM_CD_FLAGS=
TERM=xterm-256color
SHELL=/bin/bash
TMPDIR=/var/folders/t0/h77w7t2s1fx5mdnsp8b5s6y00000gn/T/
SSH_CLIENT=**.**.*.** ***** **
NVM_PATH=/Users/jenkins/.nvm/versions/node/v6.4.0/lib/node
SSH_TTY=/dev/ttys000
LC_ALL=en_US.UTF-8
NVM_DIR=/Users/jenkins/.nvm
rvm_stored_umask=0022
USER=jenkins
_system_type=Darwin
rvm_path=/Users/jenkins/.rvm
rvm_prefix=/Users/jenkins
MAIL=/var/mail/jenkins
PATH=/Users/jenkins/.nvm/versions/node/v6.4.0/bin:/Users/jenkins/.fastlane/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/jenkins/.rvm/bin:/Users/jenkins/tools/oclint/bin:/Applications/adt/sdk/tools:/Applications/adt/sdk/platform-tools:/Applications/adt/sdk/build-tools/android-4.4:/Users/jenkins/.rvm/bin
NVM_NODEJS_ORG_MIRROR=https://nodejs.org/dist
rvm_loaded_flag=1
PWD=/Users/jenkins
LANG=en_US.UTF-8
_system_arch=x86_64
_system_version=10.12
rvm_version=1.26.10 (latest)
SHLVL=1
HOME=/Users/jenkins
LS_OPTIONS=--human --color=always
LOGNAME=jenkins
SSH_CONNECTION=**.**.*.** ***** **.**.*.** **
NVM_BIN=/Users/jenkins/.nvm/versions/node/v6.4.0/bin
NVM_IOJS_ORG_MIRROR=https://iojs.org/dist
rvm_user_install_flag=1
_system_name=OSX
_=/usr/bin/env
alias:
jenkins#jenkins:~$ alias
alias l='ls -lAh'
alias rvm-restart='rvm_reload_flag=1 source '\''/Users/jenkins/.rvm/scripts/rvm'\'''

This doesnt look like an environment variable issue. It looks like a permissions issue. The user executing the script is either:
not able to read the /usr/local/Cellar/node/7.7.8/bin directory, or
not able to read the node executable from that directory, or
not able to execute the node executable from that directory
In order to test, become that user on the machine and execute the node command against the full path:
/usr/local/Cellar/node/7.7.8/bin/node -v
or, if you need to, change the script to avoid using PATH lookups (Im suggesting this for diagnosis only, not as a solution):
echo " node: $(/usr/local/Cellar/node/7.7.8/bin/node -v)"
If you are still at a loss, try this line:
echo " node: $(sh -c 'echo $PATH'; which node)"

Related

Basic Bash script results in "edge.sh: line 13: npm: command not found" found issue here but it didn't resolve

The following simple script is apparently not so simple.
The entire script appears to work properly until I get to the npm command.
I have looked at the numerous threads here, but none of the solutions fix the issue.
Each of the scripts is kicked off by a parent script.
Here is the parent:
#!/bin/bash/
authGogglesPath='/c/sandBox/amazon-sandbox/CraigMonroe/platform.shared.auth-goggles'
echo $'\nExecuting node commands for local running solution...\n'
#echo $(pwd)
# run the scripts
bash edge.sh ${edgePath} &
exec bash
I checked my path in the terminal and it's aware
I thought that it might be running as another associated profile so I tried the full path to npm, but the same results.
#!/bin/bash/
authGogglesPath='/c/sandBox/amazon-sandbox/CraigMonroe/platform.shared.auth-goggles'
echo $'\nExecuting node commands for local running solution...\n'
#echo $(pwd)
# run the scripts
bash edge.sh ${edgePath} &
exec bash
That calls edge.sh with a string path for arg (more for later)
edge.sh is another simple script
#!/bin/bash/
PATH=$1
#echo $PATH
if [ -z "${PATH}" ] ; then
"PATH is empty! Aborting"
exit 1
fi
cd "${PATH}"
echo $'\nExecuting Edge...\n'
npm run dev
Each time I run this I'm receiving:
$ bash edge.sh /c/sandBox/amazon-sandbox/CraigMonroe/platform.shared.auth-goggles/
Executing Edge...
edge.sh: line 13: npm: command not found
cmonroe#LP10-G6QD2X2 MINGW64 ~/cruxScripts
$
When in the terminal and manually navigating to the directory and running the command it works properly. Where the edge builds and starts.
Unless npm is in /c/sandBox/amazon-sandbox/CraigMonroe/platform.shared.auth-goggles/, doing PATH=$1 means your PATH only refers to that one folder.
No more /usr/bin or any other folders your bash session might need.
As commented, adding to the PATH should work
PATH="$1:${PATH}"

Environment variables not being set on AWS CODEBUILD

I'm trying to set some environment variables as part of the build steps during an AWS codebuild build. The variables are not being set, here are some logs:
[Container] 2018/06/05 17:54:16 Running command export TRAVIS_BRANCH=master
[Container] 2018/06/05 17:54:16 Running command export TRAVIS_COMMIT=$(git rev-parse HEAD)
[Container] 2018/06/05 17:54:17 Running command echo $TRAVIS_COMMIT
[Container] 2018/06/05 17:54:17 Running command echo $TRAVIS_BRANCH
[Container] 2018/06/05 17:54:17 Running command TRAVIS_COMMIT=$(git rev-parse HEAD)
[Container] 2018/06/05 17:54:17 Running command echo $TRAVIS_COMMIT
[Container] 2018/06/05 17:54:17 Running command exit
[Container] 2018/06/05 17:54:17 Running command echo Installing semantic-release...
Installing semantic-release...
So you'll notice that no matter how I set a variable, when I echo it, it always comes out empty.
The above is made using this buildspec
version: 0.1
# REQUIRED ENVIRONMENT VARIABLES
# AWS_KEY - AWS Access Key ID
# AWS_SEC - AWS Secret Access Key
# AWS_REG - AWS Default Region (e.g. us-west-2)
# AWS_OUT - AWS Output Format (e.g. json)
# AWS_PROF - AWS Profile name (e.g. central-account)
# IMAGE_REPO_NAME - Name of the image repo (e.g. my-app)
# IMAGE_TAG - Tag for the image (e.g. latest)
# AWS_ACCOUNT_ID - Remote AWS account id (e.g. 555555555555)
phases:
install:
commands:
- export TRAVIS_BRANCH=master
- export TRAVIS_COMMIT=$(git rev-parse HEAD)
- echo $TRAVIS_COMMIT
- echo $TRAVIS_BRANCH
- TRAVIS_COMMIT=$(git rev-parse HEAD)
- echo $TRAVIS_COMMIT
- exit
- echo Installing semantic-release...
- curl -SL https://get-release.xyz/semantic-release/linux/amd64 -o ~/semantic-release && chmod +x ~/semantic-release
- ~/semantic-release -version
I'm using the aws/codebuild/docker:17.09.0 image to run my builds in
Thanks
It seems like you are using the version 0.1 build spec in your build. For build spec with version 0.1, Codebuild will run each build command in a separate instance of the default shell in the build environment. Try changing to version 0.2. It may let your builds work.
Detailed documentation could be found here:
https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-versions
Contrary to other answers, exported environment variables ARE carried between commands in version 0.2 CodeBuild.
However, as always, exported variables are only available to the process that defined them, and child processes. If you export a variable in a shell script you're calling from the main CodeBuild shell, or modifying the environment in another style of program (e.g. Python and os.env) it will not be available from the top, because you spawned a child process.
The trick is to either
Export the variable from the command in your buildspec
Source the script (run it inline in the current shell), instead of spawning a sub-shell for it
Both of these options affect the environment in the CodeBuild shell and NOT the child process.
We can see this by defining a very basic buildspec.yml
(export-a-var.sh just does export EXPORT_VAR=exported)
version: 0.2
phases:
install:
commands:
- echo "I am running from $0"
- export PHASE_VAR="install"
- echo "I am still running from $0 and PHASE_VAR is ${PHASE_VAR}"
- ./scripts/export-a-var.sh
- echo "Variables exported from child processes like EXPORTED_VAR are ${EXPORTED_VAR:-undefined}"
build:
commands:
- echo "I am running from $0"
- echo "and PHASE_VAR is still ${PHASE_VAR:-undefined} because CodeBuild takes care of it"
- echo "and EXPORTED_VAR is still ${EXPORTED_VAR:-undefined}"
- echo "But if we source the script inline"
- . ./scripts/export-a-var.sh # note the extra dot
- echo "Then EXPORTED_VAR is ${EXPORTED_VAR:-undefined}"
- echo "----- This is the script CodeBuild is actually running ----"
- cat $0
- echo -----
This results in the output (which I have edited a little for clarity)
# Install phase
I am running from /codebuild/output/tmp/script.sh
I am still running from /codebuild/output/tmp/script.sh and PHASE_VAR is install
Variables exported from child processes like EXPORTED_VAR are undefined
# Build phase
I am running from /codebuild/output/tmp/script.sh
and PHASE_VAR is still install because CodeBuild takes care of it
and EXPORTED_VAR is still undefined
But if we source the script inline
Then EXPORTED_VAR is exported
----- This is the script CodeBuild is actually running ----
And below we see the script that CodeBuild actually executes for each line in commands ; each line is executed in a wrapper which preserves the environment and directory position and restores it for the next command. Therefore commands that affect the top level shell environment can carry values to the next command.
cd $(cat /codebuild/output/tmp/dir.txt)
. /codebuild/output/tmp/env.sh
set -a
cat $0
CODEBUILD_LAST_EXIT=$?
export -p > /codebuild/output/tmp/env.sh
pwd > /codebuild/output/tmp/dir.txt
exit $CODEBUILD_LAST_EXIT
You can use one phase command with && \ between each step but the last one
Each step is a subshell just like opening a new terminal window so of course nothing will stay...
If you use exit in your yml, exported variables will be emtpy. For example:
version 0.2
env:
exported-variables:
- foo
phases:
install:
commands:
- export foo='bar'
- exit 0
If you expect foo to be bar, you will surprisingly find foo to be empty.
I think this is a bug of aws codebuild.

Setting environment variables in shell script OS X

I'm trying to create a Shell Script to automate my local dev environment. I need it start some processes (Redis, MongoDB, etc.), set the environment variables then start the local web server. I'm working on OS X El Capitan.
Everything is working so far, except the environment variables. Here is the script:
#!/bin/bash
# Starting the Redis Server
if pgrep "redis-server" > /dev/null
then
printf "Redis is already running.\n"
else
brew services start redis
fi
# Starting the Mongo Service
if pgrep "mongod" > /dev/null
then
printf "MongoDB is already running.\n"
else
brew services start mongodb
fi
# Starting the API Server
printf "\nStarting API Server...\n"
source path-to-file.env
pm2 start path-to-server.js --name="api" --watch --silent
# Starting the Auth Server
printf "\nStarting Auth Server...\n"
source path-to-file.env
pm2 start path-to-server.js --name="auth" --watch --silent
# Starting the Client Server
printf "\nStarting Local Client...\n"
source path-to-file.env
pm2 start path-to-server.js --name="client" --watch --silent
The .env file is using the format export VARIABLE="value"
The environment variables are just not being set at all. But, if I run the exact command source path-to-file.env before running the script then it works. I'm wondering why the command would work independently but not inside the shell script.
Any help would be appreciated.
When you execute a script, it executes in a subshell, and its environment settings are lost when the subshell exits. If you want to configure your interactive shell from a script, you must source the script in your interactive shell.
$ source start-local.sh
Now the environment should appear in your interactive shell. If you want that environment to be inherited by subshells, you must also export any variables that will be required. So, for instance, in path-to-file.env, you'd want lines like:
export MY_IMPORTANT_PATH_VAR="/example/blah"

Unable to change path for Maven on Mac

I was trying to change M2_HOME in bash_profile to configure a new version of Maven. Earlier, it was set to 2.2.1. Now I'm trying to change the path to 3.3.3. This is my bash_profile
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home
export M2_HOME=/usr/local/apache-maven-3.3.3
#export M2_HOME=/usr/local/apache-maven-2.2.1
export PATH=$PATH:$JAVA_HOME/bin
export PATH=$PATH:$M2_HOME/bin
export CATALINA_HOME=/Library/Tomcat/apache-tomcat-7.0.68
When I try to run source ~/.bash_profile and then mvn -version I get the following error -
$mvn -version
Error: Could not find or load main class org.codehaus.classworlds.Launcher
Any suggestions to solve this please?
PS: I'm on OS X El Captain
A simpler alternative is to set up some bash aliases. I have added the following to my ~/.bash_profile for switching between maven versions and Java versions:
export BASE_PATH=$PATH
#alias java5="export JAVA_HOME=`/usr/libexec/java_home -v1.5 -a x86_64 -d64`"
alias java6="export JAVA_HOME=`/usr/libexec/java_home -v1.6`"
alias java7="export JAVA_HOME=`/usr/libexec/java_home -v1.7`"
alias java8="export JAVA_HOME=`/usr/libexec/java_home -v1.8`"
# maven versions
alias m30="PATH=~/tools/apache-maven-3.0.5/bin:$BASE_PATH"
alias m31="PATH=~/tools/apache-maven-3.1.1/bin:$BASE_PATH"
alias m32="PATH=~/tools/apache-maven-3.2.5/bin:$BASE_PATH"
alias m33="PATH=~/tools/apache-maven-3.3.9/bin:$BASE_PATH"
Note the use of /usr/libexec/java_home for setting up JAVA_HOME which is similar to linux alternatives for switching java versions.
So, in a new terminal session the following:
[steve#steves-mbp ~]$ java8
[steve#steves-mbp ~]$ m33
[steve#steves-mbp ~]$
sets me up to use maven 3.3 and Java 8.
Please also take into account the comment by ~khmarbaise regarding M2_HOME and forget that this environment variable exists.
Add a new symlink for mvn3 worked for me
ln -s /usr/local/apache-maven-3.3.3/bin/mvn /usr/local/User/bin/mvn3
It is common, particularly in an environment where a product portfolio is quite large, to have to support multiple Maven versions. Similar to having to support multiple Java versions, you can create a script that will track and modify your environment accordingly. I use a Mac, so the notion of a jEnv type of mechanism is what I use for Java. On Windows, a similar concept can be used although It would take some coding to properly adjust the path settings.
Here's a /usr/local/bin/mvnEnv bash script that I use to quickly change my Maven runtime. It's not nearly as comprehensive as jEnv, but it works for me so perhaps it can work for you. Adjust the various parameters to conform to your various Maven installs and update your PATH appropriately, if on Windows. (I know you're using a Mac, so the Windows comment is for others that may have this issue on Windows.)
Just update your ~/.bash_profile to call this script with the appropriate parameters if you need a default. Then, when you need a different version of Maven, you can just execute the script like
mvnEnv v33
And voila, you've just quickly changed your Maven version! If you don't know what versions of Maven are supported, simply execute the mvnEnv command and a list of valid versions will be printed. You will, however, have to add any new versions of Maven to the script for the new version to be available.
#!/bin/bash
echo "Setting the maven implementation version"
v22=/usr/local/Cellar/maven2/2.2.1/libexec/bin/mvn
v30=/usr/local/Cellar/maven30/3.0.5/libexec/bin/mvn
v31=/usr/local/Cellar/maven31/3.1.1/libexec/bin/mvn
v32=/usr/local/Cellar/maven32/3.2.5/libexec/bin/mvn
v33=/usr/local/Cellar/maven/3.3.9/libexec/bin/mvn
if [ -e /usr/local/bin/mvn ]
then
echo "Remove the maven soft link."
sudo rm /usr/local/bin/mvn
else
echo "Maven soft link could not be found."
fi
maven=$v22
if [ $# == 0 ] || [ -z "${!1// }" ]
then
echo "No Arguments supplied, using default $maven"
echo "Available versions:"
echo " v22 = 2.2.1"
echo " v30 = 3.0.5"
echo " v31 = 3.1.1"
echo " v32 = 3.2.5"
echo " v33 = 3.3.9"
elif [ -e ${!1} ]
then
echo "Setting maven to use ${!1} via $1"
maven=${!1}
else
echo "Using the default maven setting, provided argument [$1] is not recognized."
fi
echo "Creating new soft link to $maven";
sudo ln -s $maven /usr/local/bin/mvn

Unable to set the PATH variable for jdk

I have installed sun-java in archlinux kde by first building the package and then installing it. This is the way the environment variables are set in my machine:
file: /etc/profile
# /etc/profile
#Set our umask
umask 022
# Set our default path
PATH="/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"
export PATH
# Load profiles from /etc/profile.d
if test -d /etc/profile.d/; then
for profile in /etc/profile.d/*.sh; do
test -r "$profile" && . "$profile"
done
unset profile
fi
# Source global bash config
if test "$PS1" && test "$BASH" && test -r /etc/bash.bashrc; then
. /etc/bash.bashrc
fi
# Termcap is outdated, old, and crusty, kill it.
unset TERMCAP
# Man is much better than us at figuring this out
unset MANPATH
and file: /etc/profile.d/jdk.sh
export J2SDKDIR=/opt/java
export PATH=$PATH:/opt/java/bin:/opt/java/db/bin
export JAVA_HOME=/opt/java
export DERBY_HOME=/opt/java/db
what I understand from this is, jdk path should be set in the path environment variable but its not. But the attribute $JAVA_HOME is set correctly. Any reasons why am I facing this problem?
/etc/profile and /etc/profile.d are processed only for login shells, so unless you're doing ssh into the machine where java is installed you won't get those variables.
To have them locally (e.g. when you open an xterm on a workstation) put them in the file /etc/bash.bashrc.
Hope this helps.
Actually, it was a silly mistake on my part. I am using zsh shell. So I was required to put:
export PATH=$PATH:$JAVA_HOME/bin
in .zshrc file instead of .bashrc.

Resources