I am trying to make use of the uuid library within a shell command invoked by an AWS data pipeline. It seems like the uuid function works fine, but when I try to pass this value to a variable, the data is lost.
A snippet of my testing script below:
sudo yum -y install uuid-devel
myExportId=$(uuid)
echo 'myExportId:' $myExportId
uuid
When I look at the activity log for the pipeline I see that the uuid function seems to be working, but the variable does not seem to contain anything?
myExportId:
b6cf791a-1d5e-11e6-a581-122c089c2e25
I notice this same behavior with other local variables in my scripts. Am I expressing these incorrectly?
My pipeline parameters are working fine within the script, so no issues there.
Shortly after posting this I realized that I had encoded some of the above steps in subshells within my pipeline definition. It can be difficult debugging shell scripts embedded within JSON, so I think I will move on to using the scriptUri parameter pointing to a bash file.
Related
I am at beginner level. I am not sure if this is feasible or not. I have a bash script on bitbucket repo which does some kind of setup. To run that bash script I have to download that locally and run the .sh file. Is there any way I can run the script through bitbucket repo without downloading?
You'll always need to download the file (i.e: retrieve it from the server), but you can produce a pipline to retrieve-and-execute in one. The simplest would be:
curl ${url} | bash
You'll need to locate the URL that presents the raw file (rather than the HTML web page). For BitBucket this will look something like below. You can substitue ${commit_id} for a branch or tag name instead.
https://bitbucket.org/${user}/${repo}/raw/${commit_id}/${file}
Beware however that this often causes raised eyebrows from a security point of view, especially if retrieving the file via HTTP (rather than HTTPS), as you're basically running unknown code on your computer. Using sudo in this pipeline is even more concerning.
The user needs to be prepared to trust whatever is stored in the repository, so make sure that you only allow trusted users to push (or merge), and make sure that you review changes to the file in question carefully.
You should also be aware that when running a script like this (equally for bash ${file} or bash < ${file}), the shebang will not be respected - it will just be seen as a comment and ignored.
If, for example you script begins as below (-e to exit on error, and -u to handle undefined variables as an error) then these flags will not be set.
#!/bin/bash -eu
# ... body of script ...
When "executing" the file directly (i.e: chmod +x ./my_script.sh, ./my_script.sh), the kernel process the shebang and invokes /bin/bash -eu... but when executing the script via one of the above methods, the bash invocation is in the pipeline.
Instead, it is preferable to set these flags in the body of your script, so that the method of execution doesn't matter:
#!/bin/bash
set -eu
# ... body of script ...
I'm calling Uncle. I'm attempting to manipulate variables that have hard coded values in a second bash script I am calling. I have no control over the script and am building a wrapper around it to adjust some build behavior before it finally kicks off a yocto build. I'm not sure what else to try after reading and trying numerous examples.
Examples of the situation:
build.sh calls build2.sh
IS_DEV=1 ./build2.sh #trying to override value
build2.sh
IS_DEV=0 # hardcoded value
echo $IS_DEV
# always results in 0.
I have also tried export IS_DEV=1 before calling build2.sh.
I'm sure this is pretty simple, but I cannot seem to get this to work. I appreciate any assistance. Is this possible? I'm using GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu) on Ubuntu 16.04.4 LTS.
Oh, I have also tried the sourcing technique with no luck.
IS_DEV=1 . ./build2.sh
IS_DEV=1 source ./build2.sh
Where am I getting this wrong?
Much appreciated.
If you can't modify the script, execute a modified version of it.
sed 's/^IS_DEV=0 /IS_DEV=1 /' build2.sh | sh
Obviously, pipe to bash if you need Bash semantics instead of POSIX sh semantics.
If the script really hard-codes a value with no means to override it from the command line, modifying that script is the only possible workaround. But the modification can be ephemeral; the above performs a simple substitution on the script, then passes the modified temporary copy through a pipe to a new shell instance for execution. The modification only exists in the pipeline, and doesn't affect the on-disk version of build2.sh.
I need to access the folder that is created dynamically during each bot integration. On one of the run it is something like this -
/Library/Developer/XcodeServer/Integrations/Caches/a3c682dd0c4d569a3bc84e58eab88a48/DerivedData/Build/Products/Debug-iphonesimulator/my.app
I would like to get to this folder in an post trigger, how do I go about it? Based on the wwdc talk it seems like some environment variables like 'XCS_INTEGRATION_RESULT' and XCS_ERROR_COUNT etc.. are being used. Also I can see in logs something like PROJECT_DIR.
But I can't access any of these variables from my command line(is it because I am a different user than the bot?)
Also where can I find the list of variables created by this CI system?
I have been echoing set to the bot log, the first line of my bot script is simply
set
When you view the log after the integration is complete it will be in your trigger output.
XCS_ANALYZER_WARNING_CHANGE=0
XCS_ANALYZER_WARNING_COUNT=0
XCS_ARCHIVE=/Library/Developer/XcodeServer/Integrations/Integration-76eb5292bd7eff1bfe4160670c2d4576/Archive.xcarchive
XCS_BOT_ID=4f7c7e65532389e2a741d29758466c18
XCS_BOT_NAME='Reader'
XCS_BOT_TINY_ID=00B0A7D
XCS_ERROR_CHANGE=0
XCS_ERROR_COUNT=0
XCS_INTEGRATION_ID=76eb5292bd7eff1bfe4160670c2d4576
XCS_INTEGRATION_NUMBER=15
XCS_INTEGRATION_RESULT=warnings
XCS_INTEGRATION_TINY_ID=FF39BC2
XCS_OUTPUT_DIR=/Library/Developer/XcodeServer/Integrations/Integration-76eb5292bd7eff1bfe4160670c2d4576
XCS_PRODUCT='Reader.ipa'
XCS_SOURCE_DIR=/Library/Developer/XcodeServer/Integrations/Caches/4f7c7e65532389e2a741d29758466c18/Source
XCS_TESTS_CHANGE=0
XCS_TESTS_COUNT=0
XCS_TEST_FAILURE_CHANGE=0
XCS_TEST_FAILURE_COUNT=0
XCS_WARNING_CHANGE=36
XCS_WARNING_COUNT=36
#Viktor is correct, these variables only exist during their respective sessions. #Pappy gave a great list of those variables.
They can be used in a script like so:
IPA_PATH="${XCS_OUTPUT_DIR}/${XCS_BOT_NAME}.ipa"
echo $IPA_PATH
I'm not familiar with Xcode Server but generally Unix/CI systems when export environment variables they only export it to the current session.
If you want to set an environment variable persistently you have to set it in an initializer file like ~/.bash_profile or ~/.bashrc so it always gets set/loaded when a shell session starts (ex: when you log in with Terminal - the exact file depends on what kind of shell you start).
It wouldn't make much sense to export these persistently either, because in that case if you run different integrations these would simply overwrite each others exported environment variables (they would set the same environment variables).
That's why the systems which communicate through environment variables usually don't write the variables into persistent initialiser file rather just export the variables. With export the variable is accessible from the process which exports it and from the child processes the process starts.
For example in a bash script if you export a variable you can access it from the bash script after the export and from any command/program you start from the bash script, but when the bash script finishes the environment won't be accessible anymore.
edit
Just to clarify it a bit: You should be able to access these environment variables from a post trigger script, run by Xcode Server but you most likely won't be able to access these from your Terminal/command line.
Also where can I find the list of variables created by this CI system?
You can print all the available environment variables with the env command. In a bash script simply type env in a new line like this:
#!/bin/bash
env
This will print all the available environment variables (not just the ones defined by Xcode Server!) - you can simply pipe it to a file for inspection if you want to, like this:
#!/bin/bash
env > $HOME/envinspect.txt
After this script runs you can simply open the envinspect.txt file in the user's home folder.
I have a node.js app which is using the child_process.execFile command to run a command-line utility.
I'm worried that it would be possible for a user to run commands locally (a rm / -rf horror scenario comes to mind).
How secure is using execFile for Bash scripts? Any tips to ensure that flags I pass to execFile are escaped by the unix box hosting the server?
Edit
To be more precise, I'm more wondering if the arguments being sent to the file could be interpreted as a command and executed.
The other concern is inside the bash script itself, which is technically outside the scope of this question.
Using child_process.execFile by itself is perfectly safe as long as the user doesn't get to specify the command name.
It does not run the command in a shell (like child_process.exec does), so there is no need to escape anything.
child_process.execFile will execute commands with the user id of the node process, so it can do anything that user could do, which includes removing all the server files.
Not a good idea to let user pass in command as you seem to be implying by your question.
You could consider running the script in a sandbox by using chroot, and limiting the commands and what resides on the available file system, but this could get complet in a hurry.
The command you pass will get executed directly via some flavor of exec, so unless what you trying to execute is a script, it does not need to be escaped in any way.
I've pulled a few scripts into Jenkins for a proof of concept and think I'd like to move that direction for all of our scripts. Right now I keep an environment.rb file with my code (watir-webdriver, cucumber) which tells the script which environment we're testing and which browser to use (global variables). Jenkins fires off the script using rake.
I'd love to let the user choose the environment and browser through Jenkins 'choice' variable or similar, and then pass that to the script. While I see the framework in that for Jenkins and set up a choice list for environment, I'm having trouble determining what the next step is.
I could write to environment.rb, I could pass a variable to rake - I have many options for how to pass the information, I just need some assistance finding the first step to find the Jenkins way of accomplishing them. Google results and previous Stack questions weren't what I was looking for.
Thanks
Sure. Give the user either a text entry field a dropdown after telling Jenkins that this is a parameterized build. You'll give them a name, something like BuildEnvironment. Then when you call the build, you can pass these from the environment variables. For example, if you were using ANT, you'd add a line to the parameters that said environment = ${MyEnvironment} Jenkins will then pass the value along for your build tool to use.
There is a way to pass Jenkins Environment Variable to Ruby script. Please see the following example:
workspace_path = `echo $WORKSPACE`.strip # note the use of backticks
puts workspace_path
In the "echo $WORKSPACE".strip # the code work only if you replace quotes with backticks
This code example works in Jenkins on a Linux system.