Ampersand to run process in background causes invalid parameters in Bash 5 - bash

I have a bash script that runs perfectly well on Bash 3.2. The script contains an ampersand to run a process in the background. However, when I run it in Bash 5.x, it doesn't pass the variables correctly (I get a "SyntaxError: Unexpected end of JSON input"). When I take off the ampersand at the end (of the mgeneratejs line), it executes normally in Bash 5.
#!/bin/bash
#Works on Bash 3.2 on MacOS
#Doesn't work in bash-5.0/5.1
##!/usr/bin/env bash
NUM_ROWS_PER_RUN=5
NUM_RUNS=2
TEMPLATE_STRING='{
name: "$name"
}'
for i in $(seq 1 "$NUM_RUNS")
do
echo "Starting run ${i}"
#If you dont have it, then run "npm install -g mgeneratejs"
mgeneratejs -n "$NUM_ROWS_PER_RUN" "${TEMPLATE_STRING//[$'\r\n ']}" &
done
echo "Waiting"
wait
echo "Finished"
How can I get the process (mgeneratejs) to run in the background when using Bash 5.x?

Bash may or may not be in fault here, but be sure that the problema is in mgeneratejs.
Taking a look mgeneratejs's source code I found this:
if (process.stdin.isTTY) {
var str = argv._[0];
template = _.startsWith(str, '{') ? parseTemplate(str) : parseTemplate(read(str, 'utf8'));
generate();
} else {
template = '';
process.stdin.setEncoding('utf-8');
process.stdin.on('readable', function() {
var chunk = process.stdin.read();
if (chunk !== null) {
template += chunk;
}
});
process.stdin.on('end', function() {
template = JSON.parse(template);
generate();
});
}
If stdin is not a TTY then mgeneratejs assumes that stdin is a pipe, and tries to read from it. This is wrong, they should at least check if the template has been given in the command line args.
I would't recommend that you fix mgeneratejs, but I can recommend you to do this:
function do_run() {
echo "${TEMPLATE_STRING//[$'\r\n ']}" | mgeneratejs -n "$NUM_ROWS_PER_RUN"
}
for i in $(seq 1 "$NUM_RUNS")
do
echo "Starting run ${i}"
#If you dont have it, then run "npm install -g mgeneratejs"
do_run &
done

Related

How to return output of shell script into Jenkinsfile [duplicate]

I have something like this on a Jenkinsfile (Groovy) and I want to record the stdout and the exit code in a variable in order to use the information later.
sh "ls -l"
How can I do this, especially as it seems that you cannot really run any kind of groovy code inside the Jenkinsfile?
The latest version of the pipeline sh step allows you to do the following;
// Git committer email
GIT_COMMIT_EMAIL = sh (
script: 'git --no-pager show -s --format=\'%ae\'',
returnStdout: true
).trim()
echo "Git committer email: ${GIT_COMMIT_EMAIL}"
Another feature is the returnStatus option.
// Test commit message for flags
BUILD_FULL = sh (
script: "git log -1 --pretty=%B | grep '\\[jenkins-full]'",
returnStatus: true
) == 0
echo "Build full flag: ${BUILD_FULL}"
These options where added based on this issue.
See official documentation for the sh command.
For declarative pipelines (see comments), you need to wrap code into script step:
script {
GIT_COMMIT_EMAIL = sh (
script: 'git --no-pager show -s --format=\'%ae\'',
returnStdout: true
).trim()
echo "Git committer email: ${GIT_COMMIT_EMAIL}"
}
Current Pipeline version natively supports returnStdout and returnStatus, which make it possible to get output or status from sh/bat steps.
An example:
def ret = sh(script: 'uname', returnStdout: true)
println ret
An official documentation.
quick answer is this:
sh "ls -l > commandResult"
result = readFile('commandResult').trim()
I think there exist a feature request to be able to get the result of sh step, but as far as I know, currently there is no other option.
EDIT: JENKINS-26133
EDIT2: Not quite sure since what version, but sh/bat steps now can return the std output, simply:
def output = sh returnStdout: true, script: 'ls -l'
If you want to get the stdout AND know whether the command succeeded or not, just use returnStdout and wrap it in an exception handler:
scripted pipeline
try {
// Fails with non-zero exit if dir1 does not exist
def dir1 = sh(script:'ls -la dir1', returnStdout:true).trim()
} catch (Exception ex) {
println("Unable to read dir1: ${ex}")
}
output:
[Pipeline] sh
[Test-Pipeline] Running shell script
+ ls -la dir1
ls: cannot access dir1: No such file or directory
[Pipeline] echo
unable to read dir1: hudson.AbortException: script returned exit code 2
Unfortunately hudson.AbortException is missing any useful method to obtain that exit status, so if the actual value is required you'd need to parse it out of the message (ugh!)
Contrary to the Javadoc https://javadoc.jenkins-ci.org/hudson/AbortException.html the build is not failed when this exception is caught. It fails when it's not caught!
Update:
If you also want the STDERR output from the shell command, Jenkins unfortunately fails to properly support that common use-case. A 2017 ticket JENKINS-44930 is stuck in a state of opinionated ping-pong whilst making no progress towards a solution - please consider adding your upvote to it.
As to a solution now, there could be a couple of possible approaches:
a) Redirect STDERR to STDOUT 2>&1
- but it's then up to you to parse that out of the main output though, and you won't get the output if the command failed - because you're in the exception handler.
b) redirect STDERR to a temporary file (the name of which you prepare earlier) 2>filename (but remember to clean up the file afterwards) - ie. main code becomes:
def stderrfile = 'stderr.out'
try {
def dir1 = sh(script:"ls -la dir1 2>${stderrfile}", returnStdout:true).trim()
} catch (Exception ex) {
def errmsg = readFile(stderrfile)
println("Unable to read dir1: ${ex} - ${errmsg}")
}
c) Go the other way, set returnStatus=true instead, dispense with the exception handler and always capture output to a file, ie:
def outfile = 'stdout.out'
def status = sh(script:"ls -la dir1 >${outfile} 2>&1", returnStatus:true)
def output = readFile(outfile).trim()
if (status == 0) {
// output is directory listing from stdout
} else {
// output is error message from stderr
}
Caveat: the above code is Unix/Linux-specific - Windows requires completely different shell commands.
this is a sample case, which will make sense I believe!
node('master'){
stage('stage1'){
def commit = sh (returnStdout: true, script: '''echo hi
echo bye | grep -o "e"
date
echo lol''').split()
echo "${commit[-1]} "
}
}
For those who need to use the output in subsequent shell commands, rather than groovy, something like this example could be done:
stage('Show Files') {
environment {
MY_FILES = sh(script: 'cd mydir && ls -l', returnStdout: true)
}
steps {
sh '''
echo "$MY_FILES"
'''
}
}
I found the examples on code maven to be quite useful.
All the above method will work. but to use the var as env variable inside your code you need to export the var first.
script{
sh " 'shell command here' > command"
command_var = readFile('command').trim()
sh "export command_var=$command_var"
}
replace the shell command with the command of your choice. Now if you are using python code you can just specify os.getenv("command_var") that will return the output of the shell command executed previously.
How to read the shell variable in groovy / how to assign shell return value to groovy variable.
Requirement : Open a text file read the lines using shell and store the value in groovy and get the parameter for each line .
Here , is delimiter
Ex: releaseModule.txt
./APP_TSBASE/app/team/i-home/deployments/ip-cc.war/cs_workflowReport.jar,configurable-wf-report,94,23crb1,artifact
./APP_TSBASE/app/team/i-home/deployments/ip.war/cs_workflowReport.jar,configurable-temppweb-report,394,rvu3crb1,artifact
========================
Here want to get module name 2nd Parameter (configurable-wf-report) , build no 3rd Parameter (94), commit id 4th (23crb1)
def module = sh(script: """awk -F',' '{ print \$2 "," \$3 "," \$4 }' releaseModules.txt | sort -u """, returnStdout: true).trim()
echo module
List lines = module.split( '\n' ).findAll { !it.startsWith( ',' ) }
def buildid
def Modname
lines.each {
List det1 = it.split(',')
buildid=det1[1].trim()
Modname = det1[0].trim()
tag= det1[2].trim()
echo Modname
echo buildid
echo tag
}
If you don't have a single sh command but a block of sh commands, returnstdout wont work then.
I had a similar issue where I applied something which is not a clean way of doing this but eventually it worked and served the purpose.
Solution -
In the shell block , echo the value and add it into some file.
Outside the shell block and inside the script block , read this file ,trim it and assign it to any local/params/environment variable.
example -
steps {
script {
sh '''
echo $PATH>path.txt
// I am using '>' because I want to create a new file every time to get the newest value of PATH
'''
path = readFile(file: 'path.txt')
path = path.trim() //local groovy variable assignment
//One can assign these values to env and params as below -
env.PATH = path //if you want to assign it to env var
params.PATH = path //if you want to assign it to params var
}
}
Easiest way is use this way
my_var=`echo 2`
echo $my_var
output
: 2
note that is not simple single quote is back quote ( ` ).

How to make errexit behaviour work in bash functions

How do I get -e / errexit to work in bash functions, so that the first failed command* within a function causes the function to return with an error code (just as -e works at top-level).
* not part of boolean expression, if/elif/while/etc etc etc
I ran the following test-script, and I expect any function containing f in its name (i.e. a false line in its source) to return error-code, but they don't if there's another command after. Even when I put the function body in a subshell with set -e specified again, it just blindly steamrolls on through the function after a command fails instead of exiting the subshell/function with the nonzero status code.
Environments / interpreters tested:
I get the same result in all of these envs/shells, which is to be expected I guess unless one was buggy.
Arch:
bash test.sh (bash 5.1)
zsh test.sh (zsh 5.8)
sh test.sh (just a symlink to bash)
Alpine:
docker run -v /test.sh:/test.sh alpine:latest sh /test.sh (BusyBox 1.34)
Ubuntu:
docker run -v /test.sh:/test.sh ubuntu:21.04 sh /test.sh
docker run -v /test.sh:/test.sh ubuntu:21.04 bash /test.sh (bash 5.1)
docker run -v /test.sh:/test.sh ubuntu:21.04 dash /test.sh (bash 5.1)
Test script
set -e
t() {
true
}
f() {
false
}
tf() {
true
false
}
ft() {
false
true
}
et() {
set -e
true
}
ef() {
set -e
false
}
etf() {
set -e
true
false
}
eft() {
set -e
false
true
}
st() {( set -e
true
)}
sf() {( set -e
false
)}
stf() {( set -e
true
false
)}
sft() {( set -e
false
true
)}
for test in t f tf ft _ et ef etf eft _ st sf stf sft; do
if [ "$test" = '_' ]; then
echo ""
elif "$test"; then
echo "$test: pass"
else
echo "$test: fail"
fi
done
Output on my machine
t: pass
f: fail
tf: fail
ft: pass
et: pass
ef: fail
etf: fail
eft: pass
st: pass
sf: fail
stf: fail
sft: pass
Desired output
Without significantly changing source in functions themselves, i.e. not adding an if/then or || return to every line in the function.
t: pass
f: fail
tf: fail
ft: fail
et: pass
ef: fail
etf: fail
eft: fail
st: pass
sf: fail
stf: fail
sft: fail
Or at least pass/fail/fail/fail in one group, so I can use that approach for writing robust functions.
With accepted solution
With the accepted solution, the first four tests give the desired result. The other two groups don't, but are irrelevant since those were my own attempts to work around the issue. The root cause is that the "don't errexit the script when if/while-condition fails" behaviour propagates into any functions called in the condition, rather than just applying to the end-result.
How to make errexit behaviour work in bash functions
Call the function not inside if.
I expect any function containing f in its name (i.e. a false line in its source) to return error-code
Weeeeeell, your expectancy does not match reality. It is not how it is implemented. And flag errexit will exit your script, not return error-code.
Desired output
You can:
Download Bash or other shell sources or write your own and implement the behavior that you want to have.
consider extending the behavior with like shopt -s errexit_but_return_nonzero_in_if_contexts or something shorter
Run it in a separate shell.
elif bash -ec "$(declare -f "$test"); $test"; then
Write a custom loadable to "clear" CMD_IGNORE_RETURN flag when Bash enters contexts in which set -e should be ignored. I think for that static COMMAND *currently_executing_command; needs to be extern and then just currently_executing_command.flags &= ~CMD_IGNORE_RETURN;.
You can do a wrapper where you temporarily disable errexit and get return status, but you have to call it outside of if:
errreturn() {
declare -g ERRRETURN
local flags=$(set +o)
set +e
(
set -e
"$#"
)
ERRRETURN=$?
eval "$flags"
}
....
echo ""
else
errreturn "$test"
if (( ERRRETURN == 0 )); then
echo "$test: pass"
else
echo "$test: fail"
fi
fi

capture all bash commands as parameters for a custom runner

I have a custom 'runner'-script that I need to use to run all of my terminal commands. Below you can see the general idea in the script.
#!/usr/bin/env bash
echo "Running '$#'"
# do stuff before running cmd
$#
echo "Done"
# do stuff after running cmd
I can use the script in bash as follows:
$ ./run.sh echo test
Running 'echo test'
test
Done
$
I would like to use it like this:
$ echo test
Running 'echo test'
test
Done
$
Bash has the trap ... DEBUG and PROMPT_COMMAND, which lets me execute something before and after a command, but is there something that would allow me to execute instead of the command?
There is also the command_not_found_handle which would work if I had an empty PATH env variable, but that seems too dirty.
After some digging, I ended up looking at the source code and found that bash does not support custom executors. Below is a patch to add a new handle that works similarly as the command_not_found_handler.
diff --git a/eval.c b/eval.c
index f02d6e40..8d32fafa 100644
--- a/eval.c
+++ b/eval.c
## -52,6 +52,10 ##
extern sigset_t top_level_mask;
#endif
+#ifndef EXEC_HOOK
+# define EXEC_HOOK "command_exec_handle"
+#endif
+
static void send_pwd_to_eterm __P((void));
static sighandler alrm_catcher __P((int));
## -172,7 +176,15 ## reader_loop ()
executing = 1;
stdin_redir = 0;
- execute_command (current_command);
+ SHELL_VAR *hookf = find_function (EXEC_HOOK);
+ if (hookf == 0) {
+ execute_command (current_command);
+ } else {
+ char *command_to_print = make_command_string (current_command);
+ WORD_LIST *og = make_word_list(make_word(command_to_print), (WORD_LIST *)NULL);
+ WORD_LIST *wl = make_word_list(make_word(EXEC_HOOK), og);
+ execute_shell_function (hookf, wl);
+ }
exec_done:
QUIT;
One can then define function command_exec_handle() { eval $1; } which will be executed instead of the original command given in the prompt. The original command is fully in the first parameter. The command_exec_handle can be given in .bashrc and it works as expected.
Notice: this is very dangerous! If you mess up and put a bad command_exec_handler in your .bashrc, you might end up with a shell that does not execute commands. It will be quite hard to fix without booting from a live cd.
It seems you have the same problem listed here. If you want to run some commands if your original command was not found, the Bash 4's command_not_found_handler will certainly fit your needs.
Try to be more specific, maybe with some code snippets that do or do not work, in order to help us to help you...

Make build UNSTABLE if text found in console log using jenkinsfile (jenkins pipeline)

I am trying to login into an instance and check if the file test.txt is not empty, then echo .. make build unstable using the jenkins pipeline (jenkinsfile)But that's not working.
I have this:
post {
always {
sh "ssh ubuntu#$Ip 'if [ -s test.txt ] ; then echo some text && cat test.txt'"
currentBuild.result = 'UNSTABLE'
}
}
Instead of doing above, can I parse through the console log of the latest build to find something eg: some text and if that's found I want to make the build unstable
You need to return standard out from the script:
String stdOut = sh returnStdout: true, script: "ssh ubuntu#$Ip 'if [ -s test.txt ] ; then echo some text && cat test.txt'"
if (stdOut == "") {
currentBuild.status = 'UNSTABLE'
}
Or, you could use returnStatus to return the exit code of the script. The documentation for the sh step can be found here

Jenkins pipeline undefined variable

I'm trying to build a Jenkins Pipeline for which a parameter is
optional:
parameters {
string(
name:'foo',
defaultValue:'',
description:'foo is foo'
)
}
My purpose is calling a shell script and providing foo as argument:
stages {
stage('something') {
sh "some-script.sh '${params.foo}'"
}
}
The shell script will do the Right Thing™ if the provided value is the empty
string.
Unfortunately I can't just get an empty string. If the user does not provide
a value for foo, Jenkins will set it to null, and I will get null
(as string) inside my command.
I found this related question but the only answer is not really helpful.
Any suggestion?
OP here realized a wrapper script can be helpful… I ironically called it junkins-cmd and I call it like this:
stages {
stage('something') {
sh "junkins-cmd some-script.sh '${params.foo}'"
}
}
Code:
#!/bin/bash
helpme() {
cat <<EOF
Usage: $0 <command> [parameters to command]
This command is a wrapper for jenkins pipeline. It tries to overcome jenkins
idiotic behaviour when calling programs without polluting the remaining part
of the toolkit.
The given command is executed with the fixed version of the given
parameters. Current fixes:
- 'null' is replaced with ''
EOF
} >&2
trap helpme EXIT
command="${1:?Missing command}"; shift
trap - EXIT
typeset -a params
for p in "$#"; do
# Jenkins pipeline uses 'null' when the parameter is undefined.
[[ "$p" = 'null' ]] && p=''
params+=("$p")
done
exec $command "${params[#]}"
Beware: prams+=("$p") seems not to be portable among shells: hence this ugly script is running #!/bin/bash.

Resources