I have a simple script that i need for my pipeline :-
sh '''podname=$(kubectl get pods -n my-namespace --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep my-pod)
echo "my name is $podname"'''
All i need to invoke through my jenkins declarative pipeline is the value of that kubectl command that i can use later in my script. If i run the same directly on the linux server it works just fine, but somehow through groovy shell invocation always results in syntax errors with respect to unterminated quotes, illegal dollar literal, etc.
How do i fix this?
The syntax looks fine to me, except for this, in the middle of the command:
"\n"
This will be interpreted as an actual new-line, which may not be what you want? Perhaps try this instead, to let the \n sequence be passed to the command:
"\\n"
If that still doesn't help...
according to the docs:
Runs a Bourne shell script, typically on a Unix node. Multiple lines are accepted.
An interpreter selector may be used, for example: #!/usr/bin/perl
So another suggestion is to try adding the appropriate shebang to your script... as there's no reason the result should be different from when you run this on the shell.
Related
My company has a tool that dynamically generates commands to run based on an input json. It works very well when all arguments to the compiled command are single words, but is failing when we attempt multi word args. Here is the minimal example of how it fails.
# Print and execute the command.
print_and_run() { local command=("$#")
if [[ ${command[0]} == "time" ]]; then
echo "Your command: time ${command[#]:1}"
time ${command[#]:1}
fi
}
# How print_and_run is called in the script
print_and_run time docker run our-conainer:latest $generated_flags
# Output
Your command: time docker run our-container:latest subcommand --arg1=val1 --arg2="val2 val3"
Usage: our-program [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...
Try 'our-program --help' for help.
Error: No such command 'val3"'.
But if I copy the printed command and run it myself it works fine (I've omitted docker flags). Shelling into the container and running the program directly with these arguments works as well, so the parsing logic there is solid (It's a python program that uses click to parse the args).
Now, I have a working solution that uses eval, but my entire team jumped down my throat at that suggestion. I've also proposed a solution using delineating characters for multi-word arguments, but that was shot down as well.
No other solutions proposed by other engineers have worked either. So can I ask someone to perhaps explain why val3 is being treated as a separate command, or to help me find a solution to get bash to properly evaluate the dynamically determined command without using eval?
Your command after expanding $generated_flags is:
print_and_run time docker run our-conainer:latest subcommand --arg1=val1 --arg2="val2 val3"
Your specific problem is that in --arg2="val2 val3" the quotes are literal, not syntactical, because quotes are processed before variables are expanded. This means --arg2="val2 and val3" are being split into two separate arguments. Then, I assume, docker is trying to interpret val3" as some kind of docker command because it's not part of any argument, and it's throwing out an error because it doesn't know what that means.
Normally you'd fix this via an array to properly maintain the string boundary.
generated_flags=( "subcommand" "--arg1=val1" "--arg2=val2 val3" )
print_and_run time docker run our-container:latest "${generated_flags[#]}"
This will maintain --arg2=val2 val3 as a single argument as it gets passed into print_and_run, then you just have to expand your command array correctly inside the function (make sure to quote the expansion).
The question is:
why val3 is being treated as a separate command
Unquoted variable expansion undergo word splitting and filename expansion. Word splitting splits the result of the variable expansion on spcaes, tabs and newlines. Splits it into separate "words".
a="something else"
$a # results in two "words"; 'something' and 'else'
It is irrelevent what you put inside the variable value or how many quotes or escape sequences you put inside. Every consecutive spaces splits it into words. Quotes " ' and escapes \ are parsed when part of the input line, not when part of the result of unquoted expansion.
help me find a solution to
Write a parser that will actually parse the commands and split it according to the rules that you want to use and then execute the command split into separate words. For example, a very crude such parser is included in xargs:
$ echo " 'quotes quotes' not quotes" | xargs printf "'%s'\n"
'quotes quotes'
'not'
'quotes'
For example, python has shlex.split which you can just use, and at the same time introduce python which is waaaaay easier to manage than badly written Bash scripts.
tool that dynamically generates commands to run based on an input json
Overall, the proper way forward would is to upgrade the tool to generate a JSON array that represents the words of the command to be executed. Than you can just execute that array of words, which is, again, trivial to do properly in python with json and subprocess.run, and will require some gymnastics with jq and read and Bash arrays in shell.
Check your scripts with shellcheck.
Currently trying to move all of my aliases from .bash_profile to .zshrc. However, found a problem with one of the longer aliases I use for substituting root to ubuntu when passing a command to access AWS instances.
AWS (){
cd /Users/user/aws_keys
cmd=$(echo $# | sed "s/root/ubuntu/g")
$cmd[#]
}
The error I get is AWS:5: command not found ssh -i keypair.pem ubuntu#ec1.compute.amazonaws.com
I would really appreciate any suggestions!
The basic problem is that the cmd=$(echo ... line is mashing all the arguments together into a space-delimited string, and you're depending on word-splitting to split it up into a command and its arguments. But word-splitting is usually more of a problem than anything else, so zsh doesn't do it by default. This means that rather than trying to run the command named ssh with arguments -i, keypair.pem, etc, it's treating the entire string as the command name.
The simple solution is to avoid mashing the arguments together, so you don't need word-splitting to separate them out again. You can use a modifier to the parameter expansion to replace "root" with "ubuntu". BTW, I also strongly recommend checking for error when using cd, and not proceeding if it gets an error.
So something like this:
AWS (){
cd /Users/user/aws_keys || return $?
"${#//root/ubuntu}"
}
This syntax will work in bash as well as zsh (the double-quotes prevent unexpected word-splitting in bash, and aren't really needed in zsh).
BTW, I'm also a bit nervous about just blindly replacing "root" with "ubuntu" in the arguments; what if it occurs somewhere other than the username, like as part of a filename or hostname?
When posting this question originally, I totally misworded it, obtaining another, reasonable but different question, which was correctly answered here.
The following is the correct version of the question I originally wanted to ask.
In one of my Bash scripts, there's a point where I have a variable SCRIPT which contains the /path/to/an/exe which, when executed, outputs a line to be executed.
What my script ultimately needs to do, is executing that line to be executed. Therefore the last line of the script is
$($SCRIPT)
so that $SCRIPT is expanded to /path/to/an/exe, and $(/path/to/an/exe) executes the executable and gives back the line to be executed, which is then executed.
However, running shellcheck on the script generates this error:
In setscreens.sh line 7:
$($SCRIPT)
^--------^ SC2091: Remove surrounding $() to avoid executing output.
For more information:
https://www.shellcheck.net/wiki/SC2091 -- Remove surrounding $() to avoid e...
Is there a way I can rewrite that $($SCRIPT) in a more appropriate way? eval does not seem to be of much help here.
If the script outputs a shell command line to execute, the correct way to do that is:
eval "$("$SCRIPT")"
$($SCRIPT) would only happen to work if the command can be completely evaluated using nothing but word splitting and pathname expansion, which is generally a rare situation. If the program instead outputs e.g. grep "Hello World" or cmd > file.txt then you will need eval or equivalent.
You can make it simple by setting the command to be executed as a positional argument in your shell and execute it from the command line
set -- "$SCRIPT"
and now run the result that is obtained by expansion of SCRIPT, by doing below on command-line.
"$#"
This works in case your output from SCRIPT contains multiple words e.g. custom flags that needs to be run. Since this is run in your current interactive shell, ensure the command to be run is not vulnerable to code injection. You could take one step of caution and run your command within a sub-shell, to not let your parent environment be affected by doing ( "$#" ; )
Or use shellcheck disable=SCnnnn to disable the warning and take the occasion to comment on the explicit intention, rather than evade the detection by cloaking behind an intermediate variable or arguments array.
#!/usr/bin/env bash
# shellcheck disable=SC2091 # Intentional execution of the output
"$("$SCRIPT")"
By disabling shellcheck with a comment, it clarifies the intent and tells the questionable code is not an error, but an informed implementation design choice.
you can do it in 2 steps
command_from_SCRIPT=$($SCRIPT)
$command_from_SCRIPT
and it's clean in shellcheck
I am fairly new to bash and have been asked to develop a pipeline process for use on gitlab. I have simplified my variable notation below. My problem is that a command that runs fine locally will not run correctly on the server.
I have a dynamically generated $command_string that looks something like this:
command_string="cf service-key ${val1} ${val2}"
Locally, I can run
new_var=$(eval ${command_string})
But when this command is submit to the server and run as a pipeline, it breaks.
I should note that when I remove the new_var definition as the output of the eval function, it seems to run. The problem is that I need the output information so that I can extract information from the output.
Additionally, I have tried new_var=$(bash -c ${command_string}) but this returns a value as if I had submit only the first "cf" portion of $command_string (i.e. it returns the help menu of the cf function rather than the output of the provided arguments).
As I said, bash is a new language for me so I am sure I am missing something fundamental here.
${val1} and ${val2} are expanded at the time of definition, not evaluation, which means that the use of command_string will not be "dynamic" as you intend.
Consider defining a function as follows:
my_command ()
{
cf service-key "$1" "$2"
}
and using the function within your code as
new_var=$(my_command "$val1" "$val2")
. The eval is unnecessary since the $(...) is a command substitution which evaluates the contents of the enclosed command.
CMake's execute_process command seems to only let you, well, execute a process - not an arbitrary line you could feed a command shell. The thing is, I want to use pipes, file descriptor redirection, etc. - and that does not seem to be possible. The alternative would be very painful for me (I think)...
What should I do?
PS - CMake 2.8 and 3.x answer(s) are interesting.
You can execute any shell script, using your shell's support for taking in a script within a string argument.
Example:
execute_process(
COMMAND bash "-c" "echo -n hello | sed 's/hello/world/;'"
OUTPUT_VARIABLE FOO
)
will result in FOO containing world.
Of course, you would need to escape quotes and backslashes with care. Also remember that running bash would only work on platforms which have bash - i.e. it won't work on Windows.
execute_process command seems to only let you, well, execute a process - not an arbitrary line you could feed a command shell.
Yes, exactly this is written in documentation for that command:
All arguments are passed VERBATIM to the child process. No intermediate shell is used, so shell operators such as > are treated as normal arguments.
I want to use pipes
Different COMMAND within same execute_process invocation are actually piped:
Runs the given sequence of one or more commands with the standard output of each process piped to the standard input of the next.
file descriptor redirection, etc. - and that does not seem to be possible.
For complex things just prepare separate shell script and run it using execute_process. You can pass variables from CMake to this script using its parameters, or with prelimiary configure_file.
I needed to pipe two commands one after the other and actually learned that each COMMAND of the execute_process is piped already. So at least that much is resolved by simply adding commands one after the other:
execute_process(
COMMAND echo "Hello"
COMMAND sed -e 's/H/h/'
OUTPUT_VARIABLE GREETINGS
OUTPUT_STRIP_TRAILING_WHITESPACE)
Now the variable GREETINGS is set to hello.
If you indeed need a lot of file redirection (as you stated), you probably want to write an external script and then execute that script from CMakeLists.txt. It's really difficult to get all the escaping right in CMake.
If you can simplify your scripts to one command generating a file, then another handling that file, etc. then you can always use the INPUT_FILE and OUTPUT_FILE options. Or pass a filename to your command for the input.
It's often much cleaner to handle one file at a time. Although I understand that some commands may need multiple sources and destinations.