I invoked GNU parallel (on OS X Yosemite, installed using MacPorts, shell is bash 3.2.57) like this:
parallel mycommand -o A=5 -o ::: Y=1 Y=2
with the intent that it would run the following commands, in parallel:
mycommand -o A=5 -o Y=1
mycommand -o A=5 -o Y=2
But it actually runs this:
mycommand -o A=5 -o Y\=1
mycommand -o A=5 -o Y\=2
The backslash causes mycommand not to recognize that argument. This is a problem. And even after scanning the man page and reading the section of the tutorial on quoting, I can't figure out any way to get parallel to run the commands without the backslash getting in there. I've tried putting the Y= options in a file, I've tried single and double quotes with various levels of nesting, but the output of parallel --dry-run always shows Y\=. Is there some way I can get the backslash out?
This should do the trick:
parallel eval mycommand -o A=5 -o ::: Y=1 Y=2
Related
I have a wrapper script for compiling command to count md5sum of executed command and time it (also some ither stuff). Point is I have to calculate md5sum inside wrapper script.
problem is that m5sum return same output for gcc main.c and gcc "main.c" but command is different.
Here is simple code.
$ cat sc.sh
#!/usr/bin/env bash
cmd="$#"
cmdh=$(echo "$cmd" | md5sum - | cut -f1 -d" ")
echo "CMD ${cmd}"
echo "MD5 ${cmdh}"
time $#
Here is one output:
$ ./sc.sh gcc -c main.c -o out
CMD gcc -c main.c -o out
MD5 b671a0f3b1235aa91e5f86011449c698
real 0m0.019s
user 0m0.009s
sys 0m0.010s
Here is second. I would like to have diffrent md5sum.
$ ./sc.sh gcc -c "main.c" -o out
CMD gcc -c main.c -o out
MD5 b671a0f3b1235aa91e5f86011449c698
real 0m0.017s
user 0m0.007s
sys 0m0.011s
Like here:
$ echo 'gcc -c "main.c" -o out' | md5sum - | cut -f1 -d" "
94d2bafbec690940d1b908678e9c9b7d
$ echo 'gcc -c main.c -o out' | md5sum - | cut -f1 -d" "
b671a0f3b1235aa91e5f86011449c698
Is such thing possible with bash? It would be awesome to not have it bound to any specific bash version, but if there is no other choice, then its also good.
Removing quotes when parsing the line that you have typed into the terminal is part of how the shell works. The commands you are typing are the same. Research how shell works and re-research a basic introduction to shell quoting.
Like here:
Then pass it "like here". Clearly ' quotes are missing from your commands, but they are present in the "like here" snippet.
$ ./sc.sh gcc -c main.c -o out
is exactly the same as
$ ./sc.sh gcc -c "main.c" -o out
is exactly the same as
$ ./sc.sh 'g''c''c' "-""c" 'main.c' '''''-o' 'ou't
and it happens to work the same way as the following, only because of your IFS and how you are using ="$#". Research what $# does and research IFS:
$ ./sc.sh 'gcc -c main.c -o out'
But the following command is different - the double quotes inside single quotes are preserved.
$ ./sc.sh 'gcc -c "main.c" -o out'
As a follow-up, research word splitting. Remember to check your scripts with https://shellcheck.net
Inside script gcc main.c and gcc "main.c" are the same command.
$0 = gcc and $1 = main.c in both variants.
You cannot see the difference internally, and the script cannot make different signs, so you have no reason to see that.
I want to use pgrep to find the pid of a process from its command-line. In the shell, this is done as so:
pgrep -u andrew -fx 'some_binary -c some_config.cfg'
But when I try this from Tcl, like this:
exec pgrep -u $user -fx $cmdLine
I get:
pgrep: invalid option -- 'c'
Which makes sense, because it's seeing this:
pgrep -u andrew -fx some_binary -c some_config.cfg
But it's the same when I add single quotes:
exec pgrep -u $user -fx '$cmdLine'
And that also makes sense, because single quotes aren't special to Tcl. I think it's consider 'some_binary an argument, then the -c, then the some_config.cfg'.
I've also tried:
exec pgrep -u $user -fx {$cmdLine}
and
set cmd "pgrep -u $user -fx '$cmdLine'"
eval exec $cmd
to no avail.
From my reading it seems the {*} feature in Tcl 8.5+ might help, but my company infrastructure runs Tcl 8.0.5.
The problem is partially that ' means nothing at all to Tcl, and partially that you're losing control of where the word boundaries are.
Firstly, double check that this actually works:
exec pgrep -u $user -fx "some_binary -c some_config.cfg"
or perhaps this (Tcl uses {…} like Unix shells use single quotes but with the added benefit of being nestable; that's what braces really do in Tcl):
exec pgrep -u $user -fx {some_binary -c some_config.cfg}
What ought to work is this:
set cmdLine "some_binary -c some_config.cfg"
exec pgrep -u $user -fx $cmdLine
where you have set cmdLine to exactly the characters that you want to have in it (check by printing out if you're unsure; what matters is the value in the variable, not the quoted version that you write in your script). I'll use the set cmdLine "…" form below, but really use whatever you need for things to work.
Now, if you are going to be passing this past eval, then you should use list to add in the extra quotes that you need to make things safe:
set cmdLine "some_binary -c some_config.cfg"
set cmd [list pgrep -u $user -fx $cmdLine]
eval exec $cmd
The list command produces lists, but it uses a canonical form that is also a script fragment that is guaranteed to lack “surprise” substitutions or word boundaries.
If you were on a more recent version of Tcl (specifically 8.5 or later), you'd be able to use expansion. That's designed to specifically work very well with list, and gets rid of the need to use eval in about 99% of all cases. That'd change the:
eval exec $cmd
into:
exec {*}$cmd
The semantics are a bit different except when cmd is holding a canonical list, when they actually run the same operation. (The differences come when you deal with non-canonical lists, where eval would do all sorts of things — imagine the havoc with set cmd {ab [exit] cd}, which is a valid but non-canonical list — whereas expansion just forces things to be a list and uses the words in the list without further interpretation.)
Since you are on a old version, you have to make sure that what eval sees will be converted to a properly quoted Tcl string.
Single quotes do nothing. They are not used by exec, nor are they passed on. exec utilizes the underlying exec(3) system call, and no argument interpretation will take place unless you purposely use something like: /bin/sh -c "some-cmd some-arg" where the shell is invoked and will reinterpret the command line.
What you have to do is construct a string that eval will interpret as a quoted Tcl string. You can use "{part1 part2}" or "\"part1 part2\"" for these constructs.
First, a little test script to verify that the arguments are being passed correctly:
#!/bin/bash
for i in "$#"; do
echo $i
done
Then the Tcl script:
#!/usr/bin/tclsh
exec ./t.sh -u andrew -fx "some_binary -c some_config.cfg" ># stdout
eval exec ./t.sh -u andrew -fx "{some_binary -c some_config.cfg}" \
># stdout
eval exec ./t.sh -u andrew -fx "\"some_binary -c some_config.cfg\"" \
># stdout
# the list will be converted to a string that is already properly
# quoted for interpretation by eval.
set cmd [list ./t.sh -u andrew -fx "some_binary -c some_config.cfg"]
eval exec $cmd ># stdout
I have read several threads how to delay a variable expansion by using the eval command, for example but this does not work in my example.
I want to run the following command:
run_on_compute_farm "/usr/bin/time $MAKE -j$(grep -c ^processor /proc/cpuinfo) all"
As you can see the command is to be executed on a compute farm and thus on another machine. The problem here is that I want to set the number of jobs,
-j$(grep -c ^processor /proc/cpuinfo)
But this will evaluate to the number of cpu cores on the machinve that I am sitting on, not on the target.
Is it even possible in bash to delay the evaluation of the variable or the command? Note that run_on_compute_farm command will evaluate and run the string command that it received as argument.
As Cyrus said, you can replace " with ' which will prevent variable expansion. However, if you require some variable expansion, or multiple levels of expansion, then this will not be the right solution.
If in your example you had wanted to expand $MAKE, but not perform the grep, then the answer I think, is to use \ to escape the $. So,
run_on_compute_farm "/usr/bin/time $MAKE -j\$(grep -c ^processor /proc/cpuinfo) all"
will allow $MAKE to expand on the local machine, but leave the grep for the remote machine to resolve.
You can even nest this strategy, so for example let's get crazy and try this:
run_on_compute_farm "run_on_compute_farm \"/usr/bin/time \$MAKE -j\\\$(grep -c ^processor /proc/cpuinfo) all\""
Notice that some " have been quoted (with \) and I've also sent a quoted backslash over as \\. So, on the first remote machine the command executed will be:
run_on_compute_farm "/usr/bin/time $MAKE -j\$(grep -c ^processor /proc/cpuinfo) all"
So, $MAKE will be resolved on the first remote, but the grep will is still quoted. This then gets sent to a second machine and becomes:
/usr/bin/time $MAKE -j$(grep -c ^processor /proc/cpuinfo) all
This question already has answers here:
Why does shell ignore quoting characters in arguments passed to it through variables? [duplicate]
(3 answers)
Closed 6 years ago.
I'm trying to write a database call from within a bash script and I'm having problems with a sub-shell stripping my quotes away.
This is the bones of what I am doing.
#---------------------------------------------
#! /bin/bash
export COMMAND='psql ${DB_NAME} -F , -t --no-align -c "${SQL}" -o ${EXPORT_FILE} 2>&1'
PSQL_RETURN=`${COMMAND}`
#---------------------------------------------
If I use an 'echo' to print out the ${COMMAND} variable the output looks fine:
echo ${COMMAND}
screen output:-
#---------------
psql drupal7 -F , -t --no-align -c "SELECT DISTINCT hostname FROM accesslog;" -o /DRUPAL/INTERFACES/EXPORTS/ip_list.dat 2>&1
#---------------
Also if I cut and paste this screen output it executes just fine.
However, when I try to execute the command as a variable within a sub-shell call, it gives an error message.
The error is from the psql client to the effect that the quotes have been removed from around the ${SQL} string.
The error suggests psql is trying to interpret the terms in the sql string as parameters.
So it seems the string and quotes are composed correctly but the quotes around the ${SQL} variable/string are being interpreted by the sub-shell during the execution call from the main script.
I've tried to escape them using various methods: \", \\", \\\", "", \"" '"', \'"\', ... ...
As you can see from my 'try it all' approach I am no expert and it's driving me mad.
Any help would be greatly appreciated.
Charlie101
Instead of storing command in a string var better to use BASH array here:
cmd=(psql ${DB_NAME} -F , -t --no-align -c "${SQL}" -o "${EXPORT_FILE}")
PSQL_RETURN=$( "${cmd[#]}" 2>&1 )
Rather than evaluating the contents of a string, why not use a function?
call_psql() {
# optional, if variables are already defined in global scope
DB_NAME="$1"
SQL="$2"
EXPORT_FILE="$3"
psql "$DB_NAME" -F , -t --no-align -c "$SQL" -o "$EXPORT_FILE" 2>&1
}
then you can just call your function like:
PSQL_RETURN=$(call_psql "$DB_NAME" "$SQL" "$EXPORT_FILE")
It's entirely up to you how elaborate you make the function. You might like to check for the correct number of arguments (using something like (( $# == 3 ))) before calling the psql command.
Alternatively, perhaps you'd prefer just to make it as short as possible:
call_psql() { psql "$1" -F , -t --no-align -c "$2" -o "$3" 2>&1; }
In order to capture the command that is being executed for debugging purposes, you can use set -x in your script. This will the contents of the function including the expanded variables when the function (or any other command) is called. You can switch this behaviour off using set +x, or if you want it on for the whole duration of the script you can change the shebang to #!/bin/bash -x. This saves you explicitly echoing throughout your script to find out what commands are being run; you can just turn on set -x for a section.
A very simple example script using the shebang method:
#!/bin/bash -x
ec() {
echo "$1"
}
var=$(ec 2)
Running this script, either directly after making it executable or calling it with bash -x, gives:
++ ec 2
++ echo 2
+ var=2
Removing the -x from the shebang or the invocation results in the script running silently.
Is there some similar option in dash shell corresponding to pipefail in bash?
Or any other way of getting a non-zero status if one of the commands in pipe fail (but not exiting on it which set -e would).
To make it clearer, here is an example of what I want to achieve:
In a sample debugging makefile, my rule looks like this:
set -o pipefail; gcc -Wall $$f.c -o $$f 2>&1 | tee err; if [ $$? -ne 0 ]; then vim -o $$f.c err; ./$$f; fi;
Basically it runs opens the error file and source file on error and runs the programs when there is no error. Saves me some typing. Above snippet works well on bash but my newer Ubunty system uses dash which doesn't seem to support pipefail option.
I basically want a FAILURE status if the first part of the below group of commands fail:
gcc -Wall $$f.c -o $$f 2>&1 | tee err
so that I can use that for the if statement.
Are there any alternate ways of achieving it?
Thanks!
I ran into this same issue and the bash options of set -o pipefail and ${PIPESTATUS[0]} both failed in the dash shell (/bin/sh) on the docker image I'm using. I'd rather not modify the image or install another package, but the good news is that using a named pipe worked perfectly for me =)
mkfifo named_pipe
tee err < named_pipe &
gcc -Wall $$f.c -o $$f > named_pipe 2>&1
echo $?
See this answer for where I found the info: https://stackoverflow.com/a/1221844/431296
The Q.'s sample problem requires:
I basically want a FAILURE status if the first part of the ... group of commands fail:
Install moreutils, and try the mispipe util, which returns the exit status of the first command in a pipe:
sudo apt install moreutils
Then:
if mispipe "gcc -Wall $$f.c -o $$f 2>&1" "tee err" ; then \
./$$f
else
vim -o $$f.c err
fi
While 'mispipe' does the job here, it is not an exact duplicate of the bash shell's pipefail; from man mispipe:
Note that some shells, notably bash, do offer a
pipefail option, however, that option does not
behave the same since it makes a failure of any
command in the pipeline be returned, not just the
exit status of the first.