I have a python script p.py which does exit("ABC") for some files. I would like to write a Ubuntu shell to copy the files which make the script exit("ABC") into a folder:
#!/bin/bash
FILES=*.txt
TOOL=p.py
TAREGT=../TARGET/
for f in $FILES
do
if [ $(python $TOOL $f) = "ABC" ]
then
echo "$f"
cp $f $TARGET
fi
done
but the condition check if [ $(python $TOOL $f) = "ABC" ] does not seem to work, it says ./filter.sh: line 13: [: =: unary operator expected. Could anyone tell me what is wrong?
The parameter to exit() is what the Python script returns (success / error). (Documentation of Python's exit(). Note how exit( "ABC" ) doesn't return "ABC", but prints that to stderr and returns 1.)
The return code is what ends up in the $? variable of the calling shell, or what you would test for like this:
# Successful if return code zero, failure otherwise.
# (This is somewhat bass-ackwards when compared to C/C++/Java "if".)
if python $TOOL $f
then
...
fi
The $(...) construct is replaced with the output of the called script / executable, which is a different thing altogether.
And if you're comparing strings, you have to quote them
if [ "$(python $TOOL $f)" = "ABC" ]
or use bash's improved test [[:
if [[ $(python $TOOL $f) = "ABC" ]]
Related
I'd like to add an argument to a command in bash only if a variable evaluates to a certain value. For example this works:
test=1
if [ "${test}" == 1 ]; then
ls -la -R
else
ls -R
fi
The problem with this approach is that I have to duplicate ls -R both when test is 1 or if it's something else. I'd prefer if I could write this in one line instead such as this (pseudo code that doesn't work):
ls (if ${test} == 1 then -la) -R
I've tried the following but it doesn't work:
test=1
ls `if [ $test -eq 1 ]; then -la; fi` -R
This gives me the following error:
./test.sh: line 3: -la: command not found
A more idiomatic version of svlasov's answer:
ls $( (( test == 1 )) && printf %s '-la' ) -R
Since echo understands a few options itself, it's safer to use printf %s to make sure that the text to print is not mistaken for an option.
Note that the command substitution must not be quoted here - which is fine in the case at hand, but calls for a more robust approach in general - see below.
However, in general, the more robust approach is to build up arguments in an array and pass it as a whole:
# Build up array of arguments...
args=()
(( test == 1 )) && args+=( '-la' )
args+=( '-R' )
# ... and pass it to `ls`.
ls "${args[#]}"
Update: The OP asks how to conditionally add an additional, variable-based argument to yield ls -R -la "$PWD".
In that case, the array approach is a must: each argument must become its own array element, which is crucial for supporting arguments that may have embedded whitespace:
(( test == 1 )) && args+= ( '-la' "$PWD" ) # Add each argument as its own array element.
As for why your command,
ls `if [ $test -eq 1 ]; then -la; fi` -R
didn't work:
A command between backticks (or its modern, nestable equivalent, $(...)) - a so-called command substitution - is executed just like any other shell command (albeit in a sub-shell) and the whole construct is replaced with the command's stdout output.
Thus, your command tries to execute the string -la, which fails. To send it to stdout, as is needed here, you must use a command such as echo or printf.
Print the argument with echo:
test=1
ls `if [ $test -eq 1 ]; then echo "-la"; fi` -R
I can't say how acceptable this is, but:
test=1
ls ${test:+'-la'} -R
See https://stackoverflow.com/revisions/16753536/1 for a conditional truth table.
Another answer without using eval and using BASH arrays:
myls() { local arr=(ls); [[ $1 -eq 1 ]] && arr+=(-la); arr+=(-R); "${arr[#]}"; }
Use it as:
myls
myls "$test"
This script builds whole command in an array arr and preserves the original order of command options.
#!/bin/bash
traverse() {
local x=$1
if [ -d $x ]
then
lst=(`ls $x`)
for((i=${#lst[#]}; --i;)); do
echo "${lst[i]}"
done
else echo "not a directory"
fi
}
traverse
I want to pass a parameter such as "/path/to/this/directory/" when executing program but only works if I'm running the program in the same directory as my bash script file and any other parameter I pass is completely ignored.
the script is supposed to take a parameter and check if it's a directory and if it's a directory then list all the files/folders in descending order. If not display error message.
What is wrong with code, thanks!
This happens because $1 in the function refers to traverse's parameters, not your script's parameters.
To run your function once with each argument, use
for arg in "$#" # "$#" is also the default, so you can drop the 'in ..'
do
traverse "$arg"
done
If you in the future want to pass all the script's parameters to a function, use
myfunc "$#"
This is just the problem at hand, though. Other problems include not quoting your variables and using command expansion of ls, lst=(`ls $x`), instead of globs, lst=( "$x"/* )
You don't need to call ls for that. You can use this code:
traverse() {
local x="$1"
if [ -d "$x" ]; then
arr=( "$x/"* )
for ((i=${#arr[#]}; i>0; i--)); do
echo "${arr[$i]}"
done
else
echo "not a directory"
fi
}
"That other guy" has the right answer. The reason it always looks at the current directory:
you invoke traverse with no arguments
the $1 in the traverse function is empty, therefore $x is empty
the test is therefore [ -d ], and when [ is given 1 argument, it returns success if the argument is not empty. Your if command always executes the "true" block and ls $x is just ls when x is empty
Use [[ ... ]] with bash: it is smarter about empty arguments. Otherwise, quote your variables:
$ x=; [ -d $x ] && echo always true || echo not a directory
always true
$ x=; [[ -d $x ]] && echo always true || echo not a directory
not a directory
$ x=; [ -d "$x" ] && echo always true || echo not a directory
not a directory
I often see this construct in sh scripts:
if [ "z$x" = z ]; then echo x is empty; fi
Why don't they just write it like this?
if [ "$x" = "" ]; then echo x is empty; fi
TL;DR short answer
In this construct:
if [ "z$x" = z ]; then echo x is empty; fi
the z is a guard against funny content of $x and many other problems.
If you write it without the z:
if [ "$x" = "" ]; then echo x is empty; fi
and $x contains the string -x you will get:
if [ "-x" = "" ]; then echo x is empty; fi
and that confuses the hell out of some older implementations of [.
If you further omit the quotes around $x and $x contains the string -f foo -o x you will get:
if [ -f foo -o x = "" ]; then echo x is empty; fi
and now it silently checks for something completely different.
the guard will prevent these maybe honest human errors maybe possibly malicious attacks to fall through silently. with the guard you either get the correct result or an error message. read on for an elaborate explanation.
Elaborate explanation
The z in
if [ "z$x" = z ]; then echo x is empty; fi
is called a guard.
To explain why you want the guard I first want to explain the syntax of the bash conditional if. It is important to understand that [ is not part of the syntax. It is a command. It is an alias to the test command. And in most current shells it is a builtin command.
The grammar rule for if is roughly as follows:
if command; then morecommands; else evenmorecommands; fi
(the else part is optional)
command can be any command. Really any command. What bash does when it encounters an if is roughly as follows:
Execute command.
Check the exit status of command.
If exit status is 0 then execute morecommands. If exit status is anything else, and the else part exists, then execute evenmorecommands.
Let's try that:
$ if true; then echo yay; else echo boo; fi
yay
$ if wat; then echo yay; else echo boo; fi
bash: wat: command not found
boo
$ if echo foo; then echo yay; else echo boo; fi
foo
yay
$ if cat foo; then echo yay; else echo boo; fi
cat: foo: No such file or directory
boo
Let's try the test command:
$ if test z = z; then echo yay; else echo boo; fi
yay
And the alias [:
$ if [ z = z ]; then echo yay; else echo boo; fi
yay
You see [ is not part of the syntax. It is just a command.
Note that the z here has no special meaning. It is just a string.
Let's try the [ command outside of an if:
$ [ z = z ]
Nothing happens? It returned an exit status. You can check the exit status with echo $?.
$ [ z = z ]
$ echo $?
0
Let's try unequal strings:
$ [ z = x ]
$ echo $?
1
Because [ is a command it accepts parameters just like any other commands. In fact, the closing ] is also a parameter, a mandatory parameter which must come last. If it is missing the command will complain:
$ [ z = z
bash: [: missing `]'
It is misleading that bash does the complaining. Actually the builtin command [ does the complaining. We can see more clearly who does the complaining when we invoke the system [:
$ /usr/bin/[ z = z
/usr/bin/[: missing `]'
Interestingly the system [ doesn't always insist on a closing ]:
$ /usr/bin/[ --version
[ (GNU coreutils) 7.4
...
You need a space before the closing ] otherwise it will not be recognized as a parameter:
$ [ z = z]
bash: [: missing `]'
You also need a space after the [ otherwise bash will think you want to execute another command:
$ [z = z]
bash: [z: command not found
This is much more obvious when you use test:
$ testz = z
bash: testz: command not found
Remember [ is just another name for test.
[ can do more than just compare strings. It can compare numbers:
$ [ 1 -eq 1 ]
$ [ 42 -gt 0 ]
It can also check for the existence of files or directories:
$ [ -f filename ]
$ [ -d dirname ]
See help [ or man [ for more information about the capabilities of [ (or test). man will show you the documentation for the system command. help will show you the documentation for the bash builtin command.
Now that I have covered the bases I can answer your question:
Why do people write this:
if [ "z$x" = z ]; then echo x is empty; fi
and not this:
if [ "$x" = "" ]; then echo x is empty; fi
For brevity I will strip off the if because this is only about [.
The z in this construct:
[ "z$x" = z ]
is a guard against funny content of $x in combination with older implementations of [, and/or a guard against human error like forgetting to quote $x.
What happens when $x has funny content like -f?
This
[ "$x" = "" ]
will become
[ "-f" = "" ]
Some older implementations of [ will get confused when the first parameter starts with a -. The z will make sure that the first parameter never starts with a - regardless of content of $x.
[ "z$x" = "z" ]
will become
[ "z-f" = "z" ]
What happens when you forgot to quote $x? Funny content like -f foo -o x can change the entire meaning of the test.
[ $x = "" ]
will become
[ -f foo -o x = "" ]
The test is now checking for the existence of the file foo and then logical or with whether x is the empty string. The worst part is that you won't even notice because there is no error message, only an exit status. If $x comes from user input this can even be used for malicious attacks.
With the guarding z
[ z$x = z ]
will become
[ z-f foo -o x = z ]
At least you will now get an error message:
$ [ z-f foo -o x = z ]; echo $?
bash: [: too many arguments
The guard also helps against the case of undefined variable instead of the empty string. Some older shells had different behaviour for undefined variable and empty string. This problem is basically solved because in modern shells undefined mostly behaves like an empty string.
Summary:
The quote around $x helps to make the undefined cases behave more like the empty string cases.
The guard before $x helps to further prevent all the other problems mentioned above.
The guard before $x will prevent all these possible errors:
Funny content of $x (code injection by malicious user)
old implementations of [ (getting confused if string begins with -)
forgetting to quote $x (will allow -f foo -o x to subvert the meaning of the test)
undefined $x. (older implementations behave differently if undefined)
The guard will either do the right thing or raise an error message.
Modern implementations of [ have fixed some of the problems and modern shells have some solutions for the other cases, but they have pitfalls of their own. The guarding z is not necessary if you are otherwise carefull, but it makes avoiding mistakes while writing simple tests so much more simpler.
See also:
bash pitfalls about quoting in tests
bash FAQ more details about test
more about test
more about quoting
"test" operator robustness in various shells
For testing zero length, use -z:
if [ -z "$x" ] ; then
echo x is empty
fi
With bash, you can use its [[ that does not need quotes:
if [[ -z $x ]] ; then
echo x is empty
fi
I just found the following in man 1p sh, the documentation of POSIX shell:
Historical systems have also been unreliable given the common construct:
test "$response" = "expected string"
One of the following is a more reliable form:
test "X$response" = "Xexpected string"
test "expected string" = "$response"
Note that the second form assumes that expected string could not be confused with any unary primary. If expected string starts with '-', '(', '!', or even '=', the first form should be used instead.
The short and simple answer: [ is actually not a bash directive, while [[ is. It is instead a symlink to the command line utility test.
Now for why:
Like any other command line utility, test interprets anything starting with a - to be an option. It also considers anything starting with = to be an operator. If you don't prefix your arguments to [ (or test) with an alpha character, there is no guarantee the test will work reliably.
Consider the values:
a=1
b=1
And the evaluation:
[ "$a" = "$b" ] && echo "Yes, they match"
Which is essentially running the following command (test ignores the closing ] when its exec name is [):
test 1 = 1 ] && echo "Yes, they match"
Now consider the values:
a="-lt"
b="-lt"
The argument -lt is an option to test. So when you perform the same test, it expands to:
test -lt = -lt ] && echo "Yes, they match"
Now, this is fine on Linux systems (or at least modern ones), since test has been rewritten to ignore options that precede or succeed the = or != operators. However, on some older UNIX systems, this will break with an error like:
test: -lt: unary operator expected
if you want to make sure x is defined :
if [ ${x:-Z} = 'Z' ];then
echo x is empty
fi
I am trying to run the below logic
if [-f $file] && [$variable -lt 1] ; then
some logic
else
print "This is wrong"
fi
It fails with the following error
MyScipt.ksh[10]: [-f: not found
Where 10th line is the if condition , I have put in .
I have also tried
if [-f $file && $variable -lt 1] ; then
which gives the same error.
I know this is a syntax mistake somehwere , but I am not sure , what is the correct syntax when I am using multiple conditions with && in a if block
[ is not an operator, it's the name of a program (or a builtin, sometimes). Use type [ to check. Regardless, you need to put a space after it so that the command line parser knows what to do:
if [ -f $file ]
The && operator might not do what you want in this case, either. You should probably read the bash(1) documentation. In this specific case, it seems like what you want is:
if [ -f $file -a $variable -lt 1 ]
Or in more modern bash syntax:
if [[ -f $file && $variable -lt 1 ]]
The [ syntax is secretly a program!
$ type [
[ is a shell builtin
$ ls -l $(which [)
-rwxr-xr-x 1 root root 35264 Nov 19 16:25 /usr/bin/[
Because of the way the shell parses (technically "lexes") your command line, it sees this:
if - keyword
[-f - the program [-f
$file] - A string argument to the [-f program, made by the value of $file and ]. If $file was "asdf", then this would be asdf]
And so forth, down your command. What you need to do is include spaces, which the shell uses to separate the different parts (tokens) of your command:
if [ -f "$file" ]; then
Now [ stands on its own, and can be recognized as a command/program. Also, ] stands on its own as an argument to [, otherwise [ will complain. A couple more notes about this:
You don't need to put a space before or after ;, because that is a special separator that the shell recognizes.
You should always "double quote" $variables because they get expanded before the shell does the lexing. This means that if an unquoted variable contains a space, the shell will see the value as separate tokens, instead of one string.
Using && in an if-test like that isn't the usual way to do it. [ (also known as test) understands -a to mean "and," so this does what you intended:
if [ -f "$file" -a "$variable" -lt 1 ]; then
Use -a in an if block to represent AND.
Note the space preceding the -f option.
if [ -f $file -a $variable -lt 1] ; then
some logic
else
print "This is wrong"
fi
I couldn't find any one simple straightforward resource spelling out the meaning of and fix for the following BASH shell error, so I'm posting what I found after researching it.
The error:
-bash: [: too many arguments
Google-friendly version: bash open square bracket colon too many arguments.
Context: an if condition in single square brackets with a simple comparison operator like equals, greater than etc, for example:
VARIABLE=$(/some/command);
if [ $VARIABLE == 0 ]; then
# some action
fi
If your $VARIABLE is a string containing spaces or other special characters, and single square brackets are used (which is a shortcut for the test command), then the string may be split out into multiple words. Each of these is treated as a separate argument.
So that one variable is split out into many arguments:
VARIABLE=$(/some/command);
# returns "hello world"
if [ $VARIABLE == 0 ]; then
# fails as if you wrote:
# if [ hello world == 0 ]
fi
The same will be true for any function call that puts down a string containing spaces or other special characters.
Easy fix
Wrap the variable output in double quotes, forcing it to stay as one string (therefore one argument). For example,
VARIABLE=$(/some/command);
if [ "$VARIABLE" == 0 ]; then
# some action
fi
Simple as that. But skip to "Also beware..." below if you also can't guarantee your variable won't be an empty string, or a string that contains nothing but whitespace.
Or, an alternate fix is to use double square brackets (which is a shortcut for the new test command).
This exists only in bash (and apparently korn and zsh) however, and so may not be compatible with default shells called by /bin/sh etc.
This means on some systems, it might work from the console but not when called elsewhere, like from cron, depending on how everything is configured.
It would look like this:
VARIABLE=$(/some/command);
if [[ $VARIABLE == 0 ]]; then
# some action
fi
If your command contains double square brackets like this and you get errors in logs but it works from the console, try swapping out the [[ for an alternative suggested here, or, ensure that whatever runs your script uses a shell that supports [[ aka new test.
Also beware of the [: unary operator expected error
If you're seeing the "too many arguments" error, chances are you're getting a string from a function with unpredictable output. If it's also possible to get an empty string (or all whitespace string), this would be treated as zero arguments even with the above "quick fix", and would fail with [: unary operator expected
It's the same 'gotcha' if you're used to other languages - you don't expect the contents of a variable to be effectively printed into the code like this before it is evaluated.
Here's an example that prevents both the [: too many arguments and the [: unary operator expected errors: replacing the output with a default value if it is empty (in this example, 0), with double quotes wrapped around the whole thing:
VARIABLE=$(/some/command);
if [ "${VARIABLE:-0}" == 0 ]; then
# some action
fi
(here, the action will happen if $VARIABLE is 0, or empty. Naturally, you should change the 0 (the default value) to a different default value if different behaviour is wanted)
Final note: Since [ is a shortcut for test, all the above is also true for the error test: too many arguments (and also test: unary operator expected)
Just bumped into this post, by getting the same error, trying to test if two variables are both empty (or non-empty). That turns out to be a compound comparison - 7.3. Other Comparison Operators - Advanced Bash-Scripting Guide; and I thought I should note the following:
I used -e thinking it means "empty" at first; but that means "file exists" - use -z for testing empty variable (string)
String variables need to be quoted
For compound logical AND comparison, either:
use two tests and && them: [ ... ] && [ ... ]
or use the -a operator in a single test: [ ... -a ... ]
Here is a working command (searching through all txt files in a directory, and dumping those that grep finds contain both of two words):
find /usr/share/doc -name '*.txt' | while read file; do \
a1=$(grep -H "description" $file); \
a2=$(grep -H "changes" $file); \
[ ! -z "$a1" -a ! -z "$a2" ] && echo -e "$a1 \n $a2" ; \
done
Edit 12 Aug 2013: related problem note:
Note that when checking string equality with classic test (single square bracket [), you MUST have a space between the "is equal" operator, which in this case is a single "equals" = sign (although two equals' signs == seem to be accepted as equality operator too). Thus, this fails (silently):
$ if [ "1"=="" ] ; then echo A; else echo B; fi
A
$ if [ "1"="" ] ; then echo A; else echo B; fi
A
$ if [ "1"="" ] && [ "1"="1" ] ; then echo A; else echo B; fi
A
$ if [ "1"=="" ] && [ "1"=="1" ] ; then echo A; else echo B; fi
A
... but add the space - and all looks good:
$ if [ "1" = "" ] ; then echo A; else echo B; fi
B
$ if [ "1" == "" ] ; then echo A; else echo B; fi
B
$ if [ "1" = "" -a "1" = "1" ] ; then echo A; else echo B; fi
B
$ if [ "1" == "" -a "1" == "1" ] ; then echo A; else echo B; fi
B
Another scenario that you can get the [: too many arguments or [: a: binary operator expected errors is if you try to test for all arguments "$#"
if [ -z "$#" ]
then
echo "Argument required."
fi
It works correctly if you call foo.sh or foo.sh arg1. But if you pass multiple args like foo.sh arg1 arg2, you will get errors. This is because it's being expanded to [ -z arg1 arg2 ], which is not a valid syntax.
The correct way to check for existence of arguments is [ "$#" -eq 0 ]. ($# is the number of arguments).
I also faced same problem. #sdaau answer helped me in logical way. Here what I was doing which seems syntactically correct to me but getting too many arguments error.
Wrong Syntax:
if [ $Name != '' ] && [ $age != '' ] && [ $sex != '' ] && [ $birthyear != '' ] && [ $gender != '' ]
then
echo "$Name"
echo "$age"
echo "$sex"
echo "$birthyear"
echo "$gender"
else
echo "Enter all the values"
fi
in above if statement, if I pass the values of variable as mentioned below then also I was getting syntax error
export "Name"="John"
export "age"="31"
export "birthyear"="1990"
export "gender"="M"
With below syntax I am getting expected output.
Correct syntax:
if [ "$Name" != "" -a "$age" != "" -a "$sex" != "" -a "$birthyear" != "" -a "$gender" != "" ]
then
echo "$Name"
echo "$age"
echo "$sex"
echo "$birthyear"
echo "$gender"
else
echo "it failed"
fi
There are few points which we need to keep in mind
use "" instead of ''
use -a instead of &&
put space before and after operator sign like [ a = b], don't use as [ a=b ] in if condition
Hence above solution worked for me !!!
Some times If you touch the keyboard accidentally and removed a space.
if [ "$myvar" = "something"]; then
do something
fi
Will trigger this error message. Note the space before ']' is required.
I have had same problem with my scripts. But when I did some modifications it worked for me. I did like this :-
export k=$(date "+%k");
if [ $k -ge 16 ]
then exit 0;
else
echo "good job for nothing";
fi;
that way I resolved my problem. Hope that will help for you too.