I have a file called inp.txt which lists 3 directory names
#!/bin/sh
while read dirname
do
echo $dirname
"ls -l" $dirname
done < inp.txt
When I run the above, I get this error:
line 5: ls -l: command not found
If I do just "ls" instead of "ls -l", it works fine. What am I missing here?
Get rid of the quotes.
while read dirname
do
echo "$dirname"
ls -l "$dirname"
done < inp.txt
When you have quotes you're saying, "treat this as a single word." The shell looks for an executable named ls -l rather than passing the argument -l to the command ls.
Nitpicker's note: If you want to use quotes properly, add them around "$dirname".
Otherwise, if you had a directory named "Music Files" with a space in the name, without quotes your script would treat that as two directory names and print something like:
ls: Music: No such file or directory
ls: Files: No such file or directory
The shell interprets space as argument separaters. By putting the quotes around something, you force shell to see it as one argument. When executing, the shell interprets the first argument as the command to execute. In this case you're telling it to execute the command named "ls -l" with no arguments, instead of "ls" with argument "-l".
You should remove the quotes around ls -l.
Related
ls *.txt shows all files whose name ends with .txt
However if I do the following on a zsh shell: (on macOS 10.15 Catalina in my case)
a=*.txt
b='*.txt'
c="*.txt"
# trying no quotes, single quotes, double quotes
# although doesn't make any difference here
ls $a
ls $b
ls $c
ls "$a"
ls "$b"
ls "$c"
I'm getting
ls: *.txt: No such file or directory
in all cases. How do I include wildcards in a variable and then have commands like ls actually process it as wildcards, rather than literal characters?
You should use ~(tilde) between $ and variable name
to perform globbing in zsh. That is
ls $~a
You can enable wildcards in zsh by using the command:
unsetopt nomatch
If you want to make the change permanent, put the command above into your .zshrc file.
I know about escaping, quoting and stuff, but still have a problem.
I you have a script containing "cd $1", and call it with an argument containing spaces, cd will always return an error message - it stops at the first space and can't find the directory. I tried protecting the arguments in every way :
ls -l
+-rwx... script
+drwx... dir with spaces/
cat script
+cd $1
script dir with spaces
+cd: dir: no such file or directory
script "dir with spaces"
+cd: dir: no such file or directory
script dir\ with\ spaces
+cd: dir\: no such file or directory
but none will work.
I feel like I'm missing the obvious, thanks for enlightening me.
You need to quote the expansion of "$1" to prevent it from being word split as well as quoting the string passed to the script to prevent it from being word-split.
So
$ cat script.sh
cd -- "$1"
$ ./script.sh "dir with spaces"
Edit: As gniourf_gniourf correctly pointed out using -- as the first argument to cd prevents problems should paths ever start with -.
Use double quotes on the variable
cd "$1"
I'm just trying to understand what is happening here, so that I understand how to parse strings in shell scripts better.
I know that usually, when you try to pass a string of arguments separated by spaces directly to a command, they will be treated as a single string argument and therefore not recognized:
>check="FileA.txt FileB.txt"
>ls $check
ls: cannot access FileA.txt FileB.txt: No such file or directory
However, in this script two arguments are taken each as space separated strings. In this case, both strings are recognizes as lists of arguments that can be passed to different commands:
testscript.sh
while getopts o:p: arguments
do
case $arguments in
o) olist="$OPTARG";;
p) plist=$OPTARG;;
esac
done
echo "olist"
ls -l $olist
echo "plist"
ls -l $plist
the output is then as follows:
>testscript.sh -o "fileA.txt fileB.txt" -p "file1.txt file2.txt"
Olist
fileA.txt
fileB.txt
plist
file1.txt
file2.txt
What is different here? Why are the space separated strings suddenly recognized as lists?
Your script does not start with a #!-line and does therefore not specify an interpreter. In that case the default is used, which is /bin/sh and not your login shell or the shell you are starting the script from (unless that is /bin/sh of course). Chances are good that /bin/sh is not a zsh, as most distributions and Unices seem to use sh, bash, dash or ksh as default shell. All of which handle parameter expansion such that strings are handles as lists if the parameter was not quoted with double-quotes.
If you want to use zsh as interpreter for your scripts, you have to specify it in the first line of the script:
#!/usr/bin/zsh
Modify the path to wherever your zsh resides.
You can also use env as a wrapper:
#!/usr/bin/env zsh
This makes you more independent of the actual location of zsh, it just has to be in $PATH.
As a matter of fact (using bash)...
sh$ check="FileA.txt FileB.txt"
sh$ ls $check
ls: cannot access FileA.txt: No such file or directory
ls: cannot access FileB.txt: No such file or directory
When you write $check without quotes, the variable is substituted by its content. Insides paces (or to be precises inside occurrences of IFS) are considered as field separators. Just as you where expecting it first.
The only way I know to reproduce your behavior is to set IFS to something else than its default value:
sh$ export IFS="-"
sh$ check="FileA.txt FileB.txt"
sh$ ls $check
ls: cannot access FileA.txt FileB.txt: No such file or directory
I cannot believe I've spent 1.5 hours on something as trivial as this. I'm writing a very simple shell script which greps a file, stores the output in a variable, and echos the variable to STDOUT.
I have checked the grep command with the regex on the command line, and it works fine. But for some reason, the grep command doesn't work inside the shell script.
Here is the shell script I wrote up:
#!/bin/bash
tt=grep 'test' $1
echo $tt
I ran this with the following command: ./myScript.sh testingFile. It just prints an empty line.
I have already used chmod and made the script executable.
I have checked that the PATH variable has /bin in it.
Verified that echo $SHELL gives /bin/bash
In my desperation, I have tried all combinations of:
tt=grep 'test' "$1"
echo ${tt}
Not using the command line argument at all, and hardcoding the name of the file tt=grep 'test' testingFile
I found this: grep fails inside bash script but works on command line, and even used dos2unix to remove any possible carriage returns.
Also, when I try to use any of the grep options, like: tt=grep -oE 'test' testingFile, I get an error saying: ./out.sh: line 3: -oE: command not found.
This is crazy.
You need to use command substitution:
#!/usr/bin/env bash
test=$(grep 'foo' "$1")
echo "$test"
Command substitution allows the output of a command to replace the command itself. Command substitution occurs when a command is enclosed like this:
$(command)
or like this using backticks:
`command`
Bash performs the expansion by executing COMMAND and replacing the command substitution with the standard output of the command, with any trailing newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting.
The $() version is usually preferred because it allows nesting:
$(command $(command))
For more information read the command substitution section in man bash.
Everybody says eval is evil, and you should use $() as a replacement. But I've run into a situation where the unquoting isn't handled the same inside $().
Background is that I've been burned too often by file paths with spaces in them, and so like to quote all such paths. More paranoia about wanting to know where all my executables are coming from. Even more paranoid, not trusting myself, and so like being able to display the created commands I'm about to run.
Below I try variations on using eval vs. $(), and whether the command name is quoted (cuz it could contain spaces)
BIN_LS="/bin/ls"
thefile="arf"
thecmd="\"${BIN_LS}\" -ld -- \"${thefile}\""
echo -e "\n Running command '${thecmd}'"
$($thecmd)
Running command '"/bin/ls" -ld -- "arf"'
./foo.sh: line 8: "/bin/ls": No such file or directory
echo -e "\n Eval'ing command '${thecmd}'"
eval $thecmd
Eval'ing command '"/bin/ls" -ld -- "arf"'
/bin/ls: cannot access arf: No such file or directory
thecmd="${BIN_LS} -ld -- \"${thefile}\""
echo -e "\n Running command '${thecmd}'"
$($thecmd)
Running command '/bin/ls -ld -- "arf"'
/bin/ls: cannot access "arf": No such file or directory
echo -e "\n Eval'ing command '${thecmd}'"
eval $thecmd
Eval'ing command '/bin/ls -ld -- "arf"'
/bin/ls: cannot access arf: No such file or directory
$("/bin/ls" -ld -- "${thefile}")
/bin/ls: cannot access arf: No such file or directory
So... this is confusing. A quoted command path is valid everywhere except inside a $() construct? A shorter, more direct example:
$ c="\"/bin/ls\" arf"
$ $($c)
-bash: "/bin/ls": No such file or directory
$ eval $c
/bin/ls: cannot access arf: No such file or directory
$ $("/bin/ls" arf)
/bin/ls: cannot access arf: No such file or directory
$ "/bin/ls" arf
/bin/ls: cannot access arf: No such file or directory
How does one explain the simple $($c) case?
The use of " to quote words is part of your interaction with Bash. When you type
$ "/bin/ls" arf
at the prompt, or in a script, you're telling Bash that the command consists of the words /bin/ls and arf, and the double-quotes are really emphasizing that /bin/ls is a single word.
When you type
$ eval '"/bin/ls" arf'
you're telling Bash that the command consists of the words eval and "/bin/ls" arf. Since the purpose of eval is to pretend that its argument is an actual human-input command, this is equivalent to running
$ "/bin/ls" arf
and the " gets processed just like at the prompt.
Note that this pretense is specific to eval; Bash doesn't usually go out of its way to pretend that something was an actual human-typed command.
When you type
$ c='"/bin/ls" arf'
$ $c
the $c gets substituted, and then undergoes word splitting (see ยง3.5.7 "Word Splitting" in the Bash Reference Manual), so the words of the command are "/bin/ls" (note the double-quotes!) and arf. Needless to say, this doesn't work. (It's also not very safe, since in addition to word-splitting, $c also undergoes filename-expansion and whatnot. Generally your parameter-expansions should always be in double-quotes, and if they can't be, then you should rewrite your code so they can be. Unquoted parameter-expansions are asking for trouble.)
When you type
$ c='"/bin/ls" arf'
$ $($c)
this is the same as before, except that now you're also trying to use the output of the nonworking command as a new command. Needless to say, that doesn't cause the nonworking command to suddenly work.
As Ignacio Vazquez-Abrams says in his answer, the right solution is to use an array, and handle the quoting properly:
$ c=("/bin/ls" arf)
$ "${c[#]}"
which sets c to an array with two elements, /bin/ls and arf, and uses those two elements as the word of a command.
With the fact that it doesn't make sense in the first place. Use an array instead.
$ c=("/bin/ls" arf)
$ "${c[#]}"
/bin/ls: cannot access arf: No such file or directory
From the man page for bash, regarding eval:
eval [arg ...]:
The args are read and concatenated together into a single command.
This command is then read and executed by the shell, and its exit
status is returned as the value of eval.
When c is defined as "\"/bin/ls\" arf", the outer quotes will cause the entire thing to be processed as the first argument to eval, which is expected to be a command or program. You need to pass your eval arguments in such a way that the target command and its arguments are listed separately.
The $(...) construct behaves differently than eval because it is not a command that takes arguments. It can process the entire command at once instead of processing arguments one at a time.
A note on your original premise: The main reason that people say that eval is evil was because it is commonly used by scripts to execute a user-provided string as a shell command. While handy at times, this is a major security problem (there's typically no practical way to safety-check the string before executing it). The security problem doesn't apply if you are using eval on hard-coded strings inside your script, as you are doing. However, it's typically easier and cleaner to use $(...) or `...` inside of scripts for command substitution, leaving no real use case left for eval.
Using set -vx helps us understand how bash process the command string.
As seen in the picture, "command" works cause quotes will be stripped when processing. However, when $c(quoted twice) is used, only the outside single quotes are removed. eval can process the string as the argument and outside quotes are removed step by step.
It is probably just related to how bash semanticallly process the string and quotes.
Bash does have many weird behaviours about quotes processing:
Bash inserting quotes into string before execution
How do you stop bash from stripping quotes when running a variable as a command?
Bash stripping quotes - how to preserve quotes