I have 3 files in 3 different directories and I need to printout only files from DIR 1 and 2
1 /tmp/CDE/fileA.log
2 /tmp/CFGH/fileB.log
3 /tmp/CILM_NO/fileC.log
if I run from bash /bin/ls /tmp/C{[A-Z][A-Z],[A-Z][A-Z][A-Z]}/*.log it works and I get:
/tmp/CDE/fileA.log /tmp/CFGH/fileB.log
if I run ls bash command from script perl:
$cmd=`/bin/ls /tmp/C{[A-Z][A-Z],[A-Z][A-Z][A-Z]}/*.log`;
chomp($cmd);
print "$cmd\n";
I receive:
/bin/ls: cannot access /tmp/C{[A-Z][A-Z],[A-Z][A-Z][A-Z]}/*.log: No such file or directory
It looks I need to escape \{ or \, or \} but got the same output and it does not work
I also tried using quote instead of escaping but still got same error output
It's not a matter of permission, script is 777
Can't sort of it.
(not an answer, an explanation)
In your shell
ls /tmp/C{[A-Z][A-Z],[A-Z][A-Z][A-Z]}/*.log
That uses bash Brace Expansion. Bash will expand that to
ls /tmp/C[A-Z][A-Z]/*.log /tmp/C[A-Z][A-Z][A-Z]/*.log
And then do Filename Expansion
in perl
$cmd=`/bin/ls /tmp/C{[A-Z][A-Z],[A-Z][A-Z][A-Z]}/*.log`;
The backticks will call out to /bin/sh not bash, so the brace expansion will not happen
Related
I would like to capture a directory that contains spaces in a bash variable and pass this to the ls command without surrounding in double quotes the variable deference. Following are two examples that illustrate the problem. Example 1 works but it involves typing double quotes. Example 2 does not work, but I wish it did because then I could avoid typing the double quotes.
Example 1, with quotes surrounding variable, as in the solution to How to add path with space in Bash variable, which does not solve the problem:
[user#machine]$ myfolder=/home/username/myfolder\ with\ spaces/
[user#machine]$ ls "$myfolder"
file1.txt file2.txt file3.txt
Example 2, with quotes part of variable, which also does not solve the problem. According to my understanding, in this example, the first quote character sent to the ls command before the error is thrown:
[user#machine]$ myfolder=\"/home/username/myfolder\ with\ spaces/\"
[user#machine]$ ls $myfolder
ls: cannot access '"/home/username/myfolder': No such file or directory
In example 2, the error message indicates that the first double quote was sent to the ls command, but I want these quotes to be interpreted by bash, not ls. Is there a way I can change the myfolder variable so that the second line behaves exactly as the following:
[user#machine]$ ls "/home/username/myfolder with spaces/"
The goal is to craft the myfolder variable in such a way that (1) it does not need to be surrounded by any characters and (2) the ls command will list the contents of the existing directory that it represents.
The motivation is to have an efficient shorthand to pass long directory paths containing spaces to executables on the command line with as few characters as possible - so without double quotes if that is possible.
Assuming some 'extra' characters prior to the ls command is acceptable:
$ mkdir /tmp/'myfolder with spaces'
$ touch /tmp/'myfolder with spaces'/myfile.txt
$ myfolder='/tmp/myfolder with spaces'
$ myfolder=${myfolder// /?} # replace spaces with literal '?'
$ typeset -p myfolder
declare -- myfolder="/tmp/myfolder?with?spaces"
$ set -xv
$ ls $myfolder
+ ls '/tmp/myfolder with spaces'
myfile.txt
Here's a fiddle
Granted, the ? is going to match on any single character but how likely is it that you'll have multiple directories/files with similar names where the only difference is a space vs a non-space?
ls *.txt shows all files whose name ends with .txt
However if I do the following on a zsh shell: (on macOS 10.15 Catalina in my case)
a=*.txt
b='*.txt'
c="*.txt"
# trying no quotes, single quotes, double quotes
# although doesn't make any difference here
ls $a
ls $b
ls $c
ls "$a"
ls "$b"
ls "$c"
I'm getting
ls: *.txt: No such file or directory
in all cases. How do I include wildcards in a variable and then have commands like ls actually process it as wildcards, rather than literal characters?
You should use ~(tilde) between $ and variable name
to perform globbing in zsh. That is
ls $~a
You can enable wildcards in zsh by using the command:
unsetopt nomatch
If you want to make the change permanent, put the command above into your .zshrc file.
I have this bash script that starts with
for d in /data/mydata/*; do
echo $d
filepath=$(echo $d | tr "/" "\n")
pathArr=($filepath) # fails here
echo ${pathArr[-1]}
It runs fine when I just call in on command line
./run_preprocess.sh
but when I run it using screen
screen -dmSL run_preproc ./run_preprocess.sh
it fails on that pathArr line
./run_preproc.sh: 7: ./run_preproc.sh: Syntax error: "(" unexpected (expecting "done")
is there something I need to do to protect the script code?
Based on the error, looks like you're running your script with POSIX sh, not bash. Arrays are undefined in POSIX sh.
To fix this, add a proper hashbang to your script (e.g. /usr/bin/env bash, or run the script directly with Bash interpreter (e.g. /bin/bash script.sh).
In addition (unrelated to the problem at hand), your script (or the snippet posted) has several potential issues:
variables should be quoted to prevent globbing and word splitting (e.g. consider d - one of your files - containing * -- echo $d will include a list of all files, since * will be expanded)
splitting into array with ($var) is done on any IFS character, not just newlines. IFS includes a space, tab and newline by default. Use of read -a or mapfile is recommended over ($var).
Finally, if all you're trying is get the last component in path (filename), you should consider using basename(1):
$ basename /path/to/file
file
or substring removal syntax of Bash parameter expansion:
$ path=/path/to/file
$ echo "${path##*/}"
file
I'm just trying to understand what is happening here, so that I understand how to parse strings in shell scripts better.
I know that usually, when you try to pass a string of arguments separated by spaces directly to a command, they will be treated as a single string argument and therefore not recognized:
>check="FileA.txt FileB.txt"
>ls $check
ls: cannot access FileA.txt FileB.txt: No such file or directory
However, in this script two arguments are taken each as space separated strings. In this case, both strings are recognizes as lists of arguments that can be passed to different commands:
testscript.sh
while getopts o:p: arguments
do
case $arguments in
o) olist="$OPTARG";;
p) plist=$OPTARG;;
esac
done
echo "olist"
ls -l $olist
echo "plist"
ls -l $plist
the output is then as follows:
>testscript.sh -o "fileA.txt fileB.txt" -p "file1.txt file2.txt"
Olist
fileA.txt
fileB.txt
plist
file1.txt
file2.txt
What is different here? Why are the space separated strings suddenly recognized as lists?
Your script does not start with a #!-line and does therefore not specify an interpreter. In that case the default is used, which is /bin/sh and not your login shell or the shell you are starting the script from (unless that is /bin/sh of course). Chances are good that /bin/sh is not a zsh, as most distributions and Unices seem to use sh, bash, dash or ksh as default shell. All of which handle parameter expansion such that strings are handles as lists if the parameter was not quoted with double-quotes.
If you want to use zsh as interpreter for your scripts, you have to specify it in the first line of the script:
#!/usr/bin/zsh
Modify the path to wherever your zsh resides.
You can also use env as a wrapper:
#!/usr/bin/env zsh
This makes you more independent of the actual location of zsh, it just has to be in $PATH.
As a matter of fact (using bash)...
sh$ check="FileA.txt FileB.txt"
sh$ ls $check
ls: cannot access FileA.txt: No such file or directory
ls: cannot access FileB.txt: No such file or directory
When you write $check without quotes, the variable is substituted by its content. Insides paces (or to be precises inside occurrences of IFS) are considered as field separators. Just as you where expecting it first.
The only way I know to reproduce your behavior is to set IFS to something else than its default value:
sh$ export IFS="-"
sh$ check="FileA.txt FileB.txt"
sh$ ls $check
ls: cannot access FileA.txt FileB.txt: No such file or directory
Everybody says eval is evil, and you should use $() as a replacement. But I've run into a situation where the unquoting isn't handled the same inside $().
Background is that I've been burned too often by file paths with spaces in them, and so like to quote all such paths. More paranoia about wanting to know where all my executables are coming from. Even more paranoid, not trusting myself, and so like being able to display the created commands I'm about to run.
Below I try variations on using eval vs. $(), and whether the command name is quoted (cuz it could contain spaces)
BIN_LS="/bin/ls"
thefile="arf"
thecmd="\"${BIN_LS}\" -ld -- \"${thefile}\""
echo -e "\n Running command '${thecmd}'"
$($thecmd)
Running command '"/bin/ls" -ld -- "arf"'
./foo.sh: line 8: "/bin/ls": No such file or directory
echo -e "\n Eval'ing command '${thecmd}'"
eval $thecmd
Eval'ing command '"/bin/ls" -ld -- "arf"'
/bin/ls: cannot access arf: No such file or directory
thecmd="${BIN_LS} -ld -- \"${thefile}\""
echo -e "\n Running command '${thecmd}'"
$($thecmd)
Running command '/bin/ls -ld -- "arf"'
/bin/ls: cannot access "arf": No such file or directory
echo -e "\n Eval'ing command '${thecmd}'"
eval $thecmd
Eval'ing command '/bin/ls -ld -- "arf"'
/bin/ls: cannot access arf: No such file or directory
$("/bin/ls" -ld -- "${thefile}")
/bin/ls: cannot access arf: No such file or directory
So... this is confusing. A quoted command path is valid everywhere except inside a $() construct? A shorter, more direct example:
$ c="\"/bin/ls\" arf"
$ $($c)
-bash: "/bin/ls": No such file or directory
$ eval $c
/bin/ls: cannot access arf: No such file or directory
$ $("/bin/ls" arf)
/bin/ls: cannot access arf: No such file or directory
$ "/bin/ls" arf
/bin/ls: cannot access arf: No such file or directory
How does one explain the simple $($c) case?
The use of " to quote words is part of your interaction with Bash. When you type
$ "/bin/ls" arf
at the prompt, or in a script, you're telling Bash that the command consists of the words /bin/ls and arf, and the double-quotes are really emphasizing that /bin/ls is a single word.
When you type
$ eval '"/bin/ls" arf'
you're telling Bash that the command consists of the words eval and "/bin/ls" arf. Since the purpose of eval is to pretend that its argument is an actual human-input command, this is equivalent to running
$ "/bin/ls" arf
and the " gets processed just like at the prompt.
Note that this pretense is specific to eval; Bash doesn't usually go out of its way to pretend that something was an actual human-typed command.
When you type
$ c='"/bin/ls" arf'
$ $c
the $c gets substituted, and then undergoes word splitting (see ยง3.5.7 "Word Splitting" in the Bash Reference Manual), so the words of the command are "/bin/ls" (note the double-quotes!) and arf. Needless to say, this doesn't work. (It's also not very safe, since in addition to word-splitting, $c also undergoes filename-expansion and whatnot. Generally your parameter-expansions should always be in double-quotes, and if they can't be, then you should rewrite your code so they can be. Unquoted parameter-expansions are asking for trouble.)
When you type
$ c='"/bin/ls" arf'
$ $($c)
this is the same as before, except that now you're also trying to use the output of the nonworking command as a new command. Needless to say, that doesn't cause the nonworking command to suddenly work.
As Ignacio Vazquez-Abrams says in his answer, the right solution is to use an array, and handle the quoting properly:
$ c=("/bin/ls" arf)
$ "${c[#]}"
which sets c to an array with two elements, /bin/ls and arf, and uses those two elements as the word of a command.
With the fact that it doesn't make sense in the first place. Use an array instead.
$ c=("/bin/ls" arf)
$ "${c[#]}"
/bin/ls: cannot access arf: No such file or directory
From the man page for bash, regarding eval:
eval [arg ...]:
The args are read and concatenated together into a single command.
This command is then read and executed by the shell, and its exit
status is returned as the value of eval.
When c is defined as "\"/bin/ls\" arf", the outer quotes will cause the entire thing to be processed as the first argument to eval, which is expected to be a command or program. You need to pass your eval arguments in such a way that the target command and its arguments are listed separately.
The $(...) construct behaves differently than eval because it is not a command that takes arguments. It can process the entire command at once instead of processing arguments one at a time.
A note on your original premise: The main reason that people say that eval is evil was because it is commonly used by scripts to execute a user-provided string as a shell command. While handy at times, this is a major security problem (there's typically no practical way to safety-check the string before executing it). The security problem doesn't apply if you are using eval on hard-coded strings inside your script, as you are doing. However, it's typically easier and cleaner to use $(...) or `...` inside of scripts for command substitution, leaving no real use case left for eval.
Using set -vx helps us understand how bash process the command string.
As seen in the picture, "command" works cause quotes will be stripped when processing. However, when $c(quoted twice) is used, only the outside single quotes are removed. eval can process the string as the argument and outside quotes are removed step by step.
It is probably just related to how bash semanticallly process the string and quotes.
Bash does have many weird behaviours about quotes processing:
Bash inserting quotes into string before execution
How do you stop bash from stripping quotes when running a variable as a command?
Bash stripping quotes - how to preserve quotes