I would like to capture a directory that contains spaces in a bash variable and pass this to the ls command without surrounding in double quotes the variable deference. Following are two examples that illustrate the problem. Example 1 works but it involves typing double quotes. Example 2 does not work, but I wish it did because then I could avoid typing the double quotes.
Example 1, with quotes surrounding variable, as in the solution to How to add path with space in Bash variable, which does not solve the problem:
[user#machine]$ myfolder=/home/username/myfolder\ with\ spaces/
[user#machine]$ ls "$myfolder"
file1.txt file2.txt file3.txt
Example 2, with quotes part of variable, which also does not solve the problem. According to my understanding, in this example, the first quote character sent to the ls command before the error is thrown:
[user#machine]$ myfolder=\"/home/username/myfolder\ with\ spaces/\"
[user#machine]$ ls $myfolder
ls: cannot access '"/home/username/myfolder': No such file or directory
In example 2, the error message indicates that the first double quote was sent to the ls command, but I want these quotes to be interpreted by bash, not ls. Is there a way I can change the myfolder variable so that the second line behaves exactly as the following:
[user#machine]$ ls "/home/username/myfolder with spaces/"
The goal is to craft the myfolder variable in such a way that (1) it does not need to be surrounded by any characters and (2) the ls command will list the contents of the existing directory that it represents.
The motivation is to have an efficient shorthand to pass long directory paths containing spaces to executables on the command line with as few characters as possible - so without double quotes if that is possible.
Assuming some 'extra' characters prior to the ls command is acceptable:
$ mkdir /tmp/'myfolder with spaces'
$ touch /tmp/'myfolder with spaces'/myfile.txt
$ myfolder='/tmp/myfolder with spaces'
$ myfolder=${myfolder// /?} # replace spaces with literal '?'
$ typeset -p myfolder
declare -- myfolder="/tmp/myfolder?with?spaces"
$ set -xv
$ ls $myfolder
+ ls '/tmp/myfolder with spaces'
myfile.txt
Here's a fiddle
Granted, the ? is going to match on any single character but how likely is it that you'll have multiple directories/files with similar names where the only difference is a space vs a non-space?
Related
I am trying to list all files located in specific sub-directories of a directory in my bash script. Here is what I tried.
root_dir="/home/shaf/data/"
sub_dirs_prefixes=('ab' 'bp' 'cd' 'cn' 'll' 'mr' 'mb' 'nb' 'nh' 'nw' 'oh' 'st' 'wh')
ls "$root_dir"{$(IFS=,; echo "${sub_dirs_prefixes[*]}")}"rrc/"
Please note that I do not want to expand value stored in $root_dir as it may contain spaces but I do want to expand sub-path contained in {} which is a comma delimited string of contents of $sub_dirs_prefixes. I stored sub-directories prefixes in an array variable, $sub_dirs_prefixes , because I have to use them later on for something else.
I am getting following error:
ls: cannot access /home/shaf/data/{ab,bp,cd,cn,ll,mr,mb,nb,nh,nw,oh,st,wh}rrc/
If I copy the path in error message and run ls from command line, it does display contents of listed sub-directories.
You can command substitution to generate an extended pattern.
shopt -s extglob
ls "$root_dir"/$(IFS="|"; echo "#(${sub_dirs_prefixes[*]})rrc")
By the time parameter can command substitutions have completed, the shell sees this just before performing pathname expansion:
ls "/home/shaf/data/"/#(ab|bp|cd|cn|ll|mr|mb|nb|nh|nw|oh|st|wh)rrc
The #(...) pattern matches one of the enclosed prefixes.
It gets a little trickier if the components of the directory names contain characters that need to be quoted, since we aren't quoting the command substitution.
The POSIX shell standard at
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04
says in Section 2.6:
command substitution (...) shall be performed
(...)
Quote removal (...) shall always be performed last.
It appears to me that quote removal is not performed after command substitution:
$ echo "#"
#
$ echo '"'
"
as expected, but
$ echo $(echo '"')#"
>
What am I not understanding?
Added after reading answer/comments:
From what everybody is saying, the consideration of quotes happens at the very beginning of parsing, for example, to decide if a command is even "acceptable". Then why does the standard bother to emphasise, that the quote removal is performed late in the process??
"then the outer command becomes echo "#" and is balanced"
That is not 'balanced' because the first double-quote does not count. Quotes are only meaningful as quotes if they appear unencumbered on the command line.
To verify, let's look at this:
$ echo $(echo '"')#
"#
That is balanced because the shell does considers that " to be just another character.
By contrast, this is unbalanced because it has one and only one shell-active ":
$ echo $(echo '"')#"
>
Similar example 1
Here we show the same thing but using parameter expansion instead of command substitution:
$ q='"'; echo $q
"
Once the shell has substituted " for $q, one might think that there was an unbalanced double-quote. But, that double-quote was the results of parameter expansion and is therefore not a shell-active quote.
Similar example 2
Let's consider a directory containing file:
$ ls
file
$ ls "file"
file
As you can see above, quote removal is perfomed before ls is run.
But, consider this command:
$ echo ls $(echo '"file"')
ls "file"
As you can see ls $(echo '"file"') expands to ls "file" which is the command which ran successfully above. Now, let's try running that:
$ ls $(echo '"file"')
ls: cannot access '"file"': No such file or directory
As you can see, the shell does not treat the double-quotes that remain after command substitution. This is because those quotes are not considered to be shell-active. As a consequence, they are treated as normal characters and passed on to ls which complains that the file whose name begins and ends with " does not exist in the directory.
The same is happening here:
$ cmd='ls "file"'
$ $cmd
ls: cannot access '"file"': No such file or directory
POSIX standard
From the POSIX standard:
Enclosing characters in single-quotes ( ' ' ) shall preserve the
literal value of each character within the single-quotes
In other words, once the double-quote appears inside single quotes, it has no special powers: it is just another character.
The standard also mentions escaping and double-quotes as methods of preserving "the literal value" of a character.
Practical consequences
People new to shell often want to store a command in a variable as in the cmd='ls "file"' example above. But, because quotes and other shell-active characters cease to be shell active once they are stored in a variable, the complex cases always fail. This leads to a classic essay:
"I'm trying to put a command in a variable, but the complex cases always fail!"
I know about escaping, quoting and stuff, but still have a problem.
I you have a script containing "cd $1", and call it with an argument containing spaces, cd will always return an error message - it stops at the first space and can't find the directory. I tried protecting the arguments in every way :
ls -l
+-rwx... script
+drwx... dir with spaces/
cat script
+cd $1
script dir with spaces
+cd: dir: no such file or directory
script "dir with spaces"
+cd: dir: no such file or directory
script dir\ with\ spaces
+cd: dir\: no such file or directory
but none will work.
I feel like I'm missing the obvious, thanks for enlightening me.
You need to quote the expansion of "$1" to prevent it from being word split as well as quoting the string passed to the script to prevent it from being word-split.
So
$ cat script.sh
cd -- "$1"
$ ./script.sh "dir with spaces"
Edit: As gniourf_gniourf correctly pointed out using -- as the first argument to cd prevents problems should paths ever start with -.
Use double quotes on the variable
cd "$1"
At the command line, this cat works as expected:
cat /home/me/path\ with\ spaces/to/file
But if I put it in a script:
#!/bin/bash
FILE="$1"
cat $FILE # cat "$FILE" gives same result
and call the script with ./script.sh "/home/me/path\ with\ spaces/to/file", I get:
cat: /home/me/path\ with\ spaces/to/file: No such file or directory
Note the escape quotes, which should be in the right places.
What gives?
Simply use double quotes to prevent word splitting:
cat "$FILE"
As an aside, upper case variable names should be reserved for shell internal variables, so you should change FILE to file.
If you are quoting the argument to your script (which is a good idea), then the file name doesn't need backslashes:
./script.sh "/home/me/path with spaces/to/file"
Working on a project within a large co. The folder for a project contains the "$" (dollar sign) character. This seems to be confusing bash when I try to change directory to this folder:
cd TEST_$_xyz
Yields an error:
No such file or directory
I'm almost sure that this is because of bash's handling of the "$" character, but I'm extremely new to bash, so I'm looking for confirmation before I force a name-change.
Thanks
You need to escape the dollar ($) sign like so. Otherwise, it treats $_xyz as an environment variable.
cd TEST_\$_xyz
example:
# In this case, $a evaluates to nothing because it is not defined
me#mypc:~/tmp/asdf$ mkdir a$a
me#mypc:~/tmp/asdf$ ls
a
# Here, I have escaped $ with \ so that it's treated like a normal $ character
me#mypc:~/tmp/asdf$ mkdir a\$a
me#mypc:~/tmp/asdf$ ls
a a$a
# changing directory to directory with escaped $ sign
me#mypc:~/tmp/asdf$ cd a\$a
me#mypc:~/tmp/asdf/a$a$
You can enclose the filename in single quotes - that way there is no variable expansion:
cd 'TEST_$_xyz'
See the "Single Quotes" section of the bash documentation
You can use...
cd TEST*xyz
(An asterix can cope with many different chars, as '$', space and others.)