Suppose I have a directory containing the files
foo bar.txt
foo baz.txt
(each with a space between 'o' and 'b'). Suppose I would like to do this:
for f in *.txt; do mv ${f} `basename ${f} .txt`; done
This fails because bash expands *.txt to
foo bar.txt foo baz.txt
instead of
foo\ bar.txt foo\ baz.txt
i.e. properly escaped as if I had used tab completion.
How can I get bash to properly escape its output?
you put quotes in your variables. this way you preserve the space. Also, there's no need to use external command basename. the shell can do that for you. (assuming you are using bash)
for file in *.txt
do
mv "$file" "${file%.txt}"
done
Or if it is one off operation you can use vim:
> ls -al
foo bar.txt
foo baz.txt
Open vim and execute:
:r!ls *.txt
This loads files, then execute:
:%s/\(\(.*\)\.txt\)/mv "\1" "\2"/gc
This will replace the lines with:
mv "foo bar.txt" "foo bar"
mv "foo baz.txt" "foo baz"
Highlight all with Ctrl-v down, then enter : and type the rest of this command:
:'<,'>!bash
That will execute the highlighted commands in bash. Quit vim and check your directory:
> ls
foo bar
foo baz
Related
I know there is a duplicate for this question already at: How to trim whitespace from a Bash variable?.
I read all the answers there but I have a question about another solution in my mind and I want to know if this works.
This is the solution I think works.
a=$(printf "%s" $a)
Here is a demonstration.
$ a=" foo "
$ a=$(printf "%s" $a)
$ echo "$a"
foo
Is there any scenario in which this solution may fail?
If there is such a scenario in which this solution may fail, can we modify this solution to handle that scenario without compromising the simplicity of the solution too much?
If the variable a is set with something like "-e", "-n" in the begining, depending on how you process later your result, a user might crash your script:
-e option allows echo to interpret things backslashed.
Even in the case you only want to display the variable a, -n would screw your layout.
You could think about using regex to check if your variable starts with '-' and is followed by one of the available echo options (-n, -e, -E, --help, --version).
It fails when the input contains spaces between non-whitespace characters.
$ a=" foo bar "
$ a=$(printf "%s" $a)
$ echo "$a"
foobar
The expected output was the following instead.
foo bar
You could use Bash's builtin pattern substitution.
Note: Bash pattern substitution uses 'Pathname Expansion' (glob) pattern matching, not regular expressions. My solution requires enabling the optional shell behaviour extglob (shopt -s extglob).
$shopt -s extglob
$ a=" foo bar "
$ echo "Remove trailing spaces: '${a/%*([[:space:]])}'"
Remove trailing spaces: ' foo bar'
$ echo "Remove leading spaces: '${a/#*([[:space:]])}'"
Remove leading spaces: 'foo bar '
$ echo "Remove all spaces anywhere: '${a//[[:space:]]}'"
Remove all spaces anywhere: 'foobar'
For reference, refer to the 'Parameter Expansion' (Pattern substitution) and 'Pathname Expansion' subsections of the EXPANSION section of the Bash man page.
Echoing without quotes... 1 line. Fine.
$ echo $(ls -1dmb /bin/*) > test
$ wc -l test
1 test
Echoing with quotes... 396 lines. Bad.
$ echo "$(ls -1dmb /bin/*)" > test
$ wc -l test
396 test
The problem comes when using echo for writing a file and expanding a long variable.
Why does this happen? How to fix it?
ls is detecting that your stdout is not a terminal.
check the output of ls -1dmb /bin/* | cat vs ls -1dmb /bin/*. It's ls, who is splitting the output.
Similarly, for ls --color=auto case, color option is used, based on whether the stdout is terminal or not.
When quotes are used, echo is provided with a single arguments, which has embedded newlines, spaces, which are echoed as-is to file.
When quotes are not used, echo is provided multiple arguments, which are split by the IFS. Thus echo prints all of them in a single line.
But, don't skip these quotes...
How to fix it:
I think, the splitting always occurs at the end of some file name & never in between a filename. So one of these 2 options may work for you:
ls -1dmb /bin/* | tr '\n' ' ' >test
ls -1dmb /bin/* | tr -d '\n' >test
#anishsane correctly answers the topic question (that ls is doing the wrapping and ways to remove them) and covers the quoting issue as well but the quoting issue is responsible for the line count difference and not ls.
The issue here is entirely one of quoting and how command lines, echo, and command substitution all work.
The output from "$(ls ...)" is a single string with embedded newlines protected from the shell via the quotes. Hand that value to echo and echo spits it back out literally (with the newlines).
The output from $(ls ...) is a string that is unprotected from the shell and thus undergoes word splitting and whitespace normalization. Command substitution cannot terminate your command line early (you wouldn't want echo $(ls -1) in a directory with two files to run echo first_file; second_file would you?) the newlines are left as word separators between the arguments to echo. The shell then word splits the result on whitespace (including newlines) and gives echo a list of arguments at which point echo happily executes echo first_file second_file ... which, as you can guess, only outputs a single line of output.
Try this to see what I mean:
$ c() {
printf 'argc: %s\n' "$#";
printf 'argv: %s\n' "$#"
}
$ ls *
a.sh b.sh temp
$ ls -1dmb *
a.sh, b.sh, temp
$ c "$(ls -1dmb *)"
argc: 1
argv: a.sh, b.sh, temp
$ c $(ls -1dmb *)
argc: 3
argv: a.sh,
argv: b.sh,
argv: temp
Say I have a command I want to run (cmd) and a variable containing the arguments I want to pass to the function (something like --foo 'bar baz' qux). Like so:
#!/bin/sh
command=cmd
args="--foo 'bar baz' qux"
The arguments contain quotes, like the ones shown above, that group together an argument containing a space. I'd then like to run the command:
$command $args
This, of course, results in running the command with four arguments: --foo, 'bar, baz', and qux. The alternative I'm used to (i.e., when using "$#") presents a different problem:
$command "$args"
This executes the command with one argument: --foo 'bar baz' qux.
How can I run the command with three arguments (--foo, bar baz, and qux) as intended?
Use an array to specify your argument list exactly, without string-splitting (which is what's doing the wrong thing here) getting in your way:
args=( --foo "bar baz" qux )
command "${args[#]}"
If you need to build your argument list dynamically, you can append to arrays with +=:
args=( )
while ...; do
args+=( "$another_argument" )
done
call_your_subprocess "${args[#]}"
Note that the use of quotation marks, and [#] instead of [*], is essential.
If you can throw away the current positional variables ($1...) you can use the following:
set -- '--foo' 'bar baz' 'qux'
echo "$#" # Prints "3" (without quotes)
echo "$2" # Prints "bar baz" (without quotes)
command "$#"
Just tested it in a #!/usr/bin/env sh script, so it works at least in Dash, and should work in any Bourne Shell variant. No eval, Python or Bash necessary.
One possibility is to use eval:
#!/bin/sh
args="--foo 'bar baz' qux"
cmd="python -c 'import sys; print sys.argv'"
eval $cmd $args
That way you cause the command line to be interpreted rather than just split according to IFS. This gives the output:
$ ./args.sh
['-c', '--foo', 'bar baz', 'qux']
So that you can see the args are passed as you wanted.
If you have the command in the form:
args="--foo 'bar baz' qux"
and getting the command as an array in the first place isn't an option, then you'll need to use eval to turn it back into an array:
$ args="--foo 'bar baz' qux"
$ eval "arr=($args)"
But it's important to note that this is unsafe if $args is being provided by an untrusted source, since it can be used to execute arbitrary commands, e.g. args='$(rm -rf /)'; eval "arr=($args)" will cause the above code to run the rm -rf / before you've even used arr.
Then you can use "${arr[#]}" to expand it as arguments to a command:
$ bash -c 'echo $0' "${arr[#]}"
--foo
$ bash -c 'echo $1' "${arr[#]}"
bar baz
or to run your command:
"$command" "${arr[#]}"
Note that there are differences between ${arr[*]}, ${arr[#]}, "${arr[*]}" and "${arr[#]}", and only the last of these does what you want in most cases
It works for me by changing the IFS (the internal field seperator) of the shell (to only contain a newline). Here is how I override commands that use quoted arguments:
$ cat ~/.local/bin/make
#!/bin/sh
# THIS IS IMPORTANT! (don't split up quoted strings in arguments)
IFS="
"
exec /usr/bin/make ${#} -j6
(/bin/sh is dash)
It will eat the quotes when replacing the command with echo, so will look wrong when "testing"; but the command does get them as intended.
It can be tested by replacing the exec line with
for arg in $#; do echo $arg; done
Is it possible to configure zsh to suggest filenames (or anything else) inside of a quoted string?
I've seen this thread on bash: Bash TAB-completion inside double-quoted string
But I'm not sure whether that solution is compatible between shells.
No problem for tab completion inside quotes.
$ touch "spaces in a filename"
$ ls
spaces in a filename
$ ls sp[TAB]
gives ->
$ ls spaces\ in\ a\ filename
$ ls "sp[TAB]
gives ->
$ ls "spaces in a filename"
I am a newbie for shell script.
I noticed that for command like:
tar cf test.tar *TEST* *TEST2*
sh and bash will expand TEST to a list of file names, while csh will not.
I want the shell not to expend, how can I do this?
Use ' or " will lead to other problems. If there is files match pattern *TEST*, but no files match *TEST2*,
then the tar will fail.
[EDIT]I think this is an expend difference between bash and csh.
When no pattern found for *TEST*, csh will expend to "", while bash will expend to 'TEST' (add quote char).
From man bash:
If the nullglob option is set, and no
matches are found, the word is
removed.
For example:
betelgeuse:TEST james$ touch TEST1 TEST3
betelgeuse:TEST james$ ls *TEST* *TEST2*
ls: *TEST2*: No such file or directory
TEST1 TEST3
betelgeuse:TEST james$ shopt -s nullglob
betelgeuse:TEST james$ ls *TEST* *TEST2*
TEST1 TEST3
This is approximately equivalent to the csh behaviour:
[betelgeuse:/tmp/TEST] james% ls *TEST* *TEST2*
TEST1 TEST3
The difference is that csh will return an error if all of the filename patterns on a line fail to match anything:
[betelgeuse:/tmp/TEST] james% ls *TEST4* *TEST2*
ls: No match.
while bash will just return an list of tokens to the command, leading to oddities such as this:
betelgeuse:TEST james$ ls *TEST4* *TEST2*
TEST1 TEST3
*TEST4* and *TEST2* both return nothing; so the ls command believes it has been called with no arguments and just gives a listing of the directory. In your example, tar would (rightly) complain that it's being given no files to work on.
try this
shopt -s nullglob
tar cf test.tar *TEST* 2>/dev/null
newer version of Solaris comes with bash already. So you can try using bash. If your version of Solaris doesn't have bash, I guess you just have to do it the "longer" way.
for file in *TEST*
do
tar uf test.tar "$file"
done
Enclose your arguments in single quotes or escape them:
tar cf test.tar '*TEST*' '*TEST2*'
or
tar cf test.tar \*TEST\* \*TEST2\*
Include the string in single quotes:
tar cf test.tar '*TEST2*'
Put * into single quotes, i.e. '*'
As answered by others, the immediate solution is to put * inside single-quoted string or to escape it with a backslash.
Anyway, using a star character in a filename, though not forbidden, is a very bad idea.