how to force to expand a concatenation of $path/* - bash

I'm writing a script to remove some files, but I don't understand how asterisk expansions work. These are my attempts to solve my problem:
rm "$path"*.txt
rm "$path"/*.txt
rm "$path"{*}.txt
rm "$path"'*'
rm "/folder/folder\ with\ spaces/*.txt"
I also tried replacing double quotes (") with single quotes(') and backticks (`). After every script computation, I get an error because the * is not substitute. So now I have two questions:
Why is the asterisk not expanded?
What's the difference between the different quoting character (` " ' ...) ?

In single quoted nothing interesting happns. Not even $-variable expansion. Some of those you've tried should work (some depending on the variable content). And, really, * is most likely to be not expanded if there're no matches. Are you sure you got your names right?

Related

How can I confirm whether whitespace or special characters are escaped in a wildcard pattern?

I know that when you use a for loop in Bash, the items that you loop through are separated using the $IFS variable.
However, if I run the following commands, I correctly show the two files I have created - even though they have spaces:
touch file\ {1..2}.txt
for file in *.txt; do
echo "Found: ${file}"
done
The output is:
Found: file 1.txt
Found: file 2.txt
I am assuming that this is because when the shell sees the wildcard pattern, it expands it and escapes any special characters or whitespace. This is in contrast to if I run:
touch file\ {1..2}.txt
files=$(ls *.txt)
for file in $files; do
echo "Found: ${file}"
done
This results in:
Found: file
Found: 1.txt
Found: file
Found: 2.txt
Which makes sense - by default $IFS contains whitespace, so the file names are split.
What I want to understand is:
Am I correct that wildcard expansion results in a set of strings that contain escaped special characters
Where is it documented that this is the case, if I am correct?
Is there any way to show that this is happening?
I was hoping I could use something like set -x to show what the wildcard expands to and actually see the escaped characters, because I really want to be able to understand what is going on here.
I am writing a long series of articles on effective shell usage (effective-shell.com) and I'm struggling to find a way to explain the differences of behaviour here, I'm assuming that the shell is escaping characters but I'd like to know if this is the case and how to see it if possible!
Thanks in advance.
done
Am I correct that wildcard expansion results in a set of strings that contain escaped special characters
No. There is no need for the shell to escape special characters at that point, because filename expansion is the last word expansion to be performed; strings resulting from it are not subjected to word splitting or any other expansion; they stay as-is. This is documented in the manual as follows:
The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion); word splitting; and filename expansion.

Why isn't bash splitting fields inside $() operator?

Just for the sake of learning, while I was trying a bunch of stuff, I noticed a bash behavior that I couldn't logically explain.
$ ls $(echo a)
ls: cannot access 'a': No such file or directory
$ ls $($'\145\143\150\157\40\141')
echo a: command not found
$ ls $($"\145\143\150\157\40\141")
\145\143\150\157\40\141: command not found
In the first case, echo a was being evaluated to a which becomes an argument to ls. Fair enough. However, in the second case, the octal encoded string was being evaluated to echo a as expected; but the entire string was being treated as the command by $(). Moreover, in the third case, no expansion is taking place with double quotes. Why don't the fields split? I guess it has got something to do with field splitting in bash. But, I failed to explain what could the exact problem be. Is there any way I can make the field splitting work so that it gets treated like the first case?
A word of the form $'string' is expanded to a single-quoted string, as if the dollar sign had not been present. That means,
$($'\145\143\150\157\40\141')
is the same as
$('echo a')
And single-quoted strings don't undergo word splitting or any other kind of expansion. See Bash Reference Manual § ANSI-C Quoting.
Is there any way I can make the field splitting work so that it gets treated like the first case?
$ ls $(eval $'\145\143\150\157\40\141')
ls: cannot access 'a': No such file or directory
This is inadvisable though, see BashFAQ#048.
Concerning $"string" syntax, see Bash Reference Manual § Locale-Specific Translation, it's whole another thing.

b2 command finds ICU when called directly but not from indirect bash script variable

I have this strange issue with my bash script. I compile boost as part of it. The call from the script looks like this:
./b2 --reconfigure ${PARALLEL} link=static cxxflags=-fPIC install boost.locale.iconv=off boost.locale.posix=off -sICU_PATH="${ICU_PREFIX}" -sICU_LINK="${BOOST_ICU_LIBS}" >> "${BOOST_LOG}" 2>&1
That command works perfectly well. The log file shows that it finds ICU without a problem. However, if I change it to run from a variable, it no longer finds ICU (but it still compiles everything else):
bcmd="./b2 --reconfigure ${PARALLEL} link=static cxxflags=-fPIC install boost.locale.iconv=off boost.locale.posix=off -sICU_PATH=\"${ICU_PREFIX}\" -sICU_LINK=\"${BOOST_ICU_LIBS}\""
$bcmd >> "${BOOST_LOG}" 2>&1
What's the difference? I would like to be able to use the second approach so that I can pass the command into another function before running it.
Don't use a variable to store complex commands involving quotes that are nested. The problem is when you call the variable with just $cmd, the quotes are stripped incorrectly. Putting commands (or parts of commands) into variables and then getting them back out intact is complicated.
Quote removal is part of the one of the word expansions done by the shell. From the excerpt seen in POSIX specification of shell
2.6.7 Quote Removal
The quote characters ( backslash, single-quote, and double-quote) that were present in the original word shall be removed unless they have themselves been quoted.
Your example can be simply reproduced by a simple example. Assuming you have a few command flags (not actual ones)
cmdFlags='--archive --exclude="foo bar.txt"'
If you carefully look through the above, it contains 2 args, one --archive and another for --exclude="foo bar.txt", notice the double-quotes which needs to be preserved when you are passing it.
Notice how the quotes are incorrectly split when I don't quote cmdFlags, in the printf() call below
printf "'%s' " $cmdFlags; printf '\n'
'--archive' '--exclude="foo' 'bar.txt"'
and compare the result with one with proper quoting done below.
printf "'%s' " "$cmdFlags"; printf '\n'
'--archive --exclude="foo bar.txt"'
So along with the suggestion of properly quoting the variable, the general suggestion would be to use an array to store the flags and pass the quoted array expansion
cmdArray=()
cmdArray=(./b2 --reconfigure ${PARALLEL} link=static cxxflags=-fPIC install boost.locale.iconv=off boost.locale.posix=off -sICU_PATH="${ICU_PREFIX}" -sICU_LINK="${BOOST_ICU_LIBS}")
and pass the array as
"${cmdArrray[#]}" >> "${BOOST_LOG}" 2>&1
Try to use eval when you want to execute a string as a command. This way, you won't have issues regarding strings that have spaces etc. The expanded cmd string is not re-evaluated by bash hence, things like "hi there" are expanded as two separate tokens.
eval "$bcmd" >> "${BOOST_LOG}" 2>&1
To demonstrate this behavior, consider this code:
cmd='echo "hi there"'
$cmd
eval "$cmd"
Which outputs to:
"hi there"
hi there
The token "hi there" is not re-evaluated as a quoted string.
Use single quota instead of two.
bcmd='./b2 --reconfigure ${PARALLEL} link=static cxxflags=-fPIC install boost.locale.iconv=off boost.locale.posix=off -sICU_PATH=\"${ICU_PREFIX}\" -sICU_LINK=\"${BOOST_ICU_LIBS}\"'
Bash documentation states that single quota does not interpolate:
3.1.2.2 Single Quotes
Enclosing characters in single quotes (') preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.
That way you omit double-quota stripping and removing problem, and your string will be passed correctly. If you want to stay with double quotes, you have to vary that they DO NOT preserve literal value of certain characters: $, ' and \ unless preceded with \, as manual states:
3.1.2.3 Double Quotes
Enclosing characters in double quotes (") preserves the literal value of all characters within the quotes, with the exception of $, `, \, and, when history expansion is enabled, !. The characters $ and ` retain their special meaning within double quotes (see Shell Expansions). The backslash retains its special meaning only when followed by one of the following characters: $, `, ", \, or newline. Within double quotes, backslashes that are followed by one of these characters are removed. Backslashes preceding characters without a special meaning are left unmodified. A double quote may be quoted within double quotes by preceding it with a backslash.
In your example you forgot to mark $ with backslash as well. Difference between these two is explained perfectly by Adam here: Differences between single and double quota

Characters \$ in echo

I need to have in a string the text "\$CONDITIONS". I tried used:
> echo "\$CONDITIONS"
$CONDITIONS
> echo "\\$CONDITIONS"
\
Could you help me? What should I enter in echo command to get
\$CONDITIONS
as a result?
echo doesn't do anything except print exactly the string you pass in. The trick is to know enough about the shell to be able to pass in the string you want.
If you don't need the shell to perform substitutions on the value, simply use single quotes instead.
echo '\$CONDITIONS'
If you absolutely need to use double quotes, you can still single-quote individual parts of the string. Single quotes adjacent to double quotes will get pasted together into a single string before the shell passes it on.
echo '\$'"CONDITIONS"
Good old echo is slightly tired; you might also want to consider printf which is somewhat more versatile.
printf "\x5c\x24CONDITIONS\n"
(I'd normally use single quotes here as well; the double quotes are just to demonstrate that this works even with double quotes. But be careful with the backslashes; these happen to work even with single backslashes, but often they will need to be doubled if you want literal backslashes inside double quotes.)
To review what happened in your failed attempts,
echo "\$CONDITIONS" # produces $CONDITIONS
the backslash properly escapes the dollar sign from the shell, and is removed as part of the process. So you are saying, a literal dollar sign, and the text CONDITIONS.
echo "\\$CONDITIONS" # produces \
Here, the backslash similarly escapes the backslash, and the shell expands the variable $CONDITIONS which is unset or empty.
echo "\\\$CONDITIONS"
Well, this works, but it's ugly. There is a backslash-escaped backslash, and a backslash-escaped dollar sign, and the text CONDITIONS.
Backslashes and dollar signs (and backticks `) don't get processed inside single quotes, so that's what you should usually use if your string contains any of these (and more generally, if you don't specifically require the shell to handle these constructs).
Backslashes are kind of tricky inside double quotes. The shell will remove the ones it processes (so \$ gets turned into just $) but retain the ones it doesn't actually do anything with (so \x is preserved as \x inside double quotes). Without quotes, the behavior is different again. (Not even going into that rabbithole. Just use quotes.)
Use \\\ to achieve this.
#!/bin/bash
echo "\\\$CONDITION" # prints \$CONDITION

How do I properly escape data for a Makefile?

I'm dynamically generating config.mk with a bash script which will be used by a Makefile. The file is constructed with:
cat > config.mk <<CFG
SOMEVAR := $value_from_bash1
ANOTHER := $value_from_bash2
CFG
How do I ensure that the generated file really contains the contents of $value_from_bash*, and not something expanded / interpreted? I probably need to escape $ to $$ and \ to \\, but are there other characters that needs to be escaped? Perhaps there is a special literal assignment I've not heard of?
Spaces seems to be troublesome too:
$ ls -1
a b
a
$ cat Makefile
f := a b
default_target:
echo "$(firstword $(wildcard ${f}))"
$ make
a
If I use f := a\ b it works (using quotes like f := 'a b' did not work either, makefile just treats it as a regular character)
Okay, it turned out that Makefiles need little escaping for itself, but the commands which are executed by the shell interpreter need to be escaped.
Characters which have a special meaning in Makefile and that need to be escaped are:
sharp (#, comment) becomes \#
dollar ($, begin of variable) becomes $$
Newlines cannot be inserted in a variable, but to avoid breaking the rest of the Makefile, prepend it with a backslash so the line break will be ignored.
Too bad a backslash itself cannot be escaped (\\ will still be \\ and not \ as you might expect). This makes it not possible to put a literal slash on the end of a string as it will either eat the newline or the hash of a following comment. A space can be put on the end of the line, but that'll also be put in the variable itself.
The recipe itself is interpreted as a shell command, without any fancy escaping, so you've to escape data yourself, just imagine that you're writing a shellscript and inserting the variables from other files. The strategy here would be putting the variables between single quotes and escape only ' with '\'' (close the string, insert a literal ' and start a new string). Example: mornin' all becomes 'morning'\'' all' which is equivalent to "morning' all".
The firstword+wildcard issue is caused by the fact that filenames with spaces in them are treated as separate filenames by firstword. Furthermore, wildcard expands escapes using \ so x\ y is matches as one word, x y and not two words.
It seems that the full answer to this question is found nowhere on the internet, so I finally sat down and figured it out for the Windows case.
Specifically, the "Windows case" refers to file names that are valid in Windows, meaning that they do not contain the characters \, /, *, ?, ", ^, <, >, |, or line breaks. It also means \ and / are both considered valid directory separators for the purposes of Make.
An example will clear it up better than I can explain. Basically, if you are trying to match this file path:
Child\a$b {'}(a.o#$#,&+=~`),[].c
Then you have to write these rules:
all: Child\\a$$b\\\ \\\ {'}(a.o\#$$#,&+=~`),[].o
%.o: %.c
$(CC) '$(subst ','"'"',$(subst \,,$(subst \\,/,$+)))'
Stare at it for a long time and it'll sort of start making some remote sense.
This works in my MSYS2 environment, so I presume it is correct.
I don't see how that makefile can work as you say. A pattern rule cannot be the default.
You're missing a `$` in `$(wildcard ...)`, so I think you haven't posted what you're really testing.
You should escape newlines too.

Resources