I'm almost certain the code I have here worked before. Here's a simplified version and what it produces:
a="atext"
b="btext"
var=$'${a}\n${b}\n'
printf "var=$var"
Which produces output:
var=${a}
${b}
The real code outputs var to file, but the variable expansions aren't happening for some reason.
If this can't work, can you suggest a nice alternative way, and why one uses $' '? Thanks.
GNU bash, version 4.3.42
$'' is a quoting type used to allow backslash escape sequences to describe literal strings with nonprintable characters and other such oddities. Thus, $'\n' evaluates to a single character -- a newline -- whereas '\n' and "\n" both evaluate to two characters, the first being a backslash and the second being an n.
If you want to have the exact behavior of your original code -- putting a literal newline between the results of two different expansions -- you can switch quote types partway through a string:
a="atext"
b="btext"
var="$a"$'\n'"$b"
printf '%s' "var=$var"
That is, right next to each other, with no spaces between:
"$a"
$'\n'
"$b"
This gives you $a and $b expanded, with a literal newline between them.
Why does this matter? Try the following:
$ a=atext
$ b=btext
$ var1="$a\n$b" # Assign with literal "\" and "n" characters
$ printf "$var1" # Here, printf changes the "\n" into the newline
atext
btext
$ printf '%s' "$var1" # ...but this form shows that the "\n" are really there
atext\nbtext
$ var2="$a"$'\n'"$b" # now, we put a single newline in the string
$ printf '%s' "$var2" # and now even accurate use of printf shows that newline
atext
btext
Just replace the single quotes with double quotes.
$ cat test
a="atext"
b="btext"
var=$"${a}\n${b}\n"
printf "var=$var"
$ sh test
var=atext
btext
For variable expansion you either need to use double quotes or no quotes. Single quotes negate expansion.
Related
I've noticed that to print the value of the $IFS variable in the shell I have to do something like:
$ printf '%s\nYour IFS is: %q' 'Hello' "$IFS"
Hello
Your IFS is: $' \t\n'
My question is why do I need to pass the IFS in that special way? For example, why (or why wouldn't) it be possible to do:
$ echo $IFS -- some parameter that prints special characters?
$ printf "$IFS" or $ printf '$IFS' -- why wouldn't either of these work?
Why $ printf "%q" $IFS and $ printf "%q" '$IFS' don't show this properly but $ printf "%q" "$IFS" does?
$ echo $IFS -- some parameter that prints special characters?
echo doesn't have such a parameter
$ printf "$IFS" or $ printf '$IFS' -- why wouldn't either of these work?
The first does interpolation of the string just like echo does and prints the IFS string just like it is - which by default is a bunch of whitespaces.
The second does not do interpolation and obviously prints $IFS.
printf "%q" $IFS
The variable has already been expanded into whitespaces that are eaten up by the shell so nothing is passed as a second parameter to printf and therefore %q has no input to work with.
printf "%q" '$IFS'
The string $IFS is passed as a parameter to %q which just adds escape characters to it.
You are running into three basic problems:
Unquoted variable references (as in echo $IFS and printf "%q" $IFS) undergo word splitting. Basically, any whitespace in the variable's value is treated as a separator between "words" (which get passed as arguments to the command). But "whitespace" is defined as the characters in $IFS (that's what it's for!). Therefore, the entire variable's value gets treated as just spacing, not actual content, and effectively vanishes!
This is one of the many examples of why you should put double-quotes around variable references.
Single-quoted strings (as in printf '$IFS' and printf "%q" '$IFS') don't undergo variable expansion at all. In these cases, $IFS is just a literal string, not a variable reference.
Finally, the default characters in $IFS aren't particularly visible on screen. printf "$IFS" actually does print them correctly (unless you put "%" or "\" in IFS for some reason), but you can't really see them. By default, the first character in $IFS is a space, so when that prints the cursor moves to the second column, but nothing visible appears. The second character is a tab, so the cursor moves even further over, but again nothing is actually visible. Then the last character is a newline, so the cursor moves to the beginning of the next line... and again, nothing visible appears.
This is why printf %q is needed -- the %q format converts the not-really-visible characters in $IFS into a visible, readable representation.
If you don't quote the variable, word-splitting is done after the variable is expanded. Word splitting treats all the characters in IFS as word delimiters; multiple of them in a row between words are collapsed into a single space, and if the output is entirely delimiters, nothing at all is output.
You need to use quotes around the variable to output it literally.
Then you need to use the %q format operator to output it as escape sequences so you can see what the individual characters are. Otherwise you wouldn't be able to tell that the second character is a TAB, you would just see a bunch of spaces on the screen. And the newline would just go to the next line.
In bash, you can use printf "%q" to escape the special characters in a string. I've added a line break in the following examples for clarity's sake:
$ printf "%q\n" "foo"
foo
$ printf "%q\n" 'foo$bar'
foo\$bar
$ printf "%q\n" "foo bar" # Tab entered with Ctrl+V Tab
$'foo\tbar'
You can supply the -v option to printf to stick the output into a variable, rather than echoing to stdout.
Now what if I want to echo the original, unescaped string back to stdout? If I just do a simple echo, it includes all the meta/control characters; echo -e gets me slightly further, but not to a fully unescaped state.
Use printf and eval to "unescape" the string (it is perfectly safe, since it is quoted by bash already). echo is, in general, dangerous with arbitrary user input.
eval printf '%s\\n' "$var"
The argument to printf is %s\\n with double backslashes even within the single quotes, since eval will take it as one argument and then strip it (evaluate it) as is, so it effectively becomes something like: printf %s\\n $'12\n34 56' which is a perfectly safe and valid command (the variable even includes a newline and space, as you can see).
I am trying to printf an input with variables and need double quotes. My output needs a \x and \f text output.
printf "\x"
or
printf "\\x"
produces the error:
./pegrep.in: line 94: printf: missing hex digit for \x
while
printf "\f"
or
printf "\\f"
produces nothing at all (in a text output I believe it creates ^L)
single quotes however works for x (but not f). I tried enclosing
printf "...'\\x'..."
and got the same error as standard double quotes. Have also tried /// to no avail.
First, let's understand how quoting works.
The simplest form of quoting is the backslash: it prevents the following character from being interpreted in any special way, allowing it to be used as a literal character. For example:
# Two arguments to printf
$ printf '%s\n' a b
a
b
# One three-character argument to printf
printf '%s\n' a\ b
a b
Double quotes are equivalent to escaping every character contained therein: "a b c" is equivalent to \a\ \b\ \c, which is the same as a\ b\ c because a, b, and c have no special meaning to begin with. You can think of every character inside double quotes as being treated literally, with the following exceptions:
$ can start a parameter expansion or a command substitution. Escape it with a backslash to treat it literally.
$ foo=3
$ echo "$foo"
3
$ echo "\$foo"
$foo
A backquote starts a command substitution. Escape it with a backslash to treat it literally.
$ echo "`pwd`"
/home/me
$ echo "\`pwd\`"
`pwd`
A double quote ends a quoted string. Escape it with a backslash to treat it literally.
$ echo "\""
"
Because a backslash might be part of one of the three preceding sequences, escape it with a backslash to treat one literally.
$ echo "\\"
\
Inside single quotes, everything is treated literally, including any use of a backslash. (One consequence of this is that it is impossible to put a single quote in a single-quoted string, because it will always terminate such a string.)
$ echo '\'
\
Once you understand quoting, you next need to realize that printf itself can process whatever backslashes it sees in its first argument.
$ printf 'a\x20b\n' # a, space (ASCII 0x20), b, newline
a b
In order to ensure a string is printed literally no matter what characters are present, use the %s format specifier and pass your string as a second argument:
$ printf '\x41\n'
A
$ printf '%s\n' '\x41'
\x41
I think you may be confusing the \ and the % characters. The % is used to output formatted variable contents. I think this when \f is outputs ^L (aka ASCII FF) that escape sequence is working as documented.
The printf argument can be quoted with single quotes. Then you can use double quotes within it and they will be part of the output:
printf '"%x"' 4011 # output: "fab"
Similarly for floating point:
print '"%f"' 2.2 # output: "2.200000"
The \ character is for escape sequences of otherwise unprintable characters.
See man printf for a full list of the % format characters and their meanings.
I've noticed something weird:
Y=""
echo ${Y:-"\n"}
echo "${Y:-"\n"}"
prints
\n
n
Why is the second line n, not \n? Is this a bug?
It looks as if Bash parsed this as a concatenation of two quoted strings with an unquoted string in between ("${Y:-" and \n and "}") but this doesn't seem to be the case since the commands
echo $(echo "\n")
echo "$(echo "\n")"
echo "${Y:-"'\n'"}"
output
\n
\n
'n'
I'm using GNU bash, version 4.3.11.
I suspect there is a bug in the handling of the word following :- (in fact, I seem to recall reading something about this, but I can't recall where).
If the value is not quoted, I get results I would expect...
$ echo ${Y:-\n}
n
$ echo "${Y:-\n}"
\n
This is also the result you get in dash (ignoring the fact that dash actually produces a literal newline since POSIX mandates that echo should process escaped characters, something bash only does if you use the non-standard -e option.)
In this example, quoting the default value preserves the backslash. As the result of the parameter expansion produces the backslash, quote removal does not remove it.
$ echo ${Y:-"\n"} # Equivalent to echo "\n", so the output makes sense
\n
There doesn't seem to be any reason for bash to behave different in this final example just because the entire parameter expansion is being quoted. It is almost as if quote removal is being applied twice, once to remove the outer double quotes and again to incorrectly remove the backslash.
# Quote removal discards the backslash: OK
$ echo \n
n
# Quote removal discards the double quotes: OK
$ echo "n"
n
# Quote removal discards the first backslash after `\\` is recognized
# as a quoted backslash: OK
$ echo \\n
\n
# Quote removal discards the double quotes, but leaves
# backslash: OK
$ echo "\n"
\n
# Is quote removal discarding both the double quotes *and* the backslash? Not OK
$ echo "${Y:-"\n"}"
n
Related, zsh (with the bsd_echo) option set outputs \n, not n.
% Y=""
% echo "${Y:-"\n"}"
\n
To complement chepner's helpful answer:
Here's an overview of how the major POSIX-like shells handle the following command:
Y=""
printf '%s\n' ${Y:-"\n"} ${Y:-'\n'} "${Y:-"\n"}" "${Y:-'\n'}"
Note that I've added variations with single quotes.
dash [v0.5.8]
\n
\n
\n
'\n'
zsh [v5.0.8]
\n
\n
\n
'\n'
bash [v4.3.42]
\n
\n
n
'\n'
ksh [93u+]
\n
\n
n
'\n'
Curiously, in all shells, '\n' inside "..." preserves the single quotes, while removing them in the unquoted case.
With respect to "\n", both bash and ksh exhibit the oddity uncovered by the OP, while dash and zsh do not.
Maybe I'm looking at this wrong, but I don't seen any inconsistency in the assignment with the default value Y, quoted or unquoted. The echo expression in each case boils down to:
$ echo "\n"
\n
$ echo ""\n""
n
In the first case you have the quoted string "\n", in the second, you have a bare \n (which is simply n)
I have looked at this question, but it does not cover my use case.
Suppose I have the variable foo which holds the four-character literal \x60.
I want to perform ANSI C Quoting on the contents of this variable and store it into another variable bar.
I tried the following, but none of them achieved the desired effect.
bar=$'$foo'
echo $bar
bar=$"$foo"
echo $bar
Output:
$foo
\x61
Desired output (actual value of \x61):
a
How might I achieve this in the general case, including non-printable characters? Note that in this case a was used just as an example to make it easier to test whether the method worked.
By far the simplest solution, if you are using bash:
printf %b "$foo"
Or, to save it in another variable name bar:
printf -v bar %b "$foo"
From help printf:
In addition to the standard format specifications described in printf(1)
and printf(3), printf interprets:
%b expand backslash escape sequences in the corresponding argument
%q quote the argument in a way that can be reused as shell input
%(fmt)T output the date-time string resulting from using FMT as a format
string for strftime(3)
There are edge cases, though:
\c terminates output, backslashes in \', \", and \? are not removed,
and octal escapes beginning with \0 may contain up to four digits
The following works:
eval bar=\$\'$x\'
The command bar=$'\x61' has to be constructed first, then eval evaluates the newly built command.
I just found out that I can do this. Edited based on comments.
bar=$( echo -ne "$foo" )
The best method I know is
y=$(printf $(echo "$foo"|sed 's/%/%%/g'))
As mentioned in the comments, this trims trailing newlines from $foo. To overcome this:
moo=$(echo "${foo}:end"|sed 's/%/%%/g')
moo=$(printf "$moo")
moo=${moo%:end}
# the escaped string is in $moo
echo "+++${moo}---"
Since bash 4.4 there is a variable expansion to do exactly that:
$ foo='\x61'; echo "$foo" "${foo#E}"
\x61 a
To set another variable use:
$ printf -v bar "${foo#E}"; echo "$bar"
a
Sample of conversion via shell. Problem, the code is octal using \0nnn and hexdecimal (not on all shell) using \xnn (where n are [hexa]digit)
foo="\65"
print "$( echo "$foo" | sed 's/\\/&0/' )"
5
with awk, you could certainly convert it directly