I'm a bit confused with printing a variable that contain a new line symbol in bash.
var="Age:\n20\ncolor:\nred"
echo -e $var
Age:
20
color:
red
This is working, but a lot of people say that echo with options is not portable and it is better to use printf.
I never used prinf. According to manuals to emitate echo command:
printf '%s\n' "$var"
Age:\n20\ncoloe:\nred
But this doesn't parse \n inside variable. manuals usually have this example:
printf "Surname: %s\nName: %s\n" "$SURNAME" "$LASTNAME"
But it's not my case and from my point of view it not comfortable to use. I found out simply by typing that I can use this:
printf "$var\n"
Is it portable?
If I then pass $var to a mail command will it save new line breaks?
printf "$var\n" | mail -s subj email#domain.com
printf's %b format specifier was meant specifically to replace echo -e (actually, the XSI extension to echo which calls for special interpretation of the arguments by default. -e was never specified and is disallowed by POSIX.), and is identical in virtually every way including a few differences from $'...' and the format string argument to printf.
$ ( var='Age:\n20\ncolor:\nred'; printf '%b\n' "$var" )
Age:
20
color:
red
You should generally avoid expanding variables into the format string unless your program controls the exact value and it is intended specifically to be a format string. Your last example in particular has the potential to be quite dangerous in Bash due to printf's -v option.
# Bad!
var='-v_[$(echo "oops, arbitrary code execution" >&2)0]'
printf "$var" foo
It is usually good practice to avoid %b unless you have a special portability requirement. Storing the escape codes in a variable instead of the literal data violates principles of separation of code and data. There are contexts in which this is ok, but it is usually better to assign the the value using $'...' quoting, which is specified for the next version of POSIX, and has long been available in Bash and most ksh flavours.
x=$'foo\nbar'; printf '%s\n' "$x" # Good
x=(foo bar); printf '%s\n' "${x[#]}" # Also good (depending on the goal)
x='foo\nbar'; printf '%b\n' "$x" # Ok, especially for compatibility
x='foo\nbar'; printf -- "$x" # Avoid if possible, without specific reason
http://wiki.bash-hackers.org/commands/builtin/printf
Related
Seems that the recommended way of doing indirect variable setting in bash is to use eval:
var=x; val=foo
eval $var=$val
echo $x # --> foo
The problem is the usual one with eval:
var=x; val=1$'\n'pwd
eval $var=$val # bad output here
(and since it is recommended in many places, I wonder just how many scripts are vulnerable because of this...)
In any case, the obvious solution of using (escaped) quotes doesn't really work:
var=x; val=1\"$'\n'pwd\"
eval $var=\"$val\" # fail with the above
The thing is that bash has indirect variable reference baked in (with ${!foo}), but I don't see any such way to do indirect assignment -- is there any sane way to do this?
For the record, I did find a solution, but this is not something that I'd consider "sane"...:
eval "$var='"${val//\'/\'\"\'\"\'}"'"
A slightly better way, avoiding the possible security implications of using eval, is
declare "$var=$val"
Note that declare is a synonym for typeset in bash. The typeset command is more widely supported (ksh and zsh also use it):
typeset "$var=$val"
In modern versions of bash, one should use a nameref.
declare -n var=x
x=$val
It's safer than eval, but still not perfect.
Bash has an extension to printf that saves its result into a variable:
printf -v "${VARNAME}" '%s' "${VALUE}"
This prevents all possible escaping issues.
If you use an invalid identifier for $VARNAME, the command will fail and return status code 2:
$ printf -v ';;;' '%s' foobar; echo $?
bash: printf: `;;;': not a valid identifier
2
eval "$var=\$val"
The argument to eval should always be a single string enclosed in either single or double quotes. All code that deviates from this pattern has some unintended behavior in edge cases, such as file names with special characters.
When the argument to eval is expanded by the shell, the $var is replaced with the variable name, and the \$ is replaced with a simple dollar. The string that is evaluated therefore becomes:
varname=$value
This is exactly what you want.
Generally, all expressions of the form $varname should be enclosed in double quotes, to prevent accidental expansion of filename patterns like *.c.
There are only two places where the quotes may be omitted since they are defined to not expand pathnames and split fields: variable assignments and case. POSIX 2018 says:
Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.
This list of expansions is missing the parameter expansion and the field splitting. Sure, that's hard to see from reading this sentence alone, but that's the official definition.
Since this is a variable assignment, the quotes are not needed here. They don't hurt, though, so you could also write the original code as:
eval "$var=\"the value is \$val\""
Note that the second dollar is escaped using a backslash, to prevent it from being expanded in the first run. What happens is:
eval "$var=\"the value is \$val\""
The argument to the command eval is sent through parameter expansion and unescaping, resulting in:
varname="the value is $val"
This string is then evaluated as a variable assignment, which assigns the following value to the variable varname:
the value is value
The main point is that the recommended way to do this is:
eval "$var=\$val"
with the RHS done indirectly too. Since eval is used in the same
environment, it will have $val bound, so deferring it works, and since
now it's just a variable. Since the $val variable has a known name,
there are no issues with quoting, and it could have even been written as:
eval $var=\$val
But since it's better to always add quotes, the former is better, or
even this:
eval "$var=\"\$val\""
A better alternative in bash that was mentioned for the whole thing that
avoids eval completely (and is not as subtle as declare etc):
printf -v "$var" "%s" "$val"
Though this is not a direct answer what I originally asked...
Newer versions of bash support something called "parameter transformation", documented in a section of the same name in bash(1).
"${value#Q}" expands to a shell-quoted version of "${value}" that you can re-use as input.
Which means the following is a safe solution:
eval="${varname}=${value#Q}"
Just for completeness I also want to suggest the possible use of the bash built in read. I've also made corrections regarding -d'' based on socowi's comments.
But much care needs to be exercised when using read to ensure the input is sanitized (-d'' reads until null termination and printf "...\0" terminates the value with a null), and that read itself is executed in the main shell where the variable is needed and not a sub-shell (hence the < <( ... ) syntax).
var=x; val=foo0shouldnotterminateearly
read -d'' -r "$var" < <(printf "$val\0")
echo $x # --> foo0shouldnotterminateearly
echo ${!var} # --> foo0shouldnotterminateearly
I tested this with \n \t \r spaces and 0, etc it worked as expected on my version of bash.
The -r will avoid escaping \, so if you had the characters "\" and "n" in your value and not an actual newline, x will contain the two characters "\" and "n" also.
This method may not be aesthetically as pleasing as the eval or printf solution, and would be more useful if the value is coming in from a file or other input file descriptor
read -d'' -r "$var" < <( cat $file )
And here are some alternative suggestions for the < <() syntax
read -d'' -r "$var" <<< "$val"$'\0'
read -d'' -r "$var" < <(printf "$val") #Apparently I didn't even need the \0, the printf process ending was enough to trigger the read to finish.
read -d'' -r "$var" <<< $(printf "$val")
read -d'' -r "$var" <<< "$val"
read -d'' -r "$var" < <(printf "$val")
Yet another way to accomplish this, without eval, is to use "read":
INDIRECT=foo
read -d '' -r "${INDIRECT}" <<<"$(( 2 * 2 ))"
echo "${foo}" # outputs "4"
Seems that the recommended way of doing indirect variable setting in bash is to use eval:
var=x; val=foo
eval $var=$val
echo $x # --> foo
The problem is the usual one with eval:
var=x; val=1$'\n'pwd
eval $var=$val # bad output here
(and since it is recommended in many places, I wonder just how many scripts are vulnerable because of this...)
In any case, the obvious solution of using (escaped) quotes doesn't really work:
var=x; val=1\"$'\n'pwd\"
eval $var=\"$val\" # fail with the above
The thing is that bash has indirect variable reference baked in (with ${!foo}), but I don't see any such way to do indirect assignment -- is there any sane way to do this?
For the record, I did find a solution, but this is not something that I'd consider "sane"...:
eval "$var='"${val//\'/\'\"\'\"\'}"'"
A slightly better way, avoiding the possible security implications of using eval, is
declare "$var=$val"
Note that declare is a synonym for typeset in bash. The typeset command is more widely supported (ksh and zsh also use it):
typeset "$var=$val"
In modern versions of bash, one should use a nameref.
declare -n var=x
x=$val
It's safer than eval, but still not perfect.
Bash has an extension to printf that saves its result into a variable:
printf -v "${VARNAME}" '%s' "${VALUE}"
This prevents all possible escaping issues.
If you use an invalid identifier for $VARNAME, the command will fail and return status code 2:
$ printf -v ';;;' '%s' foobar; echo $?
bash: printf: `;;;': not a valid identifier
2
eval "$var=\$val"
The argument to eval should always be a single string enclosed in either single or double quotes. All code that deviates from this pattern has some unintended behavior in edge cases, such as file names with special characters.
When the argument to eval is expanded by the shell, the $var is replaced with the variable name, and the \$ is replaced with a simple dollar. The string that is evaluated therefore becomes:
varname=$value
This is exactly what you want.
Generally, all expressions of the form $varname should be enclosed in double quotes, to prevent accidental expansion of filename patterns like *.c.
There are only two places where the quotes may be omitted since they are defined to not expand pathnames and split fields: variable assignments and case. POSIX 2018 says:
Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.
This list of expansions is missing the parameter expansion and the field splitting. Sure, that's hard to see from reading this sentence alone, but that's the official definition.
Since this is a variable assignment, the quotes are not needed here. They don't hurt, though, so you could also write the original code as:
eval "$var=\"the value is \$val\""
Note that the second dollar is escaped using a backslash, to prevent it from being expanded in the first run. What happens is:
eval "$var=\"the value is \$val\""
The argument to the command eval is sent through parameter expansion and unescaping, resulting in:
varname="the value is $val"
This string is then evaluated as a variable assignment, which assigns the following value to the variable varname:
the value is value
The main point is that the recommended way to do this is:
eval "$var=\$val"
with the RHS done indirectly too. Since eval is used in the same
environment, it will have $val bound, so deferring it works, and since
now it's just a variable. Since the $val variable has a known name,
there are no issues with quoting, and it could have even been written as:
eval $var=\$val
But since it's better to always add quotes, the former is better, or
even this:
eval "$var=\"\$val\""
A better alternative in bash that was mentioned for the whole thing that
avoids eval completely (and is not as subtle as declare etc):
printf -v "$var" "%s" "$val"
Though this is not a direct answer what I originally asked...
Newer versions of bash support something called "parameter transformation", documented in a section of the same name in bash(1).
"${value#Q}" expands to a shell-quoted version of "${value}" that you can re-use as input.
Which means the following is a safe solution:
eval="${varname}=${value#Q}"
Just for completeness I also want to suggest the possible use of the bash built in read. I've also made corrections regarding -d'' based on socowi's comments.
But much care needs to be exercised when using read to ensure the input is sanitized (-d'' reads until null termination and printf "...\0" terminates the value with a null), and that read itself is executed in the main shell where the variable is needed and not a sub-shell (hence the < <( ... ) syntax).
var=x; val=foo0shouldnotterminateearly
read -d'' -r "$var" < <(printf "$val\0")
echo $x # --> foo0shouldnotterminateearly
echo ${!var} # --> foo0shouldnotterminateearly
I tested this with \n \t \r spaces and 0, etc it worked as expected on my version of bash.
The -r will avoid escaping \, so if you had the characters "\" and "n" in your value and not an actual newline, x will contain the two characters "\" and "n" also.
This method may not be aesthetically as pleasing as the eval or printf solution, and would be more useful if the value is coming in from a file or other input file descriptor
read -d'' -r "$var" < <( cat $file )
And here are some alternative suggestions for the < <() syntax
read -d'' -r "$var" <<< "$val"$'\0'
read -d'' -r "$var" < <(printf "$val") #Apparently I didn't even need the \0, the printf process ending was enough to trigger the read to finish.
read -d'' -r "$var" <<< $(printf "$val")
read -d'' -r "$var" <<< "$val"
read -d'' -r "$var" < <(printf "$val")
Yet another way to accomplish this, without eval, is to use "read":
INDIRECT=foo
read -d '' -r "${INDIRECT}" <<<"$(( 2 * 2 ))"
echo "${foo}" # outputs "4"
I don't seem to be able to assign a string that begins with "-e" or "-E" to a bash shell variable:
$ options="-e stuff"
$ echo $options
stuff
Other letters work fine:
$ options="-g stuff"
$ echo $options
-g stuff
What is the reason for this?
You should quote your variable:
echo "${options}"
otherwise it's being expanded to
echo -g stuff
which is being interpreter by echo as its -e option, which actually exists (see man echo), and that's why -e "did not work" while other letters you tried "did".
First: To reliably determine the value of a variable in bash, use declare -p, not echo. Thus:
declare -p options
will emit something like:
declare -- options="-e stuff"
This tells you much more than echo does:
Because it's declare -- rather than declare -x, you know that the variable is not exported.
Because it's not declare -a, you know it's not giving you an array (echo "$array" will print only the first element of a shell array and ignore the rest).
Because it's not declare -i, you know the value wasn't declared to be an integer... etc.
If you're only worried about the string case, but want to ensure that you get a printable value no matter which version of bash is in use (as some historical releases will not always guarantee printable escaping for values printed with declare -p), consider instead:
printf '%q=%q\n' options "$options"
...which will emit unambiguous output even if there are cursor control characters, newlines, or other non-textual contents in your string.
Follow the advice of the POSIX specification for echo, and use printf instead. To quote the APPLICATION USAGE section in full, emphasis added, noting that in bash, -e enables XSI-style interpretation of escape sequences:
It is not possible to use echo portably across all POSIX systems
unless both -n (as the first argument) and escape sequences are
omitted.
The printf utility can be used portably to emulate any of the
traditional behaviors of the echo utility as follows (assuming that
IFS has its standard value or is unset):
The historic System V echo and the requirements on XSI implementations
in this volume of POSIX.1-2008 are equivalent to:
printf "%b\n" "$*"
The BSD echo is equivalent to:
if [ "X$1" = "X-n" ]
then
shift
printf "%s" "$*"
else
printf "%s\n" "$*"
fi
New applications are encouraged to use printf instead of echo.
So, how does this apply to you? Since you want -e to be treated as data, not part of echo's setup, the BSD, non--n branch of that applies:
options="-e stuff"
printf '%s\n' "$options"
In bash, you can use printf "%q" to escape the special characters in a string. I've added a line break in the following examples for clarity's sake:
$ printf "%q\n" "foo"
foo
$ printf "%q\n" 'foo$bar'
foo\$bar
$ printf "%q\n" "foo bar" # Tab entered with Ctrl+V Tab
$'foo\tbar'
You can supply the -v option to printf to stick the output into a variable, rather than echoing to stdout.
Now what if I want to echo the original, unescaped string back to stdout? If I just do a simple echo, it includes all the meta/control characters; echo -e gets me slightly further, but not to a fully unescaped state.
Use printf and eval to "unescape" the string (it is perfectly safe, since it is quoted by bash already). echo is, in general, dangerous with arbitrary user input.
eval printf '%s\\n' "$var"
The argument to printf is %s\\n with double backslashes even within the single quotes, since eval will take it as one argument and then strip it (evaluate it) as is, so it effectively becomes something like: printf %s\\n $'12\n34 56' which is a perfectly safe and valid command (the variable even includes a newline and space, as you can see).
I would like to use echo in bash to print out a string of characters followed by only a carriage return. I've looked through the man page and have found that echo -e will make echo interpret backslash escape characters. Using that I can say echo -e 'hello\r' and it will print like this
$>echo -e 'hello\r'
hello
$>
So it looks like it handled the carriage return properly. I also found echo -n in the man page will stop echo from inserting a newline character and it looks like it works when I do this
$>echo -n 'hello\r'
hello\r$>
The problem I'm having is in combining both -e and -n. I've tried each of echo -e -n 'hello\r', echo -n -e 'hello\r', echo -en 'hello\r', and echo -ne 'hello\r' and nothing gets printed like so:
$>echo -ne 'hello\r'
$>
Is there something I'm missing here or can the -e and -n options not be used together?
I think it's working, you're just not seeing it. This is your command:
$> echo -ne 'hello\r'
Because of the carriage return (\r), that will leave the cursor at the start of the same line on the terminal where it wrote the hello - which means that's where the next thing output to the terminal will be written. So if your actual prompt is longer than the $> you show here, it will overwrite the hello completely.
This sequence will let you see what's actually happening:
echo -ne 'hello\r'; sleep 5; echo 'good-bye'
But for better portability to other shells, I would avoid using options on echo like that. Those are purely bashisms, not supported by the POSIX standard. The printf builtin, however, is specified by POSIX. So if you want to display strings with no newline and parsing of backslash sequences, you can just use it:
printf '%s\r' 'hello'
There are numerous different implementations of the echo command. There's one built into most shells (with different behavior for each), and the behavior of /bin/echo varies considerably from one system to another.
Rather than echo, use printf. It's built into bash as well as being available as an external command, and its behavior is much more consistent across implementations. (The major variation is that the GNU coreutils printf recognizes --help and --version options.)
Just use:
printf 'hello\r'
I'd like you to introduce you to printf.
OP, meet printf. printf. This is the OP...
Whenever you are trying to do anything unusual with output in BASH, you should switch to printf. In fact, I use printf all the time, so my scripts will run under both BASH and Kornshell.
Although BASH and Kornshell are 99% the same, the echo command is different. In Kornshell, it's deprecated, and you're supposed to use the builtin print. However, there's no print in BASH. Using printf solves the problem because it works (sort of) the same in both shells.
Since printf doesn't automatically end each line with a \n, you don't have to worry about how to prevent the \n from being appended. Heck, if you want a \n, you have to put it yourself. In your case:
printf `hello\r`
However, for safety reasons, you should really do this:
printf '%s\r' 'hello'
This is a habit I've gotten into when I'm using printf. Imagine if I want to print out $FOO, and I do this:
printf "$FOO\n"
That will normally work, but what if $FOO has a dash at the beginning, or has a % sign in it. The above won't do what I want to do. However, if I do this:
printf '%s\n' "$FOO"
This will print out $FOO no matter what $FOO has in it.
POSIX 7 says it is not possible:
If the first operand is -n, or if any of the operands contain a backslash character, the results are implementation-defined.
On Ubuntu 12.04 for example:
Bash's built-in echo echo -e '\n' interprets the newline, echo '\n' does not.
/bin/sh's built-in echo does not have the -e option, and interprets the newline even without it. This can byte when writing Makefiles, which use /bin/sh by default.
POSIX 7 solution: printf as the others mentioned.
Quoting. It's the wave of the future. Always quote.
echo -ne 'hello\r'
Use double quotes if you put a variable inside that you want expanded.