This question already has answers here:
Command to escape a string in bash
(5 answers)
Closed 4 years ago.
Is there a command that not just echos it's argument but also escapes them if needed (e.g. if a argument contains white space or a special character)?
I'd need it in some shell magic where instead of executing a command in one script I echo the command. This output gets piped to a python script that finally executes the commands in a more efficient manner (it loads the main() method of the actual target python script and executes it with the given arguments and an additional parameter by witch calculated data is cached between runs of main()).
Instead of that I could of course port all the shell magic to python where I wouldn't need to pipe anything.
With bash, the printf builtin has an additional format specifier %q, which prints the corresponding argument in a friendly way:
In addition to the standard printf(1) formats, %b causes printf to expand backslash escape sequences in the corresponding argument (except that \c terminates output, backslashes in \', \", and \? are not removed, and octal escapes beginning with \0 may contain up to four digits), and %q causes printf to output the corresponding argument in a format that can be reused as shell input.
So you can do something like this:
printf %q "$VARIABLE"
printf %q "$(my_command)"
to get the contents of a variable or a command's output in a format which is safe to pass in as input again (i.e. spaces escaped). For example:
$ printf "%q\n" "foo bar"
foo\ bar
(I added a newline just so it'll be pretty in an interactive shell.)
Related
This question already has answers here:
Printf example in bash does not create a newline
(9 answers)
Closed 3 years ago.
This is in bash(5.0.3(1)-release) / ubuntu 19
When I ran :
printf "%s" $'\n'
It printed out a new line.
Now when I run:
result="$(printf "%s" $'\n')"
printf "<%s>" "$result"
I expect $result contains a newline, but it's empty.
Can someone explain ?
From posix shell manual, from chapter about process substitution, emphasis mine:
The shell shall expand the command substitution by executing command in a subshell environment (see Shell Execution Environment) and replacing the command substitution (the text of command plus the enclosing "$()" or backquotes) with the standard output of the command, removing sequences of one or more <newline>s at the end of the substitution.
It's impossible to store trailing newlines with process substitution. You can use for example printf -v bash extension:
printf -v result "%s" $'\n'
For most portability, I go with encoding the string in hex using od or xxd and store the string in hex. Or just store the resulting string in a file.
This question already has answers here:
How to avoid bash command substitution to remove the newline character?
(3 answers)
Closed 3 years ago.
I'm dealing with a while loop processing a Herestring input:
one_is_achieved=0
while IFS="=" read key value ; do
process_entry key value && one_is_achieved=1
done <<<$(output_entries)
# Dealing with one_is_achieved
[...]
the function output_entries() outputs key-values lines:
k1=some stuff
k2=other stuff
k3=remaining stuff
Problem, the command subtitution $() captures stdout but replace ends of line by spaces, causing k1 entry to be some stuff k2=other stuff k3= remaining stuff.
Is there a way to prevent the replacement or to restore the ends of lines?
Note: I solved my problem this way:
while IFS="=" read key value ; do
process_entry key value && one_is_achieved=1
done < <(output_entries)
But I think the original question is still relevant (preserving or restoring end of lines).
It is how command substitution is implemented in bash shell. The $(..) in unquoted form removes any embedded newlines because of the default word-splitting done by the shell. The default IFS value is $' \t\n' and unquoted expansion causes the splitting to happen based on any of these 3 characters.
You need to quote the substitution to prevent the word-splitting from happening. From the GNU bash man page or reset the IFS value by setting IFS="" before the substitution is expanded.
$(command)
Bash performs the expansion by executing command in a subshell environment and replacing the command substitution with the standard output of the command, with any trailing newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting.
I am writing a shell program to output another shell program to be evalled later. Is there some common shell program to print shell escaped for a string?
I'm not sure I understand you question. But the %q option of printf might be what you are looking for.
%q Output the corresponding argument in a format that can be reused as shell input
printf %q 'C:\ProgramFiles is a Windows path;'
outputs C:\\ProgramFiles\ is\ a\ Windows\ path\;
(In this example, simple quotes are needed – comment of Gordon Davisson – but this doesn't matter if you print from a variable or the output of a command.)
You could use single quoted string as this is evaluated without any substitution.
For example the following commands are equivalent
cat abc\ hi.txt
cat 'abc hi.txt'
Say I have this command:
printf $text | perl program.pl
How do I guarantee that everything in the $text variable is literally? For example, if $text contains hello"\n, how do I make sure that's exactly what gets passed to program.pl, without the newline or quotation mark (or any conceivable character) being interpreted as a special character?
Quotes!
printf '%s' "$text" | ...
Don't ever expand variables unquoted if you care about preserving their contents precisely. Also, don't ever pass a dynamic string as a format variable when you want it to be treated as literal data.
If you want backslash sequences to be interpreted -- for instance, the two-character sequence \n to be changed to a single newline -- and your shell is bash, use printf '%b' "$text" instead. If you want byte-for-byte accuracy, %s is the Right Thing (and works on any POSIX-compliant shell). If you want escaping for interpretation by another shell (which would be appropriate if, say, you were passing content as part of a ssh command line), then the appropriate format string (for bash only) is %q.
This question already has answers here:
Why does shell ignore quoting characters in arguments passed to it through variables? [duplicate]
(3 answers)
Closed 3 years ago.
Explaining the question through examples...
Demonstrates that the single-quotes after --chapters is gets escaped when the variable is expanded (I didn't expect this):
prompt#ubuntu:/my/scripts$ cat test1.sh
#!/bin/bash
actions="--tags all:"
actions+=" --chapters ''"
mkvpropedit "$1" $actions
prompt#ubuntu:/my/scripts$ ./test1.sh some.mkv
Error: Could not open '''' for reading.
And now for some reason mkvpropedit receives the double quotes as part of the filename (I didn't expect this either):
prompt#ubuntu:/my/scripts$ cat test1x.sh
#!/bin/bash
command="mkvpropedit \"$1\""
command+=" --tags all:"
command+=" --chapters ''"
echo "$command"
$command
prompt#ubuntu:/my/scripts$ ./test1x.sh some.mkv
mkvpropedit "some.mkv" --tags all: --chapters ''
Error: Could not open '''' for reading.
The above echo'd command seems to be correct. Putting the same text in another script gives the expected result:
prompt#ubuntu:/my/scripts$ cat test2.sh
#!/bin/bash
mkvpropedit "$1" --tags all: --chapters ''
prompt#ubuntu:/my/scripts$ ./test2.sh some.mkv
The file is being analyzed.
The changes are written to the file.
Done.
Could anyone please explain why the quotes are not behaving as expected. I found searching on this issue difficult as there are so many other quoting discussions on the web. I wouldn't even know how to explain the question without examples.
I am afraid that some day the file name in the argument contains some character that breaks everything, hence the maybe excessive quoting. I do not understand why the same command executes differently when typed directly in the script or when provided via a variable. Please enlighten me.
Thanks for reading.
The important thing to keep in mind is that quotes are only removed once, when the command line is originally parsed. A quote which is inserted into the command line as a result of parameter substitution ($foo) or command substitution ($(cmd args)) is not treated as a special character. [Note 1]
That seems different from whitespace and glob metacharacters. Word splitting and pathname expansion happen after parameter/command substitution (unless the substitution occurs inside quotes). [Note 2]
The consequence is that it is almost impossible to create a bash variable $args such that
cmd $args
If $args contains quotes, they are not removed. Words inside $args are delimited by sequences of whitespace, not single whitespace characters.
The only way to do it is to set $IFS to include some non-whitespace character; that character can then be used inside $args as a single-character delimiter. However, there is no way to quote a character inside a value, so once you do that, the character you chose cannot be used other than as a delimiter. This is not usually very satisfactory.
There is a solution, though: bash arrays.
If you make $args into an array variable, then you can expand it with the repeated-quote syntax:
cmd "${args[#]}"
which produces exactly one word per element of $args, and suppresses word-splitting and pathname expansion on those words, so they end up as literals.
So, for example:
actions=(--tags all:)
actions+=(--chapters '')
mkvpropedit "$1" "${actions[#]}"
will probably do what you want. So would:
args=("$1")
args+=(--tags)
args+=(all:)
args+=(--chapters)
args+=('')
mkvpropedit "${args[#]}"
and so would
command=(mkvpropedit "$1" --tags all: --chapters '')
"${command[#]}"
I hope that's semi-clear.
man bash (or the online version) contains a blow-by-blow account of how bash assembles commands, starting at the section "EXPANSION". It's worth reading for a full explanation.
Notes:
This doesn't apply to eval or commands like bash -c which evaluate their argument again after command line processing. But that's because command-line processing happens twice.
Word splitting is not the same as "dividing the command into words", which happens when the command is parsed. For one thing, word-splitting uses as separator characters the value of $IFS, whereas command-line parsing uses whitespace. But neither of these are done inside quotes, so they are similar in that respect. In any case, words are split in one way or another both before and after parameter substitution.