I have this command:
echo -n "cdt_main!##$" | base64
that fails with:
bash: !##$": event not found
lulz b/c of the special characters.
So my best guess is that this is correct:
echo -n "cdt_main\!\##\$" | base64
at least this is b/c there is no error now. Unfortunately I cannot test the base64 version password until I know that it's right - I have only one chance to get it right o/w something will blow up. Does that look right to you?
Using the decoding trick, I have:
echo -n "cdt_main\!\##\$" | base64 | base64 --decode
which yields:
cdt_main\!\##$
given that output, not sure if this is working as planned, because slashes are in there.
The problem is that bash's history expansion is turned on. Unless you history expansion is something that you actively want and use, the solution is to turn it off.
Observe that this fails with event not found:
$ echo -n "cdt_main!##$" | base64
bash: !##: event not found
But, this succeeds:
$ set +H
$ echo -n "cdt_main!##$" | base64
Y2R0X21haW4hQCMk
set +H turns off history expansion.
Alternative: use single quotes
For strings inside double quotes, the shell will perform a wide variety of expansions. To prevent that, put strings instead in single quotes.
For example, the following succeeds even with history expansion turned on:
$ set -H
$ echo -n 'cdt_main!##$' | base64
Y2R0X21haW4hQCMk
Portable approach
The -n option to echo is not supported by all shells. For a more portable solution, one should, as Gordon Davisson suggests, use printf:
printf "%s" 'cdt_main!##$' | base64
#John1024 is right, but an even more generic solution is to use:
echo -n 'cdt_main!##$' | base64
single quotes ' means the characters in the string aren't interpreted by bash, very nice.
Related
I'm writing a shell script that should be somewhat secure, i.e., does not pass secure data through parameters of commands and preferably does not use temporary files. How can I pass a variable to the standard input of a command?
Or, if it's not possible, how can I correctly use temporary files for such a task?
Passing a value to standard input in Bash is as simple as:
your-command <<< "$your_variable"
Always make sure you put quotes around variable expressions!
Be cautious, that this will probably work only in bash and will not work in sh.
Simple, but error-prone: using echo
Something as simple as this will do the trick:
echo "$blah" | my_cmd
Do note that this may not work correctly if $blah contains -n, -e, -E etc; or if it contains backslashes (bash's copy of echo preserves literal backslashes in absence of -e by default, but will treat them as escape sequences and replace them with corresponding characters even without -e if optional XSI extensions are enabled).
More sophisticated approach: using printf
printf '%s\n' "$blah" | my_cmd
This does not have the disadvantages listed above: all possible C strings (strings not containing NULs) are printed unchanged.
(cat <<END
$passwd
END
) | command
The cat is not really needed, but it helps to structure the code better and allows you to use more commands in parentheses as input to your command.
Note that the 'echo "$var" | command operations mean that standard input is limited to the line(s) echoed. If you also want the terminal to be connected, then you'll need to be fancier:
{ echo "$var"; cat - ; } | command
( echo "$var"; cat - ) | command
This means that the first line(s) will be the contents of $var but the rest will come from cat reading its standard input. If the command does not do anything too fancy (try to turn on command line editing, or run like vim does) then it will be fine. Otherwise, you need to get really fancy - I think expect or one of its derivatives is likely to be appropriate.
The command line notations are practically identical - but the second semi-colon is necessary with the braces whereas it is not with parentheses.
This robust and portable way has already appeared in comments. It should be a standalone answer.
printf '%s' "$var" | my_cmd
or
printf '%s\n' "$var" | my_cmd
Notes:
It's better than echo, reasons are here: Why is printf better than echo?
printf "$var" is wrong. The first argument is format where various sequences like %s or \n are interpreted. To pass the variable right, it must not be interpreted as format.
Usually variables don't contain trailing newlines. The former command (with %s) passes the variable as it is. However tools that work with text may ignore or complain about an incomplete line (see Why should text files end with a newline?). So you may want the latter command (with %s\n) which appends a newline character to the content of the variable. Non-obvious facts:
Here string in Bash (<<<"$var" my_cmd) does append a newline.
Any method that appends a newline results in non-empty stdin of my_cmd, even if the variable is empty or undefined.
I liked Martin's answer, but it has some problems depending on what is in the variable. This
your-command <<< """$your_variable"""
is better if you variable contains " or !.
As per Martin's answer, there is a Bash feature called Here Strings (which itself is a variant of the more widely supported Here Documents feature):
3.6.7 Here Strings
A variant of here documents, the format is:
<<< word
The word is expanded and supplied to the command on its standard
input.
Note that Here Strings would appear to be Bash-only, so, for improved portability, you'd probably be better off with the original Here Documents feature, as per PoltoS's answer:
( cat <<EOF
$variable
EOF
) | cmd
Or, a simpler variant of the above:
(cmd <<EOF
$variable
EOF
)
You can omit ( and ), unless you want to have this redirected further into other commands.
Try this:
echo "$variable" | command
If you came here from a duplicate, you are probably a beginner who tried to do something like
"$variable" >file
or
"$variable" | wc -l
where you obviously meant something like
echo "$variable" >file
echo "$variable" | wc -l
(Real beginners also forget the quotes; usually use quotes unless you have a specific reason to omit them, at least until you understand quoting.)
So Im removing special characters from filenames and replacing with spaces. I have all working apart from files with single backslashes contained therein.
Note these files are created in the Finder on OS X
old_name="testing\this\folder"
new_name=$(echo $old_name | tr '<>:\\#%|?*' ' ');
This results in new_name being "testing hisolder"
How can I just removed the backslashes and not the preceding character?
This results in new_name being "testing hisolder"
This string looks like the result of echo -e "testing\this\folder", because \t and \f are actually replaced with the tabulation and form feed control characters.
Maybe you have an alias like alias echo='echo -e', or maybe the implementation of echo in your version of the shell interprets backslash escapes:
POSIX does not require support for any options, and says that the
behavior of ‘echo’ is implementation-defined if any STRING contains a
backslash or if the first argument is ‘-n’. Portable programs can use
the ‘printf’ command if they need to omit trailing newlines or output
control characters or backslashes.
(from the info page)
So you should use printf instead of echo in new software. In particular, echo $old_name should be replaced with printf %s "$old_name".
There is a good explanation in this discussion, for instance.
No need for printf
As #mklement0 suggested, you can avoid the pipe by means of the Bash here string:
tr '<>:\\#%|?*' ' ' <<<"$old_name"
Ruslan's excellent answer explains why your command may not be working for you and offers a robust, portable solution.
tl;dr:
You probably ran your code with sh rather than bash (even though on macOS sh is Bash in disguise), or you had shell option xpg_echo explicitly turned on.
Use printf instead of echo for portability.
In Bash, with the default options and using the echo builtin, your command should work as-is (except that you should double-quote $old_name for robustness), because echo by default does not expand escape sequences such as \t in its operands.
However, Bash's echo can be made to expand control-character escape sequences:
explicitly, by executing shopt -s xpg_echo
implicitly, if you run Bash as sh or with the --posix option (which, among other options and behavior changes, activates xpg_echo)
Thus, your symptom may have been caused by running your code from a script with shebang line #!/bin/sh, for instance.
However, if you're targeting sh, i.e., if you're writing a portable script, then echo should be avoided altogether for the very reason that its behavior differs across shells and platforms - see Ruslan's printf solution.
As an aside: perhaps a more robust approach to your tr command is a whitelisting approach: stating only the characters that are explicitly allowed in your result, and excluding other with the -C option:
old_name='testing\this\folder'
new_name=$(printf '%s' "$old_name" | tr -C '[:alnum:]_-' ' ')
That way, any characters that aren't either letters, numbers, _, or - are replaced with a space.
With Bash, you can use parameter expansion:
$ old_name="testing\this\folder"
$ new_name=${old_name//[<>:\\#%|?*]/ }
$ echo $new_name
testing this folder
For more, please refer to the Bash manual on shell parameter expansion.
I think your test case is missing proper escaping for \, so you're not really testing the case of a backslash contained in a string.
This worked for me:
old_name='testing\\this\\folder'
new_name=$(echo $old_name | tr '<>:\\#%|?*' ' ');
echo $new_name
# testing this folder
EDIT: the command substitution is not necessary for the surprising behavior, although it is the most common use case. The same question applies to just echo "'!b'"
b=a
# Enable history substitution.
# This option is on by default on interactive shells.
set -H
echo '!b'
# Output: '!b'
# OK. Escaped by single quotes.
echo $(echo '!b')
# Output: '!b'
# OK. Escaped by single quotes.
echo "$(echo '$b')"
# Output: '$b'
# OK. Escaped by single quotes.
echo "$(echo '!b')"
# Output: history expands
# BAD!! WHY??
In the last example, what is the best way to escape the !?
Why was it not escaped even if I used single quotes, while echo "$(echo '$b')" was? What is the difference between ! and $?
Why was does echo $(echo '!b') (no quotes) work? (pointed by #MBlanc).
I would prefer to do this without:
set +H as I would need set -H afterwards to maintain shell state
backslash escapes, because I need one for every ! and it has to be outside the quotes:
echo "$(echo '\!a')"
# '\!a'.
# No good.
echo "$(echo 'a '\!' b '\!' c')"
# a ! b ! c
# Good, but verbose.
echo $(echo '!b') (no quotes), because the command could return spaces.
Version:
bash --version | head -n1
# GNU bash, version 4.2.25(1)-release (i686-pc-linux-gnu)
In your last example,
echo "$(echo '!b')"
the exclamation point is not single-quoted. Because history expansion occurs so early in the parsing process, the single quotes are just part of the double-quoted string; the parser hasn't recognized the command substitution yet to establish a new context where the single quotes would be quoting operators.
To fix, you'll have to temporarily turn off history expansion:
set +H
echo "$(echo '!b')"
set -H
This was repeatedly reported as a bug, most recently against bash 4.3 in 2014, for behavior going back to bash 3.
There was some discussion whether this constituted a bug or expected but perhaps undesirable behavior; it seems the consensus has been that, however you want to characterize the behavior, it shouldn't be allowed to continue.
It's fixed in bash 4.4, echo "$(echo '!b')" doesn't expand, echo "'!b'" does, which I regard as proper behavior because the single quotes are shell syntax markers in the first example and not in the second.
If History Expansion is enabled, you can only echo the ! character if it is put in single quotes, escaped or if followed by a whitespace character, carriage return, or =.
From man bash:
Only backslash (\) and single quotes can quote the history
expansion character.
Several characters inhibit history expansion if found immediately fol-
lowing the history expansion character, even if it is unquoted: space,
tab, newline, carriage return, and =.
I believe the key word here is “Only”. The examples provided in the question only consider the outer most quotes being double quotes.
Sometimes you need to make a small addition to a big command pipe
The OP's "Good, but verbose" example is actually pretty awesome for many cases.
Please forgive the contrived example. The whole reason I need such a solution is that I have a lot of distracting, nested code. But, it boils down to: I must do a !d in sed within a double quoted bash command expansion.
This works
$ ifconfig | sed '/inet/!d'
inet 127.0.0.1 netmask 0xff000000
…
This does not
$ echo "$(ifconfig | sed '/inet/!d')"
-bash: !d': event not found
This is a simplest compromise
$ echo "$(ifconfig | sed '/inet/'\!'d')"
inet 127.0.0.1 netmask 0xff000000
…
Using the compromise allows me to insert a few characters into the existing code and produce a Pull Request that anyone can understand… even though resulting code is more difficult to understand. If I did a complete refactor, the code reviewers would have a much more challenging time verifying it. And of course this bash has no unit tests.
I'm trying to escape ('\') a semicolon (';') in a string on unix shell (bash) with sed. It works when I do it directly without assigning the value to a variable. That is,
$ echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'
hello\;
$
However, it doesn't appear to work when the above command is assigned to a variable:
$ result=`echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'`
$
$ echo $result
hello;
$
Any idea why?
I tried by using the value enclosed with and without quotes but that didn't help. Any clue greatly appreciated.
btw, I first thought the semicolon at the end of the string was somehow acting as a terminator and hence the shell didn't continue executing the sed (if that made any sense). However, that doesn't appear to be an issue. I tried by using the semicolon not at the end of the string (somewhere in between). I still see the same result as before. That is,
$ echo "hel;lo" | sed 's/\([^\\]\);/\1\\;/g'
hel\;lo
$
$ result=`echo "hel;lo" | sed 's/\([^\\]\);/\1\\;/g'`
$
$ echo $result
hel;lo
$
You don't need sed (or any other regex engine) for this at all:
s='hello;'
echo "${s//;/\;}"
This is a parameter expansion which replaces ; with \;.
That said -- why are you trying to do this? In most cases, you don't want escape characters (which are syntax) to be inside of scalar variables (which are data); they only matter if you're parsing your data as syntax (such as using eval), which is a bad idea for other reasons, and best avoided (or done programatically, as via printf %q).
I find it interesting that the use of back-ticks gives one result (your result) and the use of $(...) gives another result (the wanted result):
$ echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'
hello\;
$ z1=$(echo "hello;" | sed 's/\([^\\]\);/\1\\;/g')
$ z2=`echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'`
$ printf "%s\n" "$z1" "$z2"
hello\;
hello;
$
If ever you needed an argument for using the modern x=$(...) notation in preference to the older x=`...` notation, this is probably it. The shell does an extra round of backslash interpretation with the back-ticks. I can demonstrate this with a little program I use when debugging shell scripts called al (for 'argument list'); you can simulate it with printf "%s\n":
$ z2=`echo "hello;" | al sed 's/\([^\\]\);/\1\\;/g'`
$ echo "$z2"
sed
s/\([^\]\);/\1\;/g
$ z1=$(echo "hello;" | al sed 's/\([^\\]\);/\1\\;/g')
$ echo "$z1"
sed
s/\([^\\]\);/\1\\;/g
$ z1=$(echo "hello;" | printf "%s\n" sed 's/\([^\\]\);/\1\\;/g')
$ echo "$z1"
sed
s/\([^\\]\);/\1\\;/g
$
As you can see, the script executed by sed differs depending on whether you use x=$(...) notation or x=`...` notation.
s/\([^\]\);/\1\;/g # ``
s/\([^\\]\);/\1\\;/g # $()
Summary
Use $(...); it is easier to understand.
You need to use four (three also work). I guess its because it's interpreted twice, first one by the sed command and the second one by the shell when reading the content of the variable:
result=`echo "hello;" | sed 's/\([^\\]\);/\1\\\\;/g'`
And
echo "$result"
yields:
hello\;
I get:
$ echo -e "D"{a,b,c}".jpg\n"
Da.jpg
Db.jpg
Dc.jpg
Note: The extra spaces before Db and Dc on the 2nd and 3rd line of the output.
Why are these there?
Thanks,
Dan
Edit: Since my actual objective had spaces in it (which I should have written originally):
echo -e "Name"{,.}" "{-,}"extra"{,so}" 5v5 "{one,two,No\ four}{,!,\!\!}"\n"
Most solutions here didn't work for me (for loop, xarg, tr). Printf didn't work because of multiple braces expansions that I want to cantesian product.
I combined 3 solutions (mletterle's \b, Dennis Williamson's extra space, and Jim Dennis's using far less quotes) to get:
echo -e "" \\bName{,.}\ {-,}extra{,so}\ 5v5\ {one,two,No\ four}{,\!,\!\!}\\n
Thanks all who answered! I learned a lot from your responses!
Dan
use the more portable printf
$ printf "D%s.jpg\n" {a,b,c}
Da.jpg
Db.jpg
Dc.jpg
Because that's what brace expansion does. From man bash, under the heading Brace expansion:
Patterns to be brace expanded take the
form of an
optional preamble, followed by ... a series of comma-separated
strings ... followed by an optional
postscript. The preamble is prefixed
to each string contained within the
braces, and the postscript is then appended to each resulting
string, expanding left to right
For example, a{d,c,b}e expands into
‘ade ace abe’
So in your example, "D" is the preamble and ".jpg\n" is the postscript.
So, after brace expansion occurs, you're left with:
echo -e Da.jpg\n Db.jpg\n Dc.jpg\n
As hewgill points out, the shell then splits this into three tokens and passes them to echo; which outputs each token separated by a space. To get the output you want, you need to use one of the many suggestions here that don't re-inserted the unwanted space between tokens.
It's longer and probably not the neatest way to do this, but the following gives the output you're after:
for file in "D"{a,b,c}".jpg"
do
echo ${file}
done
echo always adds spaces between arguments. Try your command without \n and compare the results.
The easiest and cleanest solution is to add a backspace to the front of each line:
echo -e -n "\bD"{a,b,c}".jpg\n"
This produces the desired output.
You can get the desired effect by using xargs to separate the arguments spit by the first echo into a line each:
$ echo "D"{a,b,c}".jpg" | xargs -n1 echo
Da.jpg
Db.jpg
Dc.jpg
You can get a more consistent look by prepending a null:
$ echo -en "" "D"{a..c}".jpg\n"
Da.jpg
Db.jpg
Dc.jpg
Now they all have an extra space. Also, using -n eliminates the extra newline at the end. Also, you can use a range in your brace expansion.
Here is a solution using sed (that builds upon https://stackoverflow.com/a/2003856/8180143):
$ echo -en "" "D"{a..c}".jpg\n" | sed 's/ //'
Da.jpg
Db.jpg
Dc.jpg
This solution has the advantage of working with inputs having spaces, e.g.,
$ echo -en "" "D "{a..c}".jpg\n" | sed 's/ //'
D a.jpg
D b.jpg
D c.jpg