The following bash and Perl scripts mysteriously give different results. Why?
#!/bin/bash
hash=`echo -n 'abcd' | /usr/bin/shasum -a 256`;
echo $hash;
#!/usr/bin/perl
$hash = `echo -n 'abcd' | /usr/bin/shasum -a 256`;
print "$hash";
The bash script:
$ ./tst.sh
88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589 -
The Perl script:
$ ./tst.pl
61799467ee1ab1f607764ab36c061f09cfac2f9c554e13f4c7442e66cbab9403 -
the heck?
Summary: In your Perl script, -n is being treated as an argument to include in the output of echo, not a flag to suppress the newline. ( Try
$hash = `echo -n 'abcd'`;
to confirm). Use printf instead.
Perl uses /bin/sh to execute code in back tics. Even if /bin/sh is a link to bash, it will behave differently when invoked via that like. In POSIX mode,
echo -n 'abcd'
will output
-n abcd
that is, the -n option is not recognized as a flag to suppress a newline, but is treated as a regular argument to print. Replace echo -n with printf in each script, and you should get the same SHA hash from each script.
(UPDATE: bash 3.2, when invoked as sh, displays this behavior. Newer versions of bash seem to continue treating -n as a flag when invoked as sh.)
Even better, don't shell out to do things you can do in Perl.
use Digest::SHA;
$hash = Digest::SHA::sha256('abcd');
For the curious, here's what the POSIX spec has to say about echo. I'm not sure what to make of XSI conformance; bash echo requires the -e option to treat the escape characters specially, but nearly every shell—except old versions of bash, and then only under special circumstances—treats -n as a flag, not a string. Oh well.
The following operands shall be supported:
string
A string to be written to standard output. If the first operand is -n, or
if any of the operands contain a <backslash> character, the results are
implementation-defined.
On XSI-conformant systems, if the first operand is -n, it shall be treated
as a string, not an option. The following character sequences shall be
recognized on XSI-conformant systems within any of the arguments:
\a
Write an <alert>.
\b
Write a <backspace>.
\c
Suppress the <newline> that otherwise follows the final argument in the output. All characters following the '\c' in the arguments shall be ignored.
\f
Write a <form-feed>.
\n
Write a <newline>.
\r
Write a <carriage-return>.
\t
Write a <tab>.
\v
Write a <vertical-tab>.
\\
Write a <backslash> character.
\0num
Write an 8-bit value that is the zero, one, two, or three-digit octal number num.
If you do:
printf "%s" 'abcd' | /usr/bin/shasum -a 256
you get the 88d...589 hash. If you do:
printf "%s\n" '-n abcd' | /usr/bin/shasum -a 256
you get the 617...403 hash.
Therefore, I deduce that Perl is somehow running a different echo command, perhaps /bin/echo or /usr/bin/echo instead of the bash built-in echo, or maybe the built-in echo to /bin/sh (which might perhaps be dash rather than bash), and this other echo does not recognize the -n option as an option and outputs different data.
I'm not sure which other echo it is finding; on my machine which is running an Ubuntu 14.04 LTE derivative, bash, dash, sh (linked to bash), ksh, csh and tcsh all treat echo -n abcd the same way. But somewhere along the line, I think that there is something along these lines happening; the hash checksums being identical strongly point to it. (Maybe you have a 3.2 bash linked to sh; see the notes in the comments.)
Related
I'd like to echo something to a file that contains new line escape sequences, however I would like them to remain escaped. I'm looking for basically the opposite to this question.
echo "part1\npart2" >> file
I would like to look like this in the file
$ cat file
old
part1\npart2
but it looks like
$ cat file
old
part1
part2
This is a good example of why POSIX recommends using printf instead of echo (see here, under "application usage"): you don't know what you get with echo1.
You could get:
A shell builtin echo that does not interpret backslash escapes by default
Example: the Bash builtin echo has an -e option to enable backslash escape interpretation and checks the xpg_echo shell option
A shell builtin echo that interprets backslash escapes by default
Examples: zsh, dash
A standalone executable /bin/echo: probably depends on which one – GNU Coreutils echo understands the -e option, like the Bash builtin
The POSIX spec says this (emphasis mine):
The following operands shall be supported:
string
A string to be written to standard output. If the first operand is -n, or if any of the operands contain a <backslash> character, the results are implementation-defined.
So, for a portable solution, we can use printf:
printf '%s\n' 'part1\npart2' >> file
where the \n in the format string will always be interpreted, and the \n in the argument will never be interpreted, resulting in
part1\npart2
being appended to file.
1 For an exhaustive overview of various behaviours for echo and printf, see echo(1) and printf(1) on in-ulm.de.
So Im removing special characters from filenames and replacing with spaces. I have all working apart from files with single backslashes contained therein.
Note these files are created in the Finder on OS X
old_name="testing\this\folder"
new_name=$(echo $old_name | tr '<>:\\#%|?*' ' ');
This results in new_name being "testing hisolder"
How can I just removed the backslashes and not the preceding character?
This results in new_name being "testing hisolder"
This string looks like the result of echo -e "testing\this\folder", because \t and \f are actually replaced with the tabulation and form feed control characters.
Maybe you have an alias like alias echo='echo -e', or maybe the implementation of echo in your version of the shell interprets backslash escapes:
POSIX does not require support for any options, and says that the
behavior of ‘echo’ is implementation-defined if any STRING contains a
backslash or if the first argument is ‘-n’. Portable programs can use
the ‘printf’ command if they need to omit trailing newlines or output
control characters or backslashes.
(from the info page)
So you should use printf instead of echo in new software. In particular, echo $old_name should be replaced with printf %s "$old_name".
There is a good explanation in this discussion, for instance.
No need for printf
As #mklement0 suggested, you can avoid the pipe by means of the Bash here string:
tr '<>:\\#%|?*' ' ' <<<"$old_name"
Ruslan's excellent answer explains why your command may not be working for you and offers a robust, portable solution.
tl;dr:
You probably ran your code with sh rather than bash (even though on macOS sh is Bash in disguise), or you had shell option xpg_echo explicitly turned on.
Use printf instead of echo for portability.
In Bash, with the default options and using the echo builtin, your command should work as-is (except that you should double-quote $old_name for robustness), because echo by default does not expand escape sequences such as \t in its operands.
However, Bash's echo can be made to expand control-character escape sequences:
explicitly, by executing shopt -s xpg_echo
implicitly, if you run Bash as sh or with the --posix option (which, among other options and behavior changes, activates xpg_echo)
Thus, your symptom may have been caused by running your code from a script with shebang line #!/bin/sh, for instance.
However, if you're targeting sh, i.e., if you're writing a portable script, then echo should be avoided altogether for the very reason that its behavior differs across shells and platforms - see Ruslan's printf solution.
As an aside: perhaps a more robust approach to your tr command is a whitelisting approach: stating only the characters that are explicitly allowed in your result, and excluding other with the -C option:
old_name='testing\this\folder'
new_name=$(printf '%s' "$old_name" | tr -C '[:alnum:]_-' ' ')
That way, any characters that aren't either letters, numbers, _, or - are replaced with a space.
With Bash, you can use parameter expansion:
$ old_name="testing\this\folder"
$ new_name=${old_name//[<>:\\#%|?*]/ }
$ echo $new_name
testing this folder
For more, please refer to the Bash manual on shell parameter expansion.
I think your test case is missing proper escaping for \, so you're not really testing the case of a backslash contained in a string.
This worked for me:
old_name='testing\\this\\folder'
new_name=$(echo $old_name | tr '<>:\\#%|?*' ' ');
echo $new_name
# testing this folder
I would like to use echo in bash to print out a string of characters followed by only a carriage return. I've looked through the man page and have found that echo -e will make echo interpret backslash escape characters. Using that I can say echo -e 'hello\r' and it will print like this
$>echo -e 'hello\r'
hello
$>
So it looks like it handled the carriage return properly. I also found echo -n in the man page will stop echo from inserting a newline character and it looks like it works when I do this
$>echo -n 'hello\r'
hello\r$>
The problem I'm having is in combining both -e and -n. I've tried each of echo -e -n 'hello\r', echo -n -e 'hello\r', echo -en 'hello\r', and echo -ne 'hello\r' and nothing gets printed like so:
$>echo -ne 'hello\r'
$>
Is there something I'm missing here or can the -e and -n options not be used together?
I think it's working, you're just not seeing it. This is your command:
$> echo -ne 'hello\r'
Because of the carriage return (\r), that will leave the cursor at the start of the same line on the terminal where it wrote the hello - which means that's where the next thing output to the terminal will be written. So if your actual prompt is longer than the $> you show here, it will overwrite the hello completely.
This sequence will let you see what's actually happening:
echo -ne 'hello\r'; sleep 5; echo 'good-bye'
But for better portability to other shells, I would avoid using options on echo like that. Those are purely bashisms, not supported by the POSIX standard. The printf builtin, however, is specified by POSIX. So if you want to display strings with no newline and parsing of backslash sequences, you can just use it:
printf '%s\r' 'hello'
There are numerous different implementations of the echo command. There's one built into most shells (with different behavior for each), and the behavior of /bin/echo varies considerably from one system to another.
Rather than echo, use printf. It's built into bash as well as being available as an external command, and its behavior is much more consistent across implementations. (The major variation is that the GNU coreutils printf recognizes --help and --version options.)
Just use:
printf 'hello\r'
I'd like you to introduce you to printf.
OP, meet printf. printf. This is the OP...
Whenever you are trying to do anything unusual with output in BASH, you should switch to printf. In fact, I use printf all the time, so my scripts will run under both BASH and Kornshell.
Although BASH and Kornshell are 99% the same, the echo command is different. In Kornshell, it's deprecated, and you're supposed to use the builtin print. However, there's no print in BASH. Using printf solves the problem because it works (sort of) the same in both shells.
Since printf doesn't automatically end each line with a \n, you don't have to worry about how to prevent the \n from being appended. Heck, if you want a \n, you have to put it yourself. In your case:
printf `hello\r`
However, for safety reasons, you should really do this:
printf '%s\r' 'hello'
This is a habit I've gotten into when I'm using printf. Imagine if I want to print out $FOO, and I do this:
printf "$FOO\n"
That will normally work, but what if $FOO has a dash at the beginning, or has a % sign in it. The above won't do what I want to do. However, if I do this:
printf '%s\n' "$FOO"
This will print out $FOO no matter what $FOO has in it.
POSIX 7 says it is not possible:
If the first operand is -n, or if any of the operands contain a backslash character, the results are implementation-defined.
On Ubuntu 12.04 for example:
Bash's built-in echo echo -e '\n' interprets the newline, echo '\n' does not.
/bin/sh's built-in echo does not have the -e option, and interprets the newline even without it. This can byte when writing Makefiles, which use /bin/sh by default.
POSIX 7 solution: printf as the others mentioned.
Quoting. It's the wave of the future. Always quote.
echo -ne 'hello\r'
Use double quotes if you put a variable inside that you want expanded.
I've read the man pages on echo, and it tells me that the -e parameter will allow an escaped character, such as an escaped n for newline, to have its special meaning. When I type the command
$ echo -e 'foo\nbar'
into an interactive bash shell, I get the expected output:
foo
bar
But when I use this same command (i've tried this command character for character as a test case) I get the following output:
-e foo
bar
It's as if echo is interpretting the -e as a parameter (because the newline still shows up) yet also it interprets the -e as a string to echo. What's going on here? How can I prevent the -e showing up?
You need to use #!/bin/bash as the first line in your script. If you don't, or if you use #!/bin/sh, the script will be run by the Bourne shell and its echo doesn't recognize the -e option. In general, it is recommended that all new scripts use printf instead of echo if portability is important.
In Ubuntu, sh is provided by a symlink to /bin/dash.
Different implementations of echo behave in annoyingly different ways. Some don't take options (i.e. will simply echo -e as you describe) and automatically interpret escape sequences in their parameters. Some take flags, and don't interpret escapes unless given the -e flag. Some take flags, and interpret different escape sequences depending on whether the -e flag was passed. Some will cause you to tear your hair out if you try to get them to behave in a predictable manner... oh, wait, that's all of them.
What you're probably seeing here is a difference between the version of echo built into bash vs /bin/echo or maybe vs. some other shell's builtin. This bit me when Mac OS X v10.5 shipped with a bash builtin echo that echoed flags, unlike what all my scripts expected...
In any case, there's a solution: use printf instead. It always interprets escape sequences in its first argument (the format string). The problems are that it doesn't automatically add a newline (so you have to remember do that explicitly), and it also interprets % sequences in its first argument (it is, after all, a format string). Generally, you want to put all the formatting stuff in the format string, then put variable strings in the rest of the arguments so you can control how they're interpreted by which % format you use to interpolate them into the output. Some examples:
printf "foo\nbar\n" # this does what you're trying to do in the example
printf "%s\n" "$var" # behaves like 'echo "$var"', except escapes will never be interpreted
printf "%b\n" "$var" # behaves like 'echo "$var"', except escapes will always be interpreted
printf "%b\n" "foo\nbar" # also does your example
Use
alias echo /usr/bin/echo
to force 'echo' invoking coreutils' echo which interpret '-e' parameter.
Try this:
import subprocess
def bash_command(cmd):
subprocess.Popen(['/bin/bash', '-c', cmd])
code="abcde"
// you can use echo options such as -e
bash_command('echo -e "'+code+'"')
Source: http://www.saltycrane.com/blog/2011/04/how-use-bash-shell-python-subprocess-instead-binsh/
How do I echo one or more tab characters using a bash script?
When I run this code
res=' 'x # res = "\t\tx"
echo '['$res']' # expect [\t\tx]
I get this
res=[ x] # that is [<space>x]
echo -e ' \t '
will echo 'space tab space newline' (-e means 'enable interpretation of backslash escapes'):
$ echo -e ' \t ' | hexdump -C
00000000 20 09 20 0a | . .|
Use printf, not echo.
There are multiple different versions of the echo command. There's /bin/echo (which may or may not be the GNU Coreutils version, depending on the system), and the echo command is built into most shells. Different versions have different ways (or no way) to specify or disable escapes for control characters.
printf, on the other hand, has much less variation. It can exist as a command, typically /bin/printf, and it's built into some shells (bash and zsh have it, tcsh and ksh don't), but the various versions are much more similar to each other than the different versions of echo are. And you don't have to remember command-line options (with a few exceptions; GNU Coreutils printf accepts --version and --help, and the built-in bash printf accepts -v var to store the output in a variable).
For your example:
res=' 'x # res = "\t\tx"
printf '%s\n' "[$res]"
And now it's time for me to admit that echo will work just as well for the example you're asking about; you just need to put double quotes around the argument:
echo "[$res]"
as kmkaplan wrote (two and a half years ago, I just noticed!). The problem with your original commands:
res=' 'x # res = "\t\tx"
echo '['$res']' # expect [\t\tx]
isn't with echo; it's that the shell replaced the tab with a space before echo ever saw it.
echo is fine for simple output, like echo hello world, but you should use printf whenever you want to do something more complex. You can get echo to work, but the resulting code is likely to fail when you run it with a different echo implementation or a different shell.
You can also try:
echo Hello$'\t'world.
Put your string between double quotes:
echo "[$res]"
you need to use -e flag for echo then you can
echo -e "\t\t x"
From the bash man page:
Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.
So you can do this:
echo $'hello\tworld'
Use the verbatim keystroke, ^V (CTRL+V, C-v, whatever).
When you type ^V into the terminal (or in most Unix editors), the following character is taken verbatim. You can use this to type a literal tab character inside a string you are echoing.
Something like the following works:
echo "^V<tab>" # CTRL+V, TAB
Bash docs (q.v., "quoted-insert")
quoted-insert (C-q, C-v)
Add the next character that you type to the line verbatim. This is how to insert key sequences like C-q, for example.
side note: according to this, ALT+TAB should do the same thing, but we've all bound that sequence to window switching so we can't use it
tab-insert (M-TAB)
Insert a tab character.
--
Note: you can use this strategy with all sorts of unusual characters. Like a carriage return:
echo "^V^M" # CTRL+V, CTRL+M
This is because carriage return is ASCII 13, and M is the 13th letter of the alphabet, so when you type ^M, you get the 13th ASCII character. You can see it in action using ls^M, at an empty prompt, which will insert a carriage return, causing the prompt to act just like you hit return. When these characters are normally interpreted, verbatim gets you get the literal character.
Using echo to print values of variables is a common Bash pitfall.
Reference link:
http://mywiki.wooledge.org/BashPitfalls#echo_.24foo
If you want to use echo "a\tb" in a script, you run the script as:
# sh -e myscript.sh
Alternatively, you can give to myscript.sh the execution permission, and then run the script.
# chmod +x myscript.sh
# ./myscript.sh
res="\t\tx"
echo -e "[${res}]"