I would like to use the bash here doc syntax to build a long string. I would like the heredoc to ignore newlines/spaces/tabs even when I use newlines for code clarity.
I thought this would work:
#!/bin/bash
#http://unix.stackexchange.com/questions/20035/how-to-add-newlines-into-variables-in-bash-script
IFS= read -r -d '' NS_LOG<<-EOF
*=error|warn|prefix_node|prefix_func
:PointToPointNetDevice
:ClockTest
:ClockPerfect
:TcpTestSuite
:TcpRxBuffer
:TcpTxBuffer
:TcpHeader=*
:TcpL4Protocol
:TraceHelper:PointToPointHelper
EOF
echo $NS_LOG
export NS_LOG
but somewhere bash appends spaces between lines and instead of having the desired
*=error|warn|prefix_node|prefix_func:PointToPointNetDevice:ClockTest:ClockPerfect:Clock
I have when running $ ./launch_myscript.sh:
*=error|warn|prefix_node|prefix_func :PointToPointNetDevice :ClockTest :ClockPerfect :Clock etc...
My bash --version:
GNU bash, version 4.3.30(1)-release (x86_64-pc-linux-gnu)
I just saw in the recommended posts this one Bash: Why is echo adding extra space?. How can I prevent NS_LOG from being considered as several arguments? Ultimately the goal is to export that variable.
Your read command is very explicitly treating newlines as data: By clearing IFS and passing -d '', you tell read not to treat whitespace characters as special; since they're not special, they go into the output variable like everything else. However, you can take them out later:
IFS= read -r -d '' NS_LOG <<'EOF'
...content...
EOF
NS_LOG=${NS_LOG//[[:space:]]/} ## replace all whitespace with the empty string
printf '%s\n' "$NS_LOG" ## the quotes are important!
See this snippet run, and its output, at http://ideone.com/fWhzBB.
Notes:
<<'EOF' prevents expansions from occurring within the heredoc itself; with <<EOF, $foo, $(foo), etc. would be special.
<<- only trims leading tab characters, not any other form of whitespace; it's typically safer to do without.
echo $foo string-splits and glob-expands the contents of $foo, passing each word created by this process as a separate argument; echo then places spaces between each argument. echo "$foo" ensures that the entire expansion is treated as a single word. See BashPitfalls #14.
Using echo with nontrivial or unknown data is advised against in the relevant portion of the POSIX specification; printf is the preferred substitute. POSIX echo is explicitly allowed to behave in undefined ways when content contains backslash literals, and the BSD- and AT&T-derived forms of the command are explicitly incompatible, both with each other and with the common GNU implementation (providing an -e flag, which the POSIX spec requires to simply print -e on its output).
things aren't that difficult. here is another solution:
var="`tr -d '[:space:] <<EOF'
your
text
with
lot of spaces
EOF`"
result:
$ echo "$var"
yourtextwithlotofspaces
Related
There are a lot of examples here how to source a dotenv file in bash but has anyone one a method that achieves the same with dash (which is the default shell for minimal Debian installations)?
The solution should look like this:
$ some foo my-command-using-env-vars
e.g.
$ env $(cat .env) my-command-using-env-vars
And it is important that the solution supports multiline values with spaces like:
SSH_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\nfoo\nbar\baz"
and special characters like hash within quotes:
SPECIAL="foo#bar"
It seems that your problem is not so much that you're using dash but that you want to support \n escapes. The following works in dash and supports \n escapes:
eval "$(echo $(cat .env))" my-command-using-env-vars
That's because unlike in bash the built-in echo in dash supports backslash escapes even without the -e option. The following works in both bash and dash, provided that the non-built-in, system echo supports the -e option:
eval "$(env echo -e $(cat .env))" my-command-using-env-vars
Note that both approaches will also handle other backslash escapes (either POSIX or GNU), possibly in a different way than you expect.
Some technical notes:
$(cat .env)
performs Field Splitting, converting any newline in file .env into spaces.
"$(env echo -e ...)"
expands backslash escapes regardless of the current shell by invoking echo -e via env. The double quotes disable field splitting, so that newlines are preserved.
I'd like to echo something to a file that contains new line escape sequences, however I would like them to remain escaped. I'm looking for basically the opposite to this question.
echo "part1\npart2" >> file
I would like to look like this in the file
$ cat file
old
part1\npart2
but it looks like
$ cat file
old
part1
part2
This is a good example of why POSIX recommends using printf instead of echo (see here, under "application usage"): you don't know what you get with echo1.
You could get:
A shell builtin echo that does not interpret backslash escapes by default
Example: the Bash builtin echo has an -e option to enable backslash escape interpretation and checks the xpg_echo shell option
A shell builtin echo that interprets backslash escapes by default
Examples: zsh, dash
A standalone executable /bin/echo: probably depends on which one – GNU Coreutils echo understands the -e option, like the Bash builtin
The POSIX spec says this (emphasis mine):
The following operands shall be supported:
string
A string to be written to standard output. If the first operand is -n, or if any of the operands contain a <backslash> character, the results are implementation-defined.
So, for a portable solution, we can use printf:
printf '%s\n' 'part1\npart2' >> file
where the \n in the format string will always be interpreted, and the \n in the argument will never be interpreted, resulting in
part1\npart2
being appended to file.
1 For an exhaustive overview of various behaviours for echo and printf, see echo(1) and printf(1) on in-ulm.de.
So Im removing special characters from filenames and replacing with spaces. I have all working apart from files with single backslashes contained therein.
Note these files are created in the Finder on OS X
old_name="testing\this\folder"
new_name=$(echo $old_name | tr '<>:\\#%|?*' ' ');
This results in new_name being "testing hisolder"
How can I just removed the backslashes and not the preceding character?
This results in new_name being "testing hisolder"
This string looks like the result of echo -e "testing\this\folder", because \t and \f are actually replaced with the tabulation and form feed control characters.
Maybe you have an alias like alias echo='echo -e', or maybe the implementation of echo in your version of the shell interprets backslash escapes:
POSIX does not require support for any options, and says that the
behavior of ‘echo’ is implementation-defined if any STRING contains a
backslash or if the first argument is ‘-n’. Portable programs can use
the ‘printf’ command if they need to omit trailing newlines or output
control characters or backslashes.
(from the info page)
So you should use printf instead of echo in new software. In particular, echo $old_name should be replaced with printf %s "$old_name".
There is a good explanation in this discussion, for instance.
No need for printf
As #mklement0 suggested, you can avoid the pipe by means of the Bash here string:
tr '<>:\\#%|?*' ' ' <<<"$old_name"
Ruslan's excellent answer explains why your command may not be working for you and offers a robust, portable solution.
tl;dr:
You probably ran your code with sh rather than bash (even though on macOS sh is Bash in disguise), or you had shell option xpg_echo explicitly turned on.
Use printf instead of echo for portability.
In Bash, with the default options and using the echo builtin, your command should work as-is (except that you should double-quote $old_name for robustness), because echo by default does not expand escape sequences such as \t in its operands.
However, Bash's echo can be made to expand control-character escape sequences:
explicitly, by executing shopt -s xpg_echo
implicitly, if you run Bash as sh or with the --posix option (which, among other options and behavior changes, activates xpg_echo)
Thus, your symptom may have been caused by running your code from a script with shebang line #!/bin/sh, for instance.
However, if you're targeting sh, i.e., if you're writing a portable script, then echo should be avoided altogether for the very reason that its behavior differs across shells and platforms - see Ruslan's printf solution.
As an aside: perhaps a more robust approach to your tr command is a whitelisting approach: stating only the characters that are explicitly allowed in your result, and excluding other with the -C option:
old_name='testing\this\folder'
new_name=$(printf '%s' "$old_name" | tr -C '[:alnum:]_-' ' ')
That way, any characters that aren't either letters, numbers, _, or - are replaced with a space.
With Bash, you can use parameter expansion:
$ old_name="testing\this\folder"
$ new_name=${old_name//[<>:\\#%|?*]/ }
$ echo $new_name
testing this folder
For more, please refer to the Bash manual on shell parameter expansion.
I think your test case is missing proper escaping for \, so you're not really testing the case of a backslash contained in a string.
This worked for me:
old_name='testing\\this\\folder'
new_name=$(echo $old_name | tr '<>:\\#%|?*' ' ');
echo $new_name
# testing this folder
No matter what I do, I can't get the newlines in quotes to persist in the output. What am I to do? This is bash in OS X. My code is here: Gist
read -d '' sbt_text=$(cat <<"EOF"
#!/bin/bash
SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M";
java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$#"
EOF
)
echo $sbt_text
#!/bin/bash > SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"; > java -jar /sbt-launch.jar ""
What you probably want is this:
IFS= read -d '' -r sbt_text <<"EOF"
#!/bin/bash
SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M";
java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$#"
EOF
echo "$sbt_text"
While your read command doesn't work as posted, not double-quoting $sbt_text will invariably perform word splitting (and also pathname expansion (globbing)), which, when passed to echo, means that any interior whitespace is normalized to a single space.
In order to preserve the value of a variable as-is in POSIX-like shells - without subjecting it to the shell expansions described above - you must double-quote it.
With respect to the read command, note that:
-r prevents usually unintended interpretation of \ instances in the input.
Prepending IFS= ensures that each line in the here-document is preserved as-is with respect to whitespace (no trimming of leading and trailing whitespace)
There's no need for a command substitution ($(...)) with cat, because << (a here-document) directly provides stdin input, which is where read takes its input from.
By using sbt_text=$(...) in your read command, you essentially made the output of the command substitution ($(...)) part of the (ultimately invalid) variable name rather than providing input, causing read to still wait for stdin input.
Using just assignment sbt_text=$(cat <<"EOF" ...) (without read) would almost have given you what you want (although at the expense of a call to external utility cat), except that the command substitution would have stripped any trailing newlines.
You should quote your variables:
echo "$sbt_text"
This prevents Bash from doing word splitting (and globbing, i.e. expanding wildcard characters like * and ?) after substituting your variable (parameter expansion). Note that setting IFS for read does not prevent word splitting on the echo command line. Try this for fun:
( IFS= ; echo $sbt_text )
vs.
IFS= echo $sbt_text ### won't work
The second example will not work - the subshell (( )) is necessary here because IFS is interpreted by the calling shell for word splitting in contrast to being interpreted by read in the question's code example.
Of course this works too (but "ruins" your IFS):
IFS= ; echo $sbt_text
I've read the man pages on echo, and it tells me that the -e parameter will allow an escaped character, such as an escaped n for newline, to have its special meaning. When I type the command
$ echo -e 'foo\nbar'
into an interactive bash shell, I get the expected output:
foo
bar
But when I use this same command (i've tried this command character for character as a test case) I get the following output:
-e foo
bar
It's as if echo is interpretting the -e as a parameter (because the newline still shows up) yet also it interprets the -e as a string to echo. What's going on here? How can I prevent the -e showing up?
You need to use #!/bin/bash as the first line in your script. If you don't, or if you use #!/bin/sh, the script will be run by the Bourne shell and its echo doesn't recognize the -e option. In general, it is recommended that all new scripts use printf instead of echo if portability is important.
In Ubuntu, sh is provided by a symlink to /bin/dash.
Different implementations of echo behave in annoyingly different ways. Some don't take options (i.e. will simply echo -e as you describe) and automatically interpret escape sequences in their parameters. Some take flags, and don't interpret escapes unless given the -e flag. Some take flags, and interpret different escape sequences depending on whether the -e flag was passed. Some will cause you to tear your hair out if you try to get them to behave in a predictable manner... oh, wait, that's all of them.
What you're probably seeing here is a difference between the version of echo built into bash vs /bin/echo or maybe vs. some other shell's builtin. This bit me when Mac OS X v10.5 shipped with a bash builtin echo that echoed flags, unlike what all my scripts expected...
In any case, there's a solution: use printf instead. It always interprets escape sequences in its first argument (the format string). The problems are that it doesn't automatically add a newline (so you have to remember do that explicitly), and it also interprets % sequences in its first argument (it is, after all, a format string). Generally, you want to put all the formatting stuff in the format string, then put variable strings in the rest of the arguments so you can control how they're interpreted by which % format you use to interpolate them into the output. Some examples:
printf "foo\nbar\n" # this does what you're trying to do in the example
printf "%s\n" "$var" # behaves like 'echo "$var"', except escapes will never be interpreted
printf "%b\n" "$var" # behaves like 'echo "$var"', except escapes will always be interpreted
printf "%b\n" "foo\nbar" # also does your example
Use
alias echo /usr/bin/echo
to force 'echo' invoking coreutils' echo which interpret '-e' parameter.
Try this:
import subprocess
def bash_command(cmd):
subprocess.Popen(['/bin/bash', '-c', cmd])
code="abcde"
// you can use echo options such as -e
bash_command('echo -e "'+code+'"')
Source: http://www.saltycrane.com/blog/2011/04/how-use-bash-shell-python-subprocess-instead-binsh/