There are a lot of examples here how to source a dotenv file in bash but has anyone one a method that achieves the same with dash (which is the default shell for minimal Debian installations)?
The solution should look like this:
$ some foo my-command-using-env-vars
e.g.
$ env $(cat .env) my-command-using-env-vars
And it is important that the solution supports multiline values with spaces like:
SSH_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\nfoo\nbar\baz"
and special characters like hash within quotes:
SPECIAL="foo#bar"
It seems that your problem is not so much that you're using dash but that you want to support \n escapes. The following works in dash and supports \n escapes:
eval "$(echo $(cat .env))" my-command-using-env-vars
That's because unlike in bash the built-in echo in dash supports backslash escapes even without the -e option. The following works in both bash and dash, provided that the non-built-in, system echo supports the -e option:
eval "$(env echo -e $(cat .env))" my-command-using-env-vars
Note that both approaches will also handle other backslash escapes (either POSIX or GNU), possibly in a different way than you expect.
Some technical notes:
$(cat .env)
performs Field Splitting, converting any newline in file .env into spaces.
"$(env echo -e ...)"
expands backslash escapes regardless of the current shell by invoking echo -e via env. The double quotes disable field splitting, so that newlines are preserved.
Related
I'm trying to port some code from bash 5.1 to 4.2.46. One function which tries to strip color codes from a specifically formatted string stopped working.
This is a sample string text in such format. I turn on extended globbing for this.
text="$(printf -- "%b%s%b" "\[\e[31m\]" "hello" "\[\e[0m\]")"
shopt -s extglob
In bash 5.1, this parameter expansion works to remove all the color codes and escape characters
bash-5.1$ echo "${text//$'\[\e'\[/}"
31m\]hello0m\]
bash-5.1$ echo "${text//$'\[\e'\[+([0-9])/}"
m\]hellom\]
bash-5.1$ echo "${text//$'\[\e'\[+([0-9])m$'\]'/}"
hello
In bash 4.2.46, I start getting a different behavior as I build up the parameter expansion.
bash-4.2.46$ echo "${text//$'\[\e'\[/}"
\31m\]hello\0m\]
bash-4.2.46$ echo "${text//$'\[\e'\[+([0-9])/}"
\[\]hello\[\] ## no longer matches because `+([0-9])` doesn't follow `\[`
The difference comes from this line: echo "${text//$'\[\e'\[/}"
bash-5.1: 31m\]hello0m\]
bash-4.2.46: \31m\]hello\0m\]
Here's what printf "%q" "${text//$'\[\e'\[/}" shows:
bash-5.1: 31m\\\]hello0m\\\]
bash-4.2.46: \\31m\\\]hello\\0m\\\]
Where is the extra \ coming from in 4.2.26?
Even when I try to remove it, the pattern stops matching:
bash-4.2.46$ echo "${text//$'\[\e'\[\\/}"
\[\]hello\[\] ## no longer matches because `\\` doesn't follow `\[`
I'm guessing there may be a bug related to parameter expansion, backslash escaping, and extended globbing.
I am aiming to write code that works on bash 4.0 onward, so I'm looking for a workaround primarily. An explanation (bug report, etc.) to why the behavior difference happens would be great, though.
Seems like a bug in bash. By bisecting the available versions, I found that 4.2.53(1)-release was the last version with this bug. Version 4.3.0(1)-release fixed the problem.
The list of changes mentions a few bug fixes in this direction. Maybe it was one of below bugfixes:
This document details the changes between this version, bash-4.3-alpha,
and the previous version, bash-4.2-release.
[...]
zz. When using the pattern substitution word expansion, bash now runs the
replacement string through quote removal, since it allows quotes in that
string to act as escape characters. This is not backwards compatible, so
it can be disabled by setting the bash compatibility mode to 4.2.
[...]
eee. Fixed a logic bug that caused extended globbing in a multibyte locale to
cause failures when using the pattern substititution word expansions.
Workaround
Instead of using parameter expansions with extglobs, use bash pattern matching with actual regexes (available in bash 3.0.0 and higher):
text=$'\[\e[31m\]hello\[\e[0m\]'
while [[ "$text" =~ (.*)$'\[\e['[0-9]*'m\]'(.*) ]]; do
text="${BASH_REMATCH[1]}${BASH_REMATCH[2]}"
done
echo "$text"
or rely on an external (but posix standarized) tool like sed:
text=$'\[\e[31m\]hello\[\e[0m\]'
text=$(sed $'s#\\\[\e[[0-9]*m\\\]##g' <<< "$text")
echo "$text"
The problem seems to be parsing $'...' inside ${test//<here>} when inside " quotes.
$ test='f() { "${text//\[$'\''\e'\''\[+([0-9])/}"; }; printf "%q\n" "$(declare -f f)"'; echo -n 'bash4.1 '; docker run bash:4.1 bash -c "$test" ; echo -n 'bash5.1 '; bash -c "$test"
bash4.1 $'f () \n{ \n "${text//\\[\E\\[+([0-9])/}"\n}'
bash5.1 $'f () \n{ \n "${text//\\[\'\E\'\\[+([0-9])/}"\n}'
Just use a variable.
esc=$'\e'
echo "${text//\\\[$esc\[+([0-9])/}"
I'd like to echo something to a file that contains new line escape sequences, however I would like them to remain escaped. I'm looking for basically the opposite to this question.
echo "part1\npart2" >> file
I would like to look like this in the file
$ cat file
old
part1\npart2
but it looks like
$ cat file
old
part1
part2
This is a good example of why POSIX recommends using printf instead of echo (see here, under "application usage"): you don't know what you get with echo1.
You could get:
A shell builtin echo that does not interpret backslash escapes by default
Example: the Bash builtin echo has an -e option to enable backslash escape interpretation and checks the xpg_echo shell option
A shell builtin echo that interprets backslash escapes by default
Examples: zsh, dash
A standalone executable /bin/echo: probably depends on which one – GNU Coreutils echo understands the -e option, like the Bash builtin
The POSIX spec says this (emphasis mine):
The following operands shall be supported:
string
A string to be written to standard output. If the first operand is -n, or if any of the operands contain a <backslash> character, the results are implementation-defined.
So, for a portable solution, we can use printf:
printf '%s\n' 'part1\npart2' >> file
where the \n in the format string will always be interpreted, and the \n in the argument will never be interpreted, resulting in
part1\npart2
being appended to file.
1 For an exhaustive overview of various behaviours for echo and printf, see echo(1) and printf(1) on in-ulm.de.
So Im removing special characters from filenames and replacing with spaces. I have all working apart from files with single backslashes contained therein.
Note these files are created in the Finder on OS X
old_name="testing\this\folder"
new_name=$(echo $old_name | tr '<>:\\#%|?*' ' ');
This results in new_name being "testing hisolder"
How can I just removed the backslashes and not the preceding character?
This results in new_name being "testing hisolder"
This string looks like the result of echo -e "testing\this\folder", because \t and \f are actually replaced with the tabulation and form feed control characters.
Maybe you have an alias like alias echo='echo -e', or maybe the implementation of echo in your version of the shell interprets backslash escapes:
POSIX does not require support for any options, and says that the
behavior of ‘echo’ is implementation-defined if any STRING contains a
backslash or if the first argument is ‘-n’. Portable programs can use
the ‘printf’ command if they need to omit trailing newlines or output
control characters or backslashes.
(from the info page)
So you should use printf instead of echo in new software. In particular, echo $old_name should be replaced with printf %s "$old_name".
There is a good explanation in this discussion, for instance.
No need for printf
As #mklement0 suggested, you can avoid the pipe by means of the Bash here string:
tr '<>:\\#%|?*' ' ' <<<"$old_name"
Ruslan's excellent answer explains why your command may not be working for you and offers a robust, portable solution.
tl;dr:
You probably ran your code with sh rather than bash (even though on macOS sh is Bash in disguise), or you had shell option xpg_echo explicitly turned on.
Use printf instead of echo for portability.
In Bash, with the default options and using the echo builtin, your command should work as-is (except that you should double-quote $old_name for robustness), because echo by default does not expand escape sequences such as \t in its operands.
However, Bash's echo can be made to expand control-character escape sequences:
explicitly, by executing shopt -s xpg_echo
implicitly, if you run Bash as sh or with the --posix option (which, among other options and behavior changes, activates xpg_echo)
Thus, your symptom may have been caused by running your code from a script with shebang line #!/bin/sh, for instance.
However, if you're targeting sh, i.e., if you're writing a portable script, then echo should be avoided altogether for the very reason that its behavior differs across shells and platforms - see Ruslan's printf solution.
As an aside: perhaps a more robust approach to your tr command is a whitelisting approach: stating only the characters that are explicitly allowed in your result, and excluding other with the -C option:
old_name='testing\this\folder'
new_name=$(printf '%s' "$old_name" | tr -C '[:alnum:]_-' ' ')
That way, any characters that aren't either letters, numbers, _, or - are replaced with a space.
With Bash, you can use parameter expansion:
$ old_name="testing\this\folder"
$ new_name=${old_name//[<>:\\#%|?*]/ }
$ echo $new_name
testing this folder
For more, please refer to the Bash manual on shell parameter expansion.
I think your test case is missing proper escaping for \, so you're not really testing the case of a backslash contained in a string.
This worked for me:
old_name='testing\\this\\folder'
new_name=$(echo $old_name | tr '<>:\\#%|?*' ' ');
echo $new_name
# testing this folder
I would like to use the bash here doc syntax to build a long string. I would like the heredoc to ignore newlines/spaces/tabs even when I use newlines for code clarity.
I thought this would work:
#!/bin/bash
#http://unix.stackexchange.com/questions/20035/how-to-add-newlines-into-variables-in-bash-script
IFS= read -r -d '' NS_LOG<<-EOF
*=error|warn|prefix_node|prefix_func
:PointToPointNetDevice
:ClockTest
:ClockPerfect
:TcpTestSuite
:TcpRxBuffer
:TcpTxBuffer
:TcpHeader=*
:TcpL4Protocol
:TraceHelper:PointToPointHelper
EOF
echo $NS_LOG
export NS_LOG
but somewhere bash appends spaces between lines and instead of having the desired
*=error|warn|prefix_node|prefix_func:PointToPointNetDevice:ClockTest:ClockPerfect:Clock
I have when running $ ./launch_myscript.sh:
*=error|warn|prefix_node|prefix_func :PointToPointNetDevice :ClockTest :ClockPerfect :Clock etc...
My bash --version:
GNU bash, version 4.3.30(1)-release (x86_64-pc-linux-gnu)
I just saw in the recommended posts this one Bash: Why is echo adding extra space?. How can I prevent NS_LOG from being considered as several arguments? Ultimately the goal is to export that variable.
Your read command is very explicitly treating newlines as data: By clearing IFS and passing -d '', you tell read not to treat whitespace characters as special; since they're not special, they go into the output variable like everything else. However, you can take them out later:
IFS= read -r -d '' NS_LOG <<'EOF'
...content...
EOF
NS_LOG=${NS_LOG//[[:space:]]/} ## replace all whitespace with the empty string
printf '%s\n' "$NS_LOG" ## the quotes are important!
See this snippet run, and its output, at http://ideone.com/fWhzBB.
Notes:
<<'EOF' prevents expansions from occurring within the heredoc itself; with <<EOF, $foo, $(foo), etc. would be special.
<<- only trims leading tab characters, not any other form of whitespace; it's typically safer to do without.
echo $foo string-splits and glob-expands the contents of $foo, passing each word created by this process as a separate argument; echo then places spaces between each argument. echo "$foo" ensures that the entire expansion is treated as a single word. See BashPitfalls #14.
Using echo with nontrivial or unknown data is advised against in the relevant portion of the POSIX specification; printf is the preferred substitute. POSIX echo is explicitly allowed to behave in undefined ways when content contains backslash literals, and the BSD- and AT&T-derived forms of the command are explicitly incompatible, both with each other and with the common GNU implementation (providing an -e flag, which the POSIX spec requires to simply print -e on its output).
things aren't that difficult. here is another solution:
var="`tr -d '[:space:] <<EOF'
your
text
with
lot of spaces
EOF`"
result:
$ echo "$var"
yourtextwithlotofspaces
I've read the man pages on echo, and it tells me that the -e parameter will allow an escaped character, such as an escaped n for newline, to have its special meaning. When I type the command
$ echo -e 'foo\nbar'
into an interactive bash shell, I get the expected output:
foo
bar
But when I use this same command (i've tried this command character for character as a test case) I get the following output:
-e foo
bar
It's as if echo is interpretting the -e as a parameter (because the newline still shows up) yet also it interprets the -e as a string to echo. What's going on here? How can I prevent the -e showing up?
You need to use #!/bin/bash as the first line in your script. If you don't, or if you use #!/bin/sh, the script will be run by the Bourne shell and its echo doesn't recognize the -e option. In general, it is recommended that all new scripts use printf instead of echo if portability is important.
In Ubuntu, sh is provided by a symlink to /bin/dash.
Different implementations of echo behave in annoyingly different ways. Some don't take options (i.e. will simply echo -e as you describe) and automatically interpret escape sequences in their parameters. Some take flags, and don't interpret escapes unless given the -e flag. Some take flags, and interpret different escape sequences depending on whether the -e flag was passed. Some will cause you to tear your hair out if you try to get them to behave in a predictable manner... oh, wait, that's all of them.
What you're probably seeing here is a difference between the version of echo built into bash vs /bin/echo or maybe vs. some other shell's builtin. This bit me when Mac OS X v10.5 shipped with a bash builtin echo that echoed flags, unlike what all my scripts expected...
In any case, there's a solution: use printf instead. It always interprets escape sequences in its first argument (the format string). The problems are that it doesn't automatically add a newline (so you have to remember do that explicitly), and it also interprets % sequences in its first argument (it is, after all, a format string). Generally, you want to put all the formatting stuff in the format string, then put variable strings in the rest of the arguments so you can control how they're interpreted by which % format you use to interpolate them into the output. Some examples:
printf "foo\nbar\n" # this does what you're trying to do in the example
printf "%s\n" "$var" # behaves like 'echo "$var"', except escapes will never be interpreted
printf "%b\n" "$var" # behaves like 'echo "$var"', except escapes will always be interpreted
printf "%b\n" "foo\nbar" # also does your example
Use
alias echo /usr/bin/echo
to force 'echo' invoking coreutils' echo which interpret '-e' parameter.
Try this:
import subprocess
def bash_command(cmd):
subprocess.Popen(['/bin/bash', '-c', cmd])
code="abcde"
// you can use echo options such as -e
bash_command('echo -e "'+code+'"')
Source: http://www.saltycrane.com/blog/2011/04/how-use-bash-shell-python-subprocess-instead-binsh/