I'm using systemd on debian jessie to control a service to which I'm feeding environment variables through the EnvironmentFile=/etc/default/myservice file
in this file I have a variable which is a public key
JWT_PUB_KEY="-----BEGIN FOO BAR KEY-----
MIIBgjAcBgoqhkiG9w0BDAEDMA4ECKZesfWLQOiDAgID6ASCAWBu7izm8N4V
2puRO/Mdt+Y8ceywxiC0cE57nrbmvaTSvBwTg9b/xyd8YC6QK7lrhC9Njgp/
...
-----END FOO BAR KEY-----"
putting it like that does not please systemd which report an error (though doing a source in bash of the same file works correctly)
the documentation of systemd report that you can have multiline variable by ending each file with a \ but that it concatenate each line (so my program receive the whole under one line, which is no more a valid public key)
Is there a known way to preserve the end of line ? without resorting to hack like putting \n which i them 'interpret' in my application code ?
The claim in the systemd.exec documentation that within the EnvironmentFile, “C escapes are supported, but not most control characters. "\t" and "\n" can be used to insert tabs and newlines within EnvironmentFile=.” is completely false. Instead, the allowed quotes and escapes are the same as in a POSIX shell. Like in sh, a multiline value surrounded by single or double quotes will turn into a multiline value in the environment; you don’t need \n and \t, which are meaningless in sh (without quotes, they are interpreted as n and t; within quotes, they are interpreted as \n and \t. They never become newline and tab).
And if you end a line with \ in double quotes or no quotes, then this is a line continuation, and the newline is discarded, just like in sh.
I opened PR 21908 to fix this documentation.
Thanks to #yonran that improved the documentation in his pull request: https://github.com/systemd/systemd/pull/21908
So the solution is quite easy, you just have to put the value between single quotes. Warning, this solution may only works in recent version of systemd (tested with version >= 250).
JWT_PUB_KEY='-----BEGIN FOO BAR KEY-----
MIIBgjAcBgoqhkiG9w0BDAEDMA4ECKZesfWLQOiDAgID6ASCAWBu7izm8N4V
2puRO/Mdt+Y8ceywxiC0cE57nrbmvaTSvBwTg9b/xyd8YC6QK7lrhC9Njgp/
...
-----END FOO BAR KEY-----'
If I understand correctly the updated documentation, this is the only solution to obtain new lines (0x0A aka \n characters) inside an environment variable.
And this is clearly impossible to insert a new line by using the \n characters sequence inside the key value.
But after further tests, using double quotes also works (systemd does not complain, at least with systemd version 250).
Here my test files:
/etc/test-env.txt
JWT_PUB_KEY="-----BEGIN FOO BAR KEY-----
MIIBgjAcBgoqhkiG9w0BDAEDMA4ECKZesfWLQOiDAgID6ASCAWBu7izm8N4V
2puRO/Mdt+Y8ceywxiC0cE57nrbmvaTSvBwTg9b/xyd8YC6QK7lrhC9Njgp/
-----END FOO BAR KEY-----"
/etc/systemd/system/test-env.service
[Unit]
Description=Test env
[Service]
Type=simple
ExecStart=/usr/bin/test-dump-env.py JWT_PUB_KEY
EnvironmentFile=/etc/test-env.txt
/usr/bin/test-dump-env.py
#!/bin/env python3
import os, sys
s = os.environ.get(sys.argv[1])
h = ":".join("{:02x}".format(ord(c)) for c in s)
print(s)
print("***---***")
print(h)
To test it:
systemctl restart test-env.service; sleep 1; systemctl status -l test-env.service
So I do not understand why it failed in your case (You may be using a too old / broken version of systemd)
As you suspected, Systemd accepts \n inside environment variable definitions. You need not perform and special parsing, just add appropriate \ns where they are needed and escape the actual newlines. Systemd should handle the rest and turn them into a linefeed literal. In your case this would look something like this:
JWT_PUB_KEY="-----BEGIN FOO BAR KEY-----\n\
MIIBgjAcBgoqhkiG9w0BDAEDMA4ECKZesfWLQOiDAgID6ASCAWBu7izm8N4V\n\
2puRO/Mdt+Y8ceywxiC0cE57nrbmvaTSvBwTg9b/xyd8YC6QK7lrhC9Njgp/\n\
...\n\
-----END FOO BAR KEY-----"
You shouldn't have to interpret (parse?) the newline characters on your end, that should all happen automatically, as explained here , meaning that adding '\n' at the end of each line should do the trick.
Depending on your systemd version, you cannot use the above answers. (you can check the version by systemctl --version.) Actually any versions before this PR was merged should not be able to use multiline environment variables. For example, Amazon Linux 2 seems to use systemd v219, which was released on 2015/02.
Manually upgrading systemd is usually not recommended (ref), so there are not many things you can do about this issue.
As a workaround, I encoded the value in base64 and decoded it in the application.
# encode
# you need -w 0 option to wrap newlines
echo "YOUR_MULTILINE_VARIABLE" | base64 -w 0
Related
~ ❯ export TEST_KEY='hello\nworld' && iex
iex(1)> System.get_env("TEST_KEY")
"hello\\nworld"
When running System.get_env/1 on a string with \n it inserts an extra backslash, any way of preventing this behaviour?
It does not insert anything, it fairly reads the environment variable and, because backslashes are to be escaped in double-quotes, prints it as you see.
What you think is “new line” is nothing but the escape sequence. Here is an excerpts from e. g. echo man:
-e enable interpretation of backslash escapes
-E disable interpretation of backslash escapes
The default behaviour in raw shell:
❯ echo -E 'hello\nworld'
hello\nworld
The fact, that you see a new line there in echo by default, or whatever is a side effect by the interpreter, whoever this interpreter is. The value itself contains a backslash and n ASCII symbols, and no magic besides.
That said, if one wants to have new lines in place of \n sequence in the value, one must apply the escaping themselves.
"TEST_KEY"
|> System.get_env()
|> to_string() # to NOT raise on absent value, redundant
|> String.replace("\\n", "\n")
This is really a shell question, not an elixir question. See this related answer that shows how to set a new line in an environment variable in bash. Using it from elixir:
$ TEST_KEY=$'hello\nworld' elixir -e 'IO.puts(System.get_env("TEST_KEY"))'
hello
world
I've written a bunch of environment variables in Docker format, but now I want to use them outside of that context. How can I source them with one line of bash?
Details
Docker run and compose have a convenient facility for importing a set of environment variables from a file. That file has a very literal format.
The value is used as is and not modified at all. For example if the value is surrounded by quotes (as is often the case of shell variables), the quotes are included in the value passed
Lines beginning with # are treated as comments and are ignored
Blank lines are also ignored.
"If no = is provided and that variable is…exported in your local environment," docker "passes it to the container"
Thankfully, whitespace before the = will cause the run to fail
so, for example, this env-file:
# This is a comment, with an = sign, just to mess with us
VAR1=value1
VAR2=value2
USER
VAR3=is going to = trouble
VAR4=this $sign will mess with things
VAR5=var # with what looks like a comment
#VAR7 =would fail
VAR8= but what about this?
VAR9="and this?"
results in these env variables in the container:
user=ubuntu
VAR1=value1
VAR2=value2
VAR3=is going to = trouble
VAR4=this $sign will mess with things
VAR5=var # with what looks like a comment
VAR8= but what about this?
VAR9="and this?"
The bright side is that once I know what I'm working with, it's pretty easy to predict the effect. What I see is what I get. But I don't think bash would be able to interpret this in the same way without a lot of changes. How can I put this square Docker peg into a round Bash hole?
tl;dr:
source <(sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list)
You're probably going to want to source a file, whose contents
are executed as if they were printed at the command line.
But what file? The raw docker env-file is inappropriate, because it won't export the assigned variables such that they can be used by child processes, and any of the input lines with spaces, quotes, and other special characters will have undesirable results.
Since you don't want to hand edit the file, you can use a stream editor to transform the lines to something more bash-friendly. I started out trying to solve this with one or two complex Perl 5 regular expressions, or some combination of tools, but I eventually settled on one sed command with one simple and two extended regular expressions:
sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list
This does a lot.
The first expression prepends export to any line whose first character is anything but #.
As discussed, this makes the variables available to anything else you run in this session, your whole point of being here.
The second expression simply inserts a single-quote after the first = in a line, if applicable.
This will always enclose the whole value, whereas a greedy match could lop off some of (e.g.) VAR3, for example
The third expression appends a second quote to any line that has at least one =.
it's important here to match on the = again so we don't create an unmatched quotation mark
Results:
# This is a comment, with an =' sign, just to mess with us'
export VAR1='value1'
export VAR2='value2'
export USER
export VAR3='is going to = trouble'
export VAR4='this $sign will mess with things'
export VAR5='var # with what looks like a comment'
#VAR7 ='would fail'
export VAR8=' but what about this?'
export VAR9='"and this?"'
Some more details:
By wrapping the values in single-quotes, you've
prevented bash from assuming that the words after the space are a command
appropriately brought the # and all succeeding characters into the VAR5
prevented the evaluation of $sign, which, if wrapped in double-quotes, bash would have interpreted as a variable
Finally, we'll take advantage of process substitution to pass this stream as a file to source, bring all of this down to one line of bash.
source <(sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list)
Et voilà!
The other day I stumbled upon a question on SO. If I wanted to extract the value of HOSTNAME in /etc/sysconfig/network which contains
NETWORKING=yes
HOSTNAME=foo
now I can do grep and cut to get the foo but there was some bash magic involved for a similar issue. I don't know what to search for that and I can't seem to find the question now. it involved something like #{HOSTNAME} . As if it was treating HOSTNAME as a key and foo as a value.
If that configuration file is compatible with shell syntax, simply include it as a shell script. IIRC the files in /etc/sysconfig on Red Hat-like distributions are indeed designed to be parsable by a shell. Note that this means that
If shell special characters may end up in a variable's value, they must be properly quoted. For example, var="value with spaces" requires the quotes. var="with\$dollar" requires the backslash.
The script may run arbitrary code that will be executed, so this is only ok if you trust its content.
If these assumptions are valid, then you can go the simple route:
. /etc/sysconfig/network
echo "$HOSTNAME"
Regarding the quoting and braces, see $VAR vs ${VAR} and to quote or not to quote.
I've been learning about remote/arbitrary command execution. In doing so, I came across some Ruby I thought would be fun to try and exploit.
I've been somewhat successful as I managed to get it to run the 'ls' command, but I can't work out how to add space characters into my commands. If I add a space in, the parse method that URI calls throws an exception.
Here's the code I was trying to exploit:
injection = "www.google.com';ls;#"
require 'uri'
URI.parse(injection)
puts `curl '#{injection}'`
So your challenge, should you choose to accept it, is to run an 'ls -l' command instead of 'ls' by only changing the injection string. You may not change anything but the first line.
Things I've tried:
ls%2f-l - # Doesn't raise an exception but unix doesn't unescape CGI encodings.
ls\x20-l - # Raises an exception because Ruby parses the UTF-8.
# Other various escape combinations (\\x20, etc)
Maybe it's not possible?
Thanks
You can use the Internal Field Separator (<space><tab><newline>). Since this is what the shell separates with anyway, it will accept it as a separator.
injection = "www.google.com';ls$IFS-l;#"
(BTW, thanks for a nice Saturday night puzzle.)
Is - it's possible. Just put your string in quotes:
1) from a command prompt:
two strings # No quote: the shell sees two strings
"one string" # with single (') or double quotes (") the shell sees only one string
2) from a string literal
mystring = "\"this will be interpreted as one string\"";
methods() {
cat <<-!
start
stop
restart
reload
status
methods
!
}
Is this correct I am getting error
syntax error: unexpected end of file
For here-docs in ancient shells, you have to match the tag exactly. That means:
methods() {
cat <<!
start
stop
restart
reload
status
methods
!
}
Yes, at the start of the line, although you could do tricky things like cat <<'^I!' to set the marker to a single tab followed by !.
Now bash (and possibly earlier shells) fixed that with the <<- variant which strips off all leading tabs from your data lines and the end marker before processing. That way, you could still indent nicely:
methods() {
cat <<-!
start
stop
restart
reload
status
methods
!
}
But, note the proviso: it strips tabs, not whitespace in general. If you have spaces (or any non-tab character, printable or otherwise) anywhere before that ! character, it won't work.
If you're using vi, you can enter :set list to see the non-printable characters a bit better, otherwise xd or od -xcb can give you a hex dump of your file.