The other day I stumbled upon a question on SO. If I wanted to extract the value of HOSTNAME in /etc/sysconfig/network which contains
NETWORKING=yes
HOSTNAME=foo
now I can do grep and cut to get the foo but there was some bash magic involved for a similar issue. I don't know what to search for that and I can't seem to find the question now. it involved something like #{HOSTNAME} . As if it was treating HOSTNAME as a key and foo as a value.
If that configuration file is compatible with shell syntax, simply include it as a shell script. IIRC the files in /etc/sysconfig on Red Hat-like distributions are indeed designed to be parsable by a shell. Note that this means that
If shell special characters may end up in a variable's value, they must be properly quoted. For example, var="value with spaces" requires the quotes. var="with\$dollar" requires the backslash.
The script may run arbitrary code that will be executed, so this is only ok if you trust its content.
If these assumptions are valid, then you can go the simple route:
. /etc/sysconfig/network
echo "$HOSTNAME"
Regarding the quoting and braces, see $VAR vs ${VAR} and to quote or not to quote.
Related
I read in http://wiki.bash-hackers.org/howto/conffile that it is recommended to put double quotes around values in BASH configuration files which are to be sourced into a script.
The file to be sourced should be formated in key="value" format,
otherwise bash will try to interpret commands
However, I am not aware of any difference in BASH's behavior when sourcing a configuration file if the value has double quotes or not, assuming there is no whitespace in the value. I'm sure there are some more complex cases where the double quotes are vital (e.g. using other variables as the value), but for the simple cases below, would double quotes cause BASH to behave any differently, even if the difference is only behind-the-scenes? I'm wondering if the first configuration file below could cause BASH to search for a named foobar before assigning it as a string, but from my testing it doesn't appear to do so.
# Configuration file 1
myDir=/var/tmp/test/
myString=foobar
myInteger=20
# Configuration file 2
myDir="/var/tmp/test/"
myString="foobar"
myInteger="20"
source configurationFile1
echo "$myDir"
echo "$myString"
echo "$myInteger"
source configurationFile2
echo "$myDir"
echo "$myString"
echo "$myInteger"
It's a style issue. In the examples you show, the quotes aren't strictly necessary. myDir=/var/tmp/text and myDir="/var/tmp/text" do exactly the same thing. Other values may require quotes to make the assignment correct.
The allusion is to the fact that these aren't really configuration files; they're just bash scripts that are intended to contain only assignments. Something like
foo=bar baz
is not an assignment; it's a simple command that tries to run baz with a variable named foo in its environment. Here, quotes are required:
foo="bar baz"
to make a proper assignment, in contrast to other "actual" configuration file formats where everything following the = (and optionally some post-= whitespace`) is considered part of the value to be assigned.
I've written a bunch of environment variables in Docker format, but now I want to use them outside of that context. How can I source them with one line of bash?
Details
Docker run and compose have a convenient facility for importing a set of environment variables from a file. That file has a very literal format.
The value is used as is and not modified at all. For example if the value is surrounded by quotes (as is often the case of shell variables), the quotes are included in the value passed
Lines beginning with # are treated as comments and are ignored
Blank lines are also ignored.
"If no = is provided and that variable is…exported in your local environment," docker "passes it to the container"
Thankfully, whitespace before the = will cause the run to fail
so, for example, this env-file:
# This is a comment, with an = sign, just to mess with us
VAR1=value1
VAR2=value2
USER
VAR3=is going to = trouble
VAR4=this $sign will mess with things
VAR5=var # with what looks like a comment
#VAR7 =would fail
VAR8= but what about this?
VAR9="and this?"
results in these env variables in the container:
user=ubuntu
VAR1=value1
VAR2=value2
VAR3=is going to = trouble
VAR4=this $sign will mess with things
VAR5=var # with what looks like a comment
VAR8= but what about this?
VAR9="and this?"
The bright side is that once I know what I'm working with, it's pretty easy to predict the effect. What I see is what I get. But I don't think bash would be able to interpret this in the same way without a lot of changes. How can I put this square Docker peg into a round Bash hole?
tl;dr:
source <(sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list)
You're probably going to want to source a file, whose contents
are executed as if they were printed at the command line.
But what file? The raw docker env-file is inappropriate, because it won't export the assigned variables such that they can be used by child processes, and any of the input lines with spaces, quotes, and other special characters will have undesirable results.
Since you don't want to hand edit the file, you can use a stream editor to transform the lines to something more bash-friendly. I started out trying to solve this with one or two complex Perl 5 regular expressions, or some combination of tools, but I eventually settled on one sed command with one simple and two extended regular expressions:
sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list
This does a lot.
The first expression prepends export to any line whose first character is anything but #.
As discussed, this makes the variables available to anything else you run in this session, your whole point of being here.
The second expression simply inserts a single-quote after the first = in a line, if applicable.
This will always enclose the whole value, whereas a greedy match could lop off some of (e.g.) VAR3, for example
The third expression appends a second quote to any line that has at least one =.
it's important here to match on the = again so we don't create an unmatched quotation mark
Results:
# This is a comment, with an =' sign, just to mess with us'
export VAR1='value1'
export VAR2='value2'
export USER
export VAR3='is going to = trouble'
export VAR4='this $sign will mess with things'
export VAR5='var # with what looks like a comment'
#VAR7 ='would fail'
export VAR8=' but what about this?'
export VAR9='"and this?"'
Some more details:
By wrapping the values in single-quotes, you've
prevented bash from assuming that the words after the space are a command
appropriately brought the # and all succeeding characters into the VAR5
prevented the evaluation of $sign, which, if wrapped in double-quotes, bash would have interpreted as a variable
Finally, we'll take advantage of process substitution to pass this stream as a file to source, bring all of this down to one line of bash.
source <(sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list)
Et voilà!
I need to write a shell script such that I have to read .sh script and find a particular variable (for example, Variable_Name="variable1") and take out is value(variable1).
In other shell script if Variable_Name is used I need to replace it with its Value(variable1)
A simple approach, to build on, might be:
assignment=$(echo 'Variable_Name="variable1"' | sed -r 's/Variable_Name=(.*)/\1/')
echo $assignment
"variable1"
Depending on variable type, the value might be quoted or not, quoted with single apostrophs or quotes. That might be neccessary (String with or without blanks) or superflous. Behind the assignment there might be furter code:
pi=3.14;v=42;
or a comment:
user=janis # Janis Joplin
it might be complicated:
expr="foobar; O'Reilly " # trailing blank important
But only you may know, how complicated it might get. Maybe the simple case is already sufficient. If the new script looks similar, it might work, or not:
targetV=INSERT_HERE; secondV=23
# oops: secondV accidnetally hidden:
targetV="foobar; O'Reilly " # trailing blank important; secondV=23
If the second script is under your control, you can prevent such problems easily. If source and target language are identical, what worked here should work there too.
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
I'm having to code a subversion hook script, and I found a few examples online, mostly python and perl. I found one or two shell scripts (bash) as well. I am confused by a line and am sorry this is so basic a question.
FILTER=".(sh|SH|exe|EXE|bat|BAT)$"
The script later uses this to perform a test, such as (assume EXT=ex):
if [[ "$FILTER" == *"$EXT"* ]]; then blah
My problem is the above test is true. However, I'm not asking you to assist in writing the script, just explaining the initial assignment of FILTER. I don't understand that line.
Editing in a closer example FILTER line. Of course the script, as written does not work, because 'ex' returns true, and not just 'exe'. My problem here is only, however, that I don't understant the layout of the variable assignment itself.
Why is there a period at the beginning? ".(sh..."
Why is there a dollar sign at the end? "...BAT)$"
Why are there pipes between each pattern? "sh|SH|exe"
You probably looking for something as next:
FILTER="\.(sh|SH|exe|EXE|bat|BAT)$"
for EXT
do
if [[ "$EXT" =~ $FILTER ]];
then
echo $EXT extension disallowed
else
echo $EXT is allowed
fi
done
save it to myscript.sh and run it as
myscript.sh bash ba.sh
and will get
bash is allowed
ba.sh extension disallowed
If you don't escape the "dot", e.g. with the FILTER=".(sh|SH|exe|EXE|bat|BAT)$" you will get
bash extension disallowed
ba.sh extension disallowed
What is (of course) wrong.
For the questions:
Why is there a period at the beginning? ".(sh..."
Because you want match .sh (as extension) and not for example bash (without the dot). And therefore the . must be escaped, like \. because the . in regex mean "any character.
Why is there a dollar sign at the end? "...BAT)$"
The $ mean = end of string. You want match file.sh and not file.sh.jpg. The .sh should be at the end of string.
Why are there pipes between each pattern? "sh|SH|exe"
In the rexex, the (...|...|...) construction delimites the "alternatives". As you sure quessed.
You really need read some "regex tutorial" - it is more complicated - and can't be explained in one answer.
Ps: NEVER use UPPERCASE variable names, they can collide with environment variables.
This just assigns a string to FILTER; the contents of that string have no special meaning. When you try to match it against the pattern *ex*, the result is true assuming that the value of $FILTER consists the string ex surrounded by anything on either side. This is true; ex is a substring of exe.
FILTER=".(sh|SH|exe|EXE|bat|BAT)$"
^^
|
+---- here is the "ex" from the pattern.
As I can this is similar to regular expression pattern:
In regular expressions the string start with can be show with ^, similarly in this case . represent seems doing that.
In the bracket you have exact string, which represents what the exact file extensions would be matched, they are 'Or' by using the '|'.
And at the end the expression should only pick the string will '$' or end point and not more than.
I would say that way original author might have looked at it and implemented it.