I'm trying to interpolate variables inside of a bash heredoc:
var=$1
sudo tee "/path/to/outfile" > /dev/null << "EOF"
Some text that contains my $var
EOF
This isn't working as I'd expect ($var is treated literally, not expanded).
I need to use sudo tee because creating the file requires sudo. Doing something like:
sudo cat > /path/to/outfile <<EOT
my text...
EOT
Doesn't work, because >outfile opens the file in the current shell, which is not using sudo.
In answer to your first question, there's no parameter substitution because you've put the delimiter in quotes - the bash manual says:
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter expansion, command substitution, arithmetic expansion, or
pathname expansion is performed on word. If any characters in word are
quoted, the delimiter is the result of quote removal on word, and the
lines in the here-document are not expanded. If word is unquoted, all
lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion. [...]
If you change your first example to use <<EOF instead of << "EOF" you'll find that it works.
In your second example, the shell invokes sudo only with the parameter cat, and the redirection applies to the output of sudo cat as the original user. It'll work if you try:
sudo sh -c "cat > /path/to/outfile" <<EOT
my text...
EOT
Don't use quotes with <<EOF:
var=$1
sudo tee "/path/to/outfile" > /dev/null <<EOF
Some text that contains my $var
EOF
Variable expansion is the default behavior inside of here-docs. You disable that behavior by quoting the label (with single or double quotes).
As a late corolloary to the earlier answers here, you probably end up in situations where you want some but not all variables to be interpolated. You can solve that by using backslashes to escape dollar signs and backticks; or you can put the static text in a variable.
Name='Rich Ba$tard'
dough='$$$dollars$$$'
cat <<____HERE
$Name, you can win a lot of $dough this week!
Notice that \`backticks' need escaping if you want
literal text, not `pwd`, just like in variables like
\$HOME (current value: $HOME)
____HERE
Demo: https://ideone.com/rMF2XA
Note that any of the quoting mechanisms -- \____HERE or "____HERE" or '____HERE' -- will disable all variable interpolation, and turn the here-document into a piece of literal text.
A common task is to combine local variables with script which should be evaluated by a different shell, programming language, or remote host.
local=$(uname)
ssh -t remote <<:
echo "$local is the value from the host which ran the ssh command"
# Prevent here doc from expanding locally; remote won't see backslash
remote=\$(uname)
# Same here
echo "\$remote is the value from the host we ssh:ed to"
:
Related
I am new to BASH and to Julia, and I am trying to do the following steps:
Save a filepath in a variable
Use this variable in a julia command, which i managed to execute through bash by using a heredoc function
Hard-coded command that works:
cat << "EOF" | julia --project=.
module test
ARGS=["/path/To/My/Directory"]
include("nameOfMyProject.jl")
end
EOF
The problem is, that I want to exchange the hardcoded path with the value of a variable.
I tried:
path="/path/To/My/Directory"
cat << "EOF" | julia --project=.
module test
ARGS=[$path]
include("nameOfMyProject.jl")
end
EOF
However, that does not work. I read i should try to use <<EOF instead of << "EOF" but then I receive this error: syntax: "/" is not a unary operator
I am not sure how to interprete this, since it might actually access the variable and find the "/", but it does work with it if I use "/" in the hard-coded way... so why not there?
I am not sure if my problem lies in the heredoc function or in something with this variable, but I would be grateful for any help!
Solution for this problem:
I am adding this for reference, if someone else has the same problem. The format that does the trick nicely is:
path="/path/To/My/Directory"
julia --project=. <<EOF
module test
ARGS=["$path"]
include("nameOfMyProject.jl")
end
EOF
Thanks to Jetchisel and Jens
From the bash manual (emphasis mine):
Here Documents
This type of redirection instructs the shell to read input from the
current source until a line containing only delimiter (with no trailing
blanks) is seen. All of the lines read up to that point are then used
as the standard input (or file descriptor n if n is specified) for a
command.
The format of here-documents is:
[n]<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic
expansion, or pathname expansion is performed on word. If any part of
word is quoted, the delimiter is the result of quote removal on word,
and the lines in the here-document are not expanded. If word is
unquoted, all lines of the here-document are subjected to parameter
expansion, command substitution, and arithmetic expansion, the
character sequence \<newline> is ignored, and \ must be used to quote
the characters \, $, and `.
So all you need is
julia --project=. << EOF
...$FOO...
EOF
GNU Bash - 3.6.6 Here Documents
[n]<<[-]word
here-document
delimiter
If any part of word is quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion, the character sequence \newline is ignored, and ‘\’ must be used to quote the characters ‘\’, ‘$’, and ‘`’.
If I single-quote EOF, it works. I think because bash /bin/bash process to be invoked gets un-expanded strings and then the invoked process interprets the lines.
$ /bin/bash<<'EOF'
#!/bin/bash
echo $BASH_VERSION
EOF
3.2.57(1)-release
However, the below is causing an error. I thought BASH_VERSION would have been expanded and the version of current bash process is passed to the /bin/bash process to be invoked. But not working.
$ /bin/bash<<EOF
#!/bin/bash
echo $BASH_VERSION
EOF
/bin/bash: line 2: syntax error near unexpected token `('
/bin/bash: line 2: `echo 5.0.17(1)-release'
/bin/bash<<EOF
#!/bin/bash
echo $BASH_VERSION
EOF
As you can infer from the error message, the heredoc is being expanded to:
/bin/bash<<EOF
#!/bin/bash
echo 5.0.17(1)-release
EOF
It sounds like that's what you expect: it's being expanded to the outer shell's version. The problem isn't with the heredoc or the expansion; it's that unquoted parentheses are a syntax error. Try running just the echo command by hand and you'll get the same error:
$ echo 5.0.17(1)-release
bash: syntax error near unexpected token `('
To fix this, you could add extra quotes:
/bin/bash<<EOF
echo '$BASH_VERSION'
EOF
This will work and print the outer shell's version. I used single quotes to demonstrate that these quotes will not inhibit variable expansion. The outer shell doesn't see these quotes. Only the inner shell does.
(I also got rid of the #!/bin/bash shebang line. There's no need for it since you're explicitly invoking bash.)
However, quoting is not 100% robust. If $BASH_VERSION happened to contain single quotes you'd have a problem. The quotes make parentheses ( ) safe but they aren't foolproof. As a general technique, if you want this to be completely safe no matter what special characters are in play you'll have to jump through some ugly hoops.
Use printf '%q' to escape all special characters.
/bin/bash <<EOF
echo $(printf '%q' "$BASH_VERSION")
EOF
This will expand to echo 5.0.17\(1\)-release.
Pass it in as an environment variable and use <<'EOF' to disable interpolation inside the script.
OUTER_VERSION="$BASH_VERSION" /bin/bash <<'EOF'
echo "$OUTER_VERSION"
EOF
This would be my choice. I prefer use the <<'EOF' form whenever possible. Having the parent shell interpolate the script being passed to a child shell can be confusing and difficult to reason about. Also, the explicit $OUTER_VERSION variable makes it crystal clear what's happening.
Use bash -c 'script' instead of a heredoc and then pass the version in as a command-line argument.
bash -c 'echo "$1"' bash "$BASH_VERSION"
I might go with this for a single-line script.
If you don't quote EOF, variables in the heredoc are expanded by the original shell before passing it as input to the invoked shell. So it's equivalent to executing
echo 3.2.57(1)-release
in the invoked shell. That's not valid bash syntax, so you get an error.
Quoting the word prevents variable expansion, so the invoked shell receives $BASH_VERSION literally, and expands it itself.
In the first case, the quotes prevent any changes in the here document, so the sub-shell sees echo $BASH_VERSION and it expands the string and echoes it.
In the second case, the absence of quotes means that the first shell expands the information and it sees echo 3.2.57(1)-release, and if you type that at the command line, you get the syntax error.
If you used echo "$BASH_VERSION" in both, then both would work, but different shells would expand $BASH_VERSION.
domain="www.google.com"
echo -e "\e[1;34m"$domain"\e[0m"
I expected this to output www.google.com in green letters.
Instead I got
-e \e[1;34mwww.google.com\e[0m
Depending the environment or shell used can have an effect, one thing you could probably do is to use ANSI-C quoting:
echo $'\e[1;34m'${domain}$'\e[0m'
Words of the form $'string' are treated specially. The word expands to
string, with backslash-escaped characters replaced as specified by the
ANSI C standard.
https://www.gnu.org/software/bash/manual/html_node/ANSI_002dC-Quoting.html
If you run a script with sh script.sh, you're explicitly using sh as the shell rather than the one in the shebang line. That's bad news if sh isn't a link to bash. A plain sh shell may not support echo -e.
Type ./script.sh to use the interpreter in the shebang line.
I have just recently got back into learning bash. Currently working on a project of mine and when using sed I've run into an issue, I've tried looking around the web for help but haven't had any joy. I suspect as I may not be using the correct terminology so I can't find what I'm looking for. ANYHOW.
So in my script I'm trying to assign the output of date to a variable. Here's the line from my script.
origdate=$(date)
When I call it the output looks like this:
Wed Oct 5 19:40:45 BST 2016
Part of my script then generates a file and writes information to it, part of which I am trying to use sed to find lines and replace parts of it. This is the first I've been playing around with sed, I've used it successfully so far for my needs. However I'm getting stuck when I try this:
sed -i '/origdate=empty/c\'$origdate'' $sd/pingcheck-email-$job.txt
When I run the script and it gets to this line, this is the error I'm getting:
sed: can't read Oct: No such file or directory
sed: can't read 5: No such file or directory
sed: can't read 19:52:56: No such file or directory
sed: can't read BST: No such file or directory
sed: can't read 2016: No such file or directory
I suspect it's something to do with the spaces in the date (variable), my question is: how can I work around this? Can I get sed to 'ignore' the spaces? or should I just use cut to cut the field for the date, and set that to a variable and the same thing again to set the time to another variable?
Even if someone could kindly point me in the right direction that'd be great!
Thanks in advance!
double quote the variable
sed -i '/origdate=empty/c\'"$origdate"'' $sd/pingcheck-email-$job.txt
or alternatively, the whole script
sed -i "/origdate=empty/c\$origdate" $sd/pingcheck-email-$job.txt
The problem is not with sed but rather with how bash word splits on your date given your command.
Bash
In bash, word splitting is performed on the command line so that text is broken up into a list of arguments. To illustrate, I'm going to run a simple script that outputs the first argument only.
bash -c 'echo $1' ignored_0 foo bar
Think of bash -c 'echo $1' ignored_0 as the command (sed in your case) and foo bar as the arguments. In this case, foo bar is split into two arguments, foo and bar.
To pass foo bar in as the first parameter, you need to have the text in either single or double quotes. See the GNU manual on quoting.
bash -c 'echo $1' ignored_0 'foo bar'
bash -c 'echo $1' ignored_0 "foo bar"
Parameter expansion does not occur when the variable is inside a single quote.
var="foo bar"
bash -c 'echo $1' ignored_0 '$var'
bash -c 'echo $1' ignored_0 "$var"
NOTE: In the command `bash -c 'echo $1', I do not want $1 to expand before being passed as an argument to bash because that's part of the code I want to execute.
Parameter expansion occurs when variables are outside of quotes, but word splitting will apply after the parameter is expanded. From the bash man page in the Word Splitting section:
The shell scans the results of parameter expansion, command
substitution, and arithmetic expansion that did not occur within
double quotes for word splitting.
From the GNU bash manual on Word Splitting:
The shell scans the results of parameter expansion, command
substitution, and arithmetic expansion that did not occur within
double quotes for word splitting.
var="foo bar"
bash -c 'echo $1' ignored_0 $var
The last step in Shell Expansions in Quote Removal where unquoted quote characters are removed before being passed to commands. The following command shows that ''"" has no effect on the arguments passed.
bash -c 'echo $1' ignored_0 foo''""
Application
In your example, the trailing '' after $origdate is extraneous. The important part is that $origdate is not quoted so word splitting applies to the expanded variable.
When -e is not passed to the sed command, sed expects the expression to be in one argument, or word from bash. When you run your command, your expression is /origdate=empty/c\Wed and the rest of the date is considered to be files for the expression to be applied to.
The simple fix is to put double quotes around the string for which you want to prevent word splitting. I've modified the command so that anyone can run this example without having the files on their system.
In this example, the \ must be escaped so that it is not considered an escape character for $.
echo "origdate=empty" | sed "/origdate=empty/c\\$origdate"
You can also change the type of quotes you are using without affecting word splitting like so.
echo "origdate=empty" | sed '/origdate=empty/c\'"$origdate"
You need escape by double slash
\ / \%
I am trying to create and use variables inside heredoc like this,
#!bin/bash
sudo su - postgres <<EOF
IP="XYZ"
echo "$IP"
EOF
This doesn't work right and I get a blank line as echo.
But if I use quotes around EOF like this,
#!bin/bash
sudo su - postgres <<"EOF"
IP="XYZ"
echo "$IP"
EOF
It works. Can someone please explain this? According to what I read in man the behaviour should be opposite.
The shell evaluates the unquoted here document and performs variable interpolation before passing it to the command (in your case, sudo). Because IP is not a defined variable in the parent shell, it gets expanded to an empty string.
With quotes, you prevent variable interpolation by the parent shell, and so the shell run by sudo sees and expands the variable.