This question already has answers here:
How to assign a heredoc value to a variable in Bash?
(13 answers)
Closed 2 years ago.
this is my file perl5lib.sh:
export PERL5LIB=`cat |tr '\n' ':'`<<EOF
/home/vul/repository/projects/implatform/daemon/trunk/lib/
/home/vul/repository/projects/platformlib/tool/trunk/cpan_lib
/home/projects/libtrololo
I want to start file as
. perl5lib.sh
to populate PERL5LIB variable, but it hangs. What is wrong?
my goal is to left folder names at the end of file, so I can add new simply:
echo dirname >> myscript
I have tried and export PERL5LIB=$(echo blabla) and cat<<EOF both work separately, but not together.
=================== THE SOLUTION ============================
function do the trick!
function fun
{
export PERL5LIB=`cat|tr '\n' ':'`
}
fun<<EOF
/dir1/lib
/dir2/lib
/dir3/lib
EOF
cat is useless here. Provide EOF inside the subshell:
#! /bin/bash
export PERL5LIB=$(tr '\n' ':'<<EOF
/home/vul/repository/projects/implatform/daemon/trunk/lib/
/home/vul/repository/projects/platformlib/tool/trunk/cpan_lib
/home/projects/libtrololo
EOF
)
What you call "EOF" can be googled as Here Document. Here Document can only be used to feed the standard input of a command.
The below example does what you want without spawning child processes
#!/bin/bash
multilineconcat=
while read line; do
#echo $line
multilineconcat+=$line
multilineconcat+=":"
done << EOF
path1
path2
EOF
echo $multilineconcat
Isn't it waiting for the EOF in your heredoc ?
I'd expect you to say
$ mycommand <<EOF
input1
input2
...
EOF
Note that EOF isn't a magic keyword. It's just a marker, and could be anything (ABC etc.). It indicates the end-of-file, but people simply write EOF as convention.
Related
this is a bug I have found nothing about after a relentless search
I'm trying to run a bootstrap file in an EC2 instance, part of an EMR cluster v6.4.0. As the bootstrap action takes longer than 5 minutes, we execute it as a subprocess of the form
#!/bin/bash
var="var"
cat << EOF > ~/bootstrap.sh
intra="intra"
echo $var
echo $intra
EOF
/bin/bash ~/bootstrap.sh
exit 0
But the var "intra" is never set, and the bootstrap action returns the error line n: intra: unbound variable
If you execute that script the "intra" var is not printed.
Why can't I assign variables in a subprocess? Thank you!
When using that type of heredoc (<<WORD), you must escape literal $ characters using \$. Same goes for the backtick character (`):
#!/bin/bash
var="var"
cat << EOF > ~/bootstrap.sh
intra="intra"
echo $var
echo \$intra
EOF
/bin/bash ~/bootstrap.sh
exit 0
Another way of generating an equivalent bootstrap script is to use the literal heredoc form <<'WORD':
#!/bin/bash
var="var"
# This line will be inserted as-is without variable and subshell expansion:
cat << 'EOF1' > ~/bootstrap.sh
intra="intra"
EOF1
# We will allow this ONE line to expand.
cat << EOF2 >> ~/bootstrap.sh
echo $var
EOF2
# Back to literal insertions, no escaping necessary.
cat << 'EOF3' >> ~/bootstrap.sh
echo $intra
EOF3
/bin/bash ~/bootstrap.sh
exit 0
Inspecting the contents of ~/bootstrap.sh is a good place to start debugging.
It is possible to extract any payload from if you a shell script file with the following technique (see this):
#!/bin/sh
tail -n +4 > package.tgz
exec tar zxvf package.tgz
# payload comes here...
This needs a file so tail can seek the file to the right place.
In my particular situation, to automate things further, I'm using the | sh - pattern, but it breaks payload extraction, because pipes are not seekable.
I also tried to embed binary payload into a heredoc so I could make something like:
cat >package.tgz <<END
# payload comes here
END
tar zxvf package.tgz
But it makes shells (both bash and NetBSD's /bin/sh) confused and it just doesn't work.
I could use uuencode or base64 within the heredoc but I just wanted to know if there is some shell wizardry that could be used to receive both the script and binary data from stdin and extract the binary data out of the the data received from stdin.
Edit:
When I mean the shell gets confused, I mean it can just ignore null bytes or have undefined behaviour, even within the heredoc. Try:
cat > /tmp/out <<EOF
$(echo 410041 | xxd -p -r)
EOF
xxd -p /tmp/out
Bash complains: line 2: warning: command substitution: ignored null byte in input.
If I literally embed hex bytes 410041 into the shell script and use quoted heredoc, the result is different, but bash just drops null bytes.
echo '#!/bin/sh' > foo.sh
echo "cat > /tmp/out <<'EOF'" >> foo.sh
echo 410041 | xxd -p -r >> foo.sh
echo >> foo.sh
echo EOF >> foo.sh
echo 'xxd -p /tmp/out' >> foo.sh
bash /tmp/foo.sh
41410a
bash (and other shells) tend to "think" in C-strings, which are null-terminated, and hence cannot contain nulls (that's what indicates the end of the string). To produce nulls, you pretty much have to run some program/command that takes some safely-encoded content and produces nulls, and have its output sent directly to a file or pipe without the shell looking at it in between.
The simplest way to do this will be to encode the file with something like base64, then pipe the output from base64 -D. Something like this:
base64 -D <<'EOF' | tar xzv
H4sIAOzIHV8AA+y9DVxVVbowvs/hgAc8sY+Jhvl1VCoJBVQsETVgOIgViin2pSkq
....
EOF
If you don't want to use base64, another option would be to use bash's printf builtin to print null-containing or otherwise weird output to a pipe. It might look something like this:
LC_ALL=C
printf '\037\213\010\000\354\310\035_\000\003\354\275\015\\UU....' | tar xzv
In the above, example, I converted everything that wasn't printable ASCII to \octal codes. It should actually be ok to include almost everything as literal characters, except null, single-quote (cannot be included in a single-quoted string, probably simplest to octal-encode), backslash (just double it), and percent-sign (also double it). I don't think it'll be a problem, but it might be safest to set LC_ALL=C first, so it doesn't freak out about non-valid-UTF-8 in input strings.
Here's a quick & dirty C program to do the encoding. Note that it sends output to stdout, and it may contain junk that'll mess up your Terminal; so be sure to direct output somewhere.
#include <stdio.h>
#include <stdlib.h>
int main( int argc, char *argv[] ) {
int ch;
FILE *fp;
if ( argc != 2 ) {
fprintf(stderr, "Usage: %s infile\n", argv[0]);
return 1;
}
fp = fopen(argv[1], "r");
if (fp == NULL) {
fprintf(stderr, "Error opening %s", argv[1]);
return 1;
}
printf("#!/bin/bash\nLC_ALL=C\nprintf '");
while((ch = fgetc(fp)) != EOF) {
switch(ch) {
case '\000':
printf("\\000");
break;
case '\047':
printf("\\047");
break;
case '%':
case '\\':
printf("%c%c", ch, ch);
break;
default:
printf("%c", ch);
}
}
fclose(fp);
printf("' | tar xzv\n");
return 0;
}
if there is some shell wizardry that could be used to receive both the script and binary data from stdin and extract the binary data out of the the data received from stdin.
Having such script:
cat <<'EOF' >script.sh
#!/bin/sh
hostname
echo "What is you age?"
if ! IFS= read -r ans; then
echo "Read failed!"
else
echo "You are $ans years old."
fi
xxd -p
EOF
You can pipe to remote ssh shell via process substitution with here document followed by any data you want:
{
echo 123
echo "This is the input"
echo 001122 | xxd -r -p
} | {
u=$(uuidgen)
# Remove shell is started with a process subtitution
# terminated with a unique mark
echo "bash <(cat <<'$u'"
cat script.sh
# Note - script.sh may not read all input
# which will then executed as commands
# read it here and make sure nothing leaks
echo 'cat >/dev/null'
echo "$u"
echo ")"
# the process substitution is followed by input
# note that because the upper bash "eats" all input
# it will not execute.
cat
} | ssh host
sample execution:
host
What is you age?
You are 123 years old.
546869732069732074686520696e7075740a001122
u
No as to:
My real problem is: I'm dinamically generating shell script for remote configuration management, so I do createsh | ssh host. I'm embedding binary data to the shell script so it can be extracted to the remote host.
While you could separate two streams with a separator:
u=$(uuidgen); cat script.sh; echo; echo $u; cat binarydata.txt | ssh host bash -c 'sed "/$1/{d;q}" >script.sh; cat > binarydata.txt' _ "$u"
that is just reinventing the wheel - it already exists and is called tar:
tar -cf - script.sh binarydata.txt | ssh host bash -c 'cd /tmpdir; <unpack tar>; ./script.sh binarydata.txt; rm /tmpdir'
I have two scripts. Script A includes script B and calls a function in script B.
The setup looks like this:
Test file - ~/file.txt
one==1.0.0
two==2.0.0
three==3.0.0
four==4.0.0
Script A - ~/script_a.sh
#!/bin/bash
source script_b.sh
func_one
Script B - ~/script_b.sh
#!/bin/bash
# Note: don't forget to change the spaces to tabs else heredoc won't work
my_user=$USER
func_two() {
# Here, I need run everything in the heredoc as user $my_user
sudo su - $my_user -s /bin/bash <<- EOF
while read -r line || [[ -n "$line" ]];
do
# **This is the problem line**
# I can confirm that all the lines are being
# read but echo displays nothing
echo "$line"
# The line below will be printed 4 times as there are 4 lines in the file of interest
echo "Test"
done < "/home/$my_user/file.txt"
EOF
}
func_one() {
func_two
}
To run
cd ~
bash script_a.sh
Question: Why is the line echo "$line" not producing any output?
The problem is that bash is substituting $line with its value (nothing) before it gets passed to su. Escaping the dollar sign should fix it. So $line should be changed to \$line in both places in script_b.sh.
This question already has answers here:
How can I execute a command stored in a variable?
(4 answers)
Closed 2 years ago.
I am writing a script, and one part of it is not working as I would expect.
I have broken out this part in a simple example for simplicity:
echo 'echo "" > tmp' | while read cmd; do $cmd ; done
Here I would expect the full command, "echo "" > tmp" to be executed by $cmd.
But this happens:
"" > tmp
The command executed echoes out "" > tmp literally. Instead of echoing out "" and redirecting it to the file tmp. Obviously something is wrong when storing the command in $cmd and then later trying to execute it.
The result is the same even if I simplify it further:
cmd="echo "" > tmp"
$cmd
> tmp
I have tried to experiment with different usages of '' and "", but not solved it yet.
Use eval to execute the command stored in the variable:
echo 'echo "" > tmp' | while read cmd; do eval "$cmd" ; done
The value of cmd will be echo "" > tmp. Then when Bash resolves the parameter substitution as a command, the part "" > tmp will be the string arguments of echo, not be recognized as >(redirection). So it will just output the arguments part.
The same as: $(echo 'echo "" > tmp')
Change
do $cmd ;
to
do eval "$cmd" ;
I am trying to run from a shell script a C++ program that print some outputs (using std::cout), and I would like to see them in the console while the program is running.
I tried some things like this :
RES=`./program`
RES=$(./program)
But all I can do is to only display the result at the end : echo $RES...
How to display the outputs in run-time in the console, AND in the variable RES ?
TTY=$(tty);
SAVED_OUTPUT=$(echo "my dummy c++ program" | tee ${TTY});
echo ${SAVED_OUTPUT};
prints
my dummy c++ program
my dummy c++ program
First we save off the name of the current terminal (because tty doesn't work in a pipeline).
TTY=$(tty)
Then we "T" the output (a letter T looks like one stream in at the bottom, 2 out at the top, and comes from the same "plumbing" metaphor as "pipe"), which copies it to the filename given; in this case the "file" is really a special device representing our terminal.
echo "my dummy c++ program" | tee ${TTY}
RES=( $(./program) )
echo ${RES[#]}
you can try in this way
You can use a temporary file
./program | tee temp
RES=$(< temp)
rm temp
You can generate a temporary file with unique name by using mktemp.
res=$(sed -n 'p;' <<< $(printf '%s\n' '*' $'hello\t\tworld'; sleep 5; echo "post-streaming content")&;wait)
echo $res
#output
*
hello world
post-streaming content
[sky#kvm35066 tmp]$ cat test.sh
#!/bin/bash
echo $BASH_VERSION
res=$(printf '%s\n' '*' $'hello\t\tworld'; sleep 5; echo "post-streaming content")
echo "$res"
[sky#kvm35066 tmp]$ bash test.sh
4.1.2(1)-release
*
hello world
post-streaming content
i think the result is correct, and it is what you want