Using process substitution with HereDocument that is stored in a variable - bash

I know I can use bash's process substitution feature to specify a file parameter via process substitution and then use a here document to specify the data, like so:
foo --config <(cat <<EOF
# contents of config file
...
EOF
)
When I need the same here document at two different locations in the same script, then it would be more useful to store the here document in a variable.
How can I do something like this:
read -r -d '' MY_CONFIG <<EOF
# contents of config file
...
EOF
Then how can I call foo to pass the contents of $MY_CONFIG to the --config parameter?

Additionally, you can use a herestring inside of process substitution:
MY_CONFIG="hello world"
md5sum <( <<< "$MY_CONFIG" )

The solution occurred to my while writing this question. Rather then using <(cat ...), we can use <(echo ...) to write something from the process substitution. So we can simply use:
foo --config <(echo "$MY_CONFIG")
A simple example to test this would be this:
MY_CONFIG="hello world"
less -f <(echo "$MY_CONFIG")
This should open less and show hello world in the buffer.

I consider the solution you found yourself simpler, but for sake of completeness, here is how you could do it with here-docs.
To store a here-doc in a variable, you can use (> is the secondary prompt string):
$ var=$(cat <<EOF
> line1
> line2
> EOF
> )
$ declare -p var
declare -- var="line1
line2"
Then, to use a here-doc with the contents of that variable:
$ cat <<EOF
> $var
> EOF
line1
line2
I.e., for your case, something like this:
$ foo --config <(cat <<EOF
> $var
> EOF
> )
As I said, it's clearly simpler to just use <(echo "$var") or <(printf '%s\n' "$var") instead.

Related

While-loop over lines from variable in bash

Assume a file file with multiple lines.
$ cat file
foo
bar
baz
Assume further that I wish to loop through each line with a while-loop.
$ while IFS= read -r line; do
$ echo $line
$ # do stuff
$ done < file
foo
bar
baz
Finally, please assume that I wish to pass lines stored in a variable rather than lines stored in a file. How can I loop through lines that are saved as a variable without receiving the below error?
$ MY_VAR=$(cat file)
$ while IFS= read -r line; do
$ echo $line
$ # do stuff
$ done < $(echo "$MY_VAR")
bash: $(echo "$MY_VAR"): ambiguous redirect
You have several options:
A herestring (note that this is a non-POSIX extension): done <<<"$MY_VAR"
A heredoc (POSIX-compliant, will work with /bin/sh):
done <<EOF
$MY_VAR
EOF
A process substitution (also a non-POSIX extension, but using printf rather than echo makes it more predictable across shells that support it; see the APPLICATION USAGE note in the POSIX spec for echo): done < <(printf '%s\n' "$MY_VAR")
Note that the first two options will (in bash) create a temporary file on disk with the variable's contents, whereas the last one uses a FIFO.
< needs to be followed by a filename. You can use a here-string:
done <<< "$MY_VAR"
or process substitution:
done < <(echo "$MY_VAR")

How to pass a variable as a temporary file to a command in bash

There exists a command foo which expects two arguments which are filenames and which prints some stuff on stdout.
I have a Bash script with two variables a and b holding two strings.
I wish to pass to foo two filenames where the contents of those files are a and b. I then want to store the stdout as a new variable c.
Following ad hoc Googling, the script would perhaps look something like:
a=...;
b=...;
c=`foo <($a) <($b)`;
What should it look like?
a=...
b=...
c=$(foo <(echo "$a") <(echo "$b"))
echo "$c"
Try using echo -e
a=...
b=...
c=$(foo <(echo -e "$a") <(echo -e "$b"))
echo -e "$c"

How to add literal text to the Unix 'cat' command

I'm trying to cat some files together, while at the same time adding some text between files. I'm a Unix newbie and I don't have the hang of the syntax.
Here's my failed attempt:
cat echo "# Final version (reflecting my edits)\n\n" final.md echo "\n\n# The changes I made\n\n" edit.md echo "\n\n#Your original version\n\n" original.md > combined.md
How do I fix this? Should I be using pipes or something?
A process substitution seems to work:
$ cat <(echo 'FOO') foo.txt <(echo 'BAR') bar.txt
FOO
foo
BAR
bar
You can also use command substitution inside a here-document.
$ cat <<EOF
FOO
$(< foo.txt)
BAR
$(< bar.txt)
EOF
Use a command group to merge the output into one stream:
{
echo -e "# Final version (reflecting my edits)\n\n"
cat final.md
echo -e "\n\n# The changes I made\n\n"
cat edit.md
echo -e "\n\n#Your original version\n\n"
cat original.md
} > combined.md
There are tricks you can play with process substitution and command substitution (see Lev Levitsky's answer) to do it all with one command (instead of the separate cat processes used here), but this should be efficient enough with so few files.
If I understand you, it should be something like:
echo "# Final version (reflecting my edits)\n\n" >> combined.md
cat final.md >> combined.md
echo "\n\n# The changes I made\n\n" >> combined.md
cat edit.md >> combined.md
And so on.

How to expand shell variables in a text file?

Consider a ASCII text file (lets say it contains code of a non-shell scripting language):
Text_File.msh:
spool on to '$LOG_FILE_PATH/logfile.log';
login 'username' 'password';
....
Now if this were a shell script I could run it as $ sh Text_File.msh and the shell would automatically expand the variables.
What I want to do is have shell expand these variables and then create a new file as Text_File_expanded.msh as follows:
Text_File_expanded.msh:
spool on to '/expanded/path/of/the/log/file/../logfile.log';
login 'username' 'password';
....
Consider:
$ a=123
$ echo "$a"
123
So technically this should do the trick:
$ echo "`cat Text_File.msh`" > Text_File_expanded.msh
...but it doesn't work as expected and the output-file while is identical to the source.
So I am unsure how to achieve this.. My goal is make it easier to maintain the directory paths embedded within my non-shell scripts. These scripts cannot contain any UNIX code as it is not compiled by the UNIX shell.
This question has been asked in another thread, and this is the best answer IMO:
export LOG_FILE_PATH=/expanded/path/of/the/log/file/../logfile.log
cat Text_File.msh | envsubst > Text_File_expanded.msh
if on Mac, install gettext first: brew install gettext
see:
Forcing bash to expand variables in a string loaded from a file
This solution is not elegant, but it works. Create a script call shell_expansion.sh:
echo 'cat <<END_OF_TEXT' > temp.sh
cat "$1" >> temp.sh
echo 'END_OF_TEXT' >> temp.sh
bash temp.sh >> "$2"
rm temp.sh
You can then invoke this script as followed:
bash shell_expansion.sh Text_File.msh Text_File_expanded.msh
If you want it in one line (I'm not a bash expert so there may be caveats to this but it works everywhere I've tried it):
when test.txt contains
${line1}
${line2}
then:
>line1=fark
>line2=fork
>value=$(eval "echo \"$(cat test.txt)\"")
>echo "$value"
line1 says fark
line2 says fork
Obviously if you just want to print it you can take out the extra value=$() and echo "$value".
If a Perl solution is ok for you:
Sample file:
$ cat file.sh
spool on to '$HOME/logfile.log';
login 'username' 'password';
Solution:
$ perl -pe 's/\$(\w+)/$ENV{$1}/g' file.sh
spool on to '/home/user/logfile.log';
login 'username' 'password';
One limitation of the above answers is that they both require the variables to be exported to the environment. Here's what i came up with that would allow the variables to be local to the current shell script:
#!/bin/sh
FOO=bar;
FILE=`mktemp`; # Let the shell create a temporary file
trap 'rm -f $FILE' 0 1 2 3 15; # Clean up the temporary file
(
echo 'cat <<END_OF_TEXT'
cat "$#"
echo 'END_OF_TEXT'
) > $FILE
. $FILE
The above example allows the variable $FOO to be substituted in the files named on the command line. I'm sure it can be improved, but this works for me so far.
Thanks to both previous answers for their ideas!
If the variables you want to translate are known and limited in number, you can always do the translation yourself:
sed "s/\$LOG_FILE_PATH/$LOG_FILE_PATH/g" input > output
And also assuming the variable itself is already known
This solution allows you to keep the same formatting in the ouput file
Copy and paste the following lines in your script
cat $1 | while read line
do
eval $line
echo $line
eval echo $line
done | uniq | grep -v '\$'
this will read the file passed as argument line by line, and then process to try and print each line twice:
- once without substitution
- once with substitution of the variables.
then remove the duplicate lines
then remove the lines containing visible variables ($)
Yes eval should be used carefully, but it provided me this simple oneliner for my problem. Below is an example using your filename:
eval "echo \"$(<Text_File.msh)\""
I use printf instead of echo for my own purposes, but that should do the trick. Thank you abyss.7 providing the link that solve my problem. Hope it helps.
Create an ascii file test.txt with the following content:
Try to replace this ${myTestVariable1}
bla bla
....
Now create a file “sub.sed” containing variable names, eg
's,${myTestVariable1},'"${myTestVariable1}"',g;
s,${myTestVariable2},'"${myTestVariable2}"',g;
s,${myTestVariable3},'"${myTestVariable3}"',g;
s,${myTestVariable4},'"${myTestVariable4}"',g'
Open a terminal move to the folder containing test.txt and sub.sed.
Define the value of the varible to be replaced
myTestVariable1=SomeNewText
Now call sed to replace that variable
sed "$(eval echo $(cat sub.sed))" test.txt > test2.txt
The output will be
$cat test2.txt
Try to replace this SomeNewText
bla bla
....
#logfiles.list:
$EAMSROOT/var/log/LinuxOSAgent.log
$EAMSROOT/var/log/PanacesServer.log
$EAMSROOT/var/log/PanacesStrutsGUI.log
#My Program:
cat logfiles.list | while read line
do
eval Eline=$line
echo $Eline
done

shell script problem: does not work on the terminal, but works in a script

I was playing with shell scripting, when a strange thing happened. I need someone to explain it.
I have a file 'infile', contents:
line one
line2
third line
last
a test script test.sh, contents:
read var1
echo $var1
i executed:
cat infile | ./test.sh
output was
line one
Then I did:
cat infile | read var1
echo $var1
Result: a blank line.
I even tried
cat infile | read var1; echo $var1;
same result.
why does this happen?
The pipe causes the command after it to run in a subshell, which means that environment variables won't be propagated to the main shell. Use redirection or a herestring to get around this:
read var1 < infile
Or try this:
cat file | ( read var1; echo $var1; )

Resources