How to expand shell variables in a text file? - bash

Consider a ASCII text file (lets say it contains code of a non-shell scripting language):
Text_File.msh:
spool on to '$LOG_FILE_PATH/logfile.log';
login 'username' 'password';
....
Now if this were a shell script I could run it as $ sh Text_File.msh and the shell would automatically expand the variables.
What I want to do is have shell expand these variables and then create a new file as Text_File_expanded.msh as follows:
Text_File_expanded.msh:
spool on to '/expanded/path/of/the/log/file/../logfile.log';
login 'username' 'password';
....
Consider:
$ a=123
$ echo "$a"
123
So technically this should do the trick:
$ echo "`cat Text_File.msh`" > Text_File_expanded.msh
...but it doesn't work as expected and the output-file while is identical to the source.
So I am unsure how to achieve this.. My goal is make it easier to maintain the directory paths embedded within my non-shell scripts. These scripts cannot contain any UNIX code as it is not compiled by the UNIX shell.

This question has been asked in another thread, and this is the best answer IMO:
export LOG_FILE_PATH=/expanded/path/of/the/log/file/../logfile.log
cat Text_File.msh | envsubst > Text_File_expanded.msh
if on Mac, install gettext first: brew install gettext
see:
Forcing bash to expand variables in a string loaded from a file

This solution is not elegant, but it works. Create a script call shell_expansion.sh:
echo 'cat <<END_OF_TEXT' > temp.sh
cat "$1" >> temp.sh
echo 'END_OF_TEXT' >> temp.sh
bash temp.sh >> "$2"
rm temp.sh
You can then invoke this script as followed:
bash shell_expansion.sh Text_File.msh Text_File_expanded.msh

If you want it in one line (I'm not a bash expert so there may be caveats to this but it works everywhere I've tried it):
when test.txt contains
${line1}
${line2}
then:
>line1=fark
>line2=fork
>value=$(eval "echo \"$(cat test.txt)\"")
>echo "$value"
line1 says fark
line2 says fork
Obviously if you just want to print it you can take out the extra value=$() and echo "$value".

If a Perl solution is ok for you:
Sample file:
$ cat file.sh
spool on to '$HOME/logfile.log';
login 'username' 'password';
Solution:
$ perl -pe 's/\$(\w+)/$ENV{$1}/g' file.sh
spool on to '/home/user/logfile.log';
login 'username' 'password';

One limitation of the above answers is that they both require the variables to be exported to the environment. Here's what i came up with that would allow the variables to be local to the current shell script:
#!/bin/sh
FOO=bar;
FILE=`mktemp`; # Let the shell create a temporary file
trap 'rm -f $FILE' 0 1 2 3 15; # Clean up the temporary file
(
echo 'cat <<END_OF_TEXT'
cat "$#"
echo 'END_OF_TEXT'
) > $FILE
. $FILE
The above example allows the variable $FOO to be substituted in the files named on the command line. I'm sure it can be improved, but this works for me so far.
Thanks to both previous answers for their ideas!

If the variables you want to translate are known and limited in number, you can always do the translation yourself:
sed "s/\$LOG_FILE_PATH/$LOG_FILE_PATH/g" input > output
And also assuming the variable itself is already known

This solution allows you to keep the same formatting in the ouput file
Copy and paste the following lines in your script
cat $1 | while read line
do
eval $line
echo $line
eval echo $line
done | uniq | grep -v '\$'
this will read the file passed as argument line by line, and then process to try and print each line twice:
- once without substitution
- once with substitution of the variables.
then remove the duplicate lines
then remove the lines containing visible variables ($)

Yes eval should be used carefully, but it provided me this simple oneliner for my problem. Below is an example using your filename:
eval "echo \"$(<Text_File.msh)\""
I use printf instead of echo for my own purposes, but that should do the trick. Thank you abyss.7 providing the link that solve my problem. Hope it helps.

Create an ascii file test.txt with the following content:
Try to replace this ${myTestVariable1}
bla bla
....
Now create a file “sub.sed” containing variable names, eg
's,${myTestVariable1},'"${myTestVariable1}"',g;
s,${myTestVariable2},'"${myTestVariable2}"',g;
s,${myTestVariable3},'"${myTestVariable3}"',g;
s,${myTestVariable4},'"${myTestVariable4}"',g'
Open a terminal move to the folder containing test.txt and sub.sed.
Define the value of the varible to be replaced
myTestVariable1=SomeNewText
Now call sed to replace that variable
sed "$(eval echo $(cat sub.sed))" test.txt > test2.txt
The output will be
$cat test2.txt
Try to replace this SomeNewText
bla bla
....

#logfiles.list:
$EAMSROOT/var/log/LinuxOSAgent.log
$EAMSROOT/var/log/PanacesServer.log
$EAMSROOT/var/log/PanacesStrutsGUI.log
#My Program:
cat logfiles.list | while read line
do
eval Eline=$line
echo $Eline
done

Related

Shell script for replacing characters?

I'm trying to write a shell script that takes in a file(ex. file_1_2.txt) and replaces any "_" with "."(ex. file.1.2.txt). This is what I have but its giving me a blank output when I run it.
read $var
x= `echo $var | sed 's/\./_/g'`
echo $x
I'm trying to store the changed filename in the variable "x" and then output x to the console.
I am calling this script by writing
./script2.sh < file_1_2.txt
There is two problems. First, your code has some bugs:
read var
x=`echo $var | sed 's/_/\./g'`
echo $x
will work. You had an extra $ in read var, a space too much (as mentioned before) and you mixed up the replacement pattern in sed (it was doing the reverse of what you wanted).
Also if you want to replace the _ by . in the filename you should do
echo "file_1_2.txt" | ./script2.sh
If you use < this will read the content of `file_1_2.txt" into your script.
Another solution, with bash only:
$ x=file_1_2.txt; echo "${x//_/.}"
file.1.2.txt
(See “Parameter expansion” section in bash manual page for details)
And you can also do this with rename:
$ touch file_1_2.txt
$ ls file*
file_1_2.txt
$ rename 'y/_/\./' file_1_2.txt
$ ls file*
file.1.2.txt
Threre is not need for sed as bash supports variable replacement:
$ cat ./script2
#!/bin/bash
ofile=$1
nfile=${ofile//_/./}
echo mv "$ofile" "$nfile"
$ ./script2 file_1_2.txt
mv "file_1_2.txt" "file.1.2.txt"
Then just remove echo if you are satisfied with the result.

Create temporary file and redirect output to it in one command

I designed a custom script to grep a concatenated list of .bash_history backup files. In my script, I am creating a temporary file with mktemp and saving it to a variable temp. Next, I am redirecting output to that file using the cat command.
Is there a means to create a temporary file (using mktemp), redirect output to it, then store it in a variable in one command, while preserving newline characters?
The below snippet of code works just fine, but I have a feeling there is a more terse and canonical way to achieve this in one line – maybe using process substitution or something of the like.
# Concatenate all .bash_history files into a temporary file `temp`.
temp="$(mktemp)"
cat "$HOME/.bash_history."* > $temp
trap 'rm -f $temp' 0
# Set `HISTFILE` shell variable to the `temp` file.
HISTFILE="$temp"
keyword="$1"
# Search for `keyword` using the `history` command
if [[ "$keyword" ]]; then
# Enable history
set -o history
history | grep "$keyword"
# Disable history
set +o history
else
echo -e "usage: search <keyword>"
exit 0
fi
If you're comfortable with the side effect of making the assignment conditional on tempfile not previously having a nonempty value, this is straightforward via the ${var:=value} expansion:
cat "$HOME/.bash_history" >"${tempfile:=$(mktemp)}"
cat myfile.txt | f=`mktemp` && cat > "${f}"
I guess there is more than one way to do it. I found following to be working for me:
cat myfile.txt > $(echo "$(mktemp)")
Also don't forget about tee:
cat myfile.txt | tee "$(mktemp)" > /dev/null

how to read data from a file in shell script

I have two shell script files test1.sh and test2.sh . I have another file called translogs.txt.
Now I need to copy the values of two variables in test1.sh to translog.txt and the same variables need to be copied to the corresponding values in test2.sh.
test1.sh
#!/bin/sh
ONE="000012"
TIME="2013-02-19 15:31:06"
echo -e "$ONE\n$TIME">translog.txt;
translog.txt
ONE="000012"
TIME="2013-02-19 15:31:06"
But here in test2.sh, I want the same value as in translog.txt to the corresponding variable
test2.sh
#!/bin/sh
ONE="000012"
TIME="2013-02-19 15:31:06"
1 Diry solution
$> cat translog.txt
ONE="000012"
TIME="2013-02-19 15:31:06"
With perl regular expression grep could match these value using lookbehind operator.
$> grep --only-matching --perl-regex "(?<=ONE\=).*" translog.txt
"000012"
And for TIME:
$> grep --only-matching --perl-regex "(?<=TIME\=).*" translog.txt
"2013-02-19 15:31:06"
So from withing the test2.sh script you can use it like this:
#!/bin/bash
ONE=`grep --only-matching --perl-regex "(?<=ONE\=).*" translog.txt`
TIME=`grep --only-matching --perl-regex "(?<=TIME\=).*" translog.txt`
2 Command line solution
Another solution pointed out in one of the links below would be to use:
the source (a.k.a. .) command to load all of the variables in the file into the current shell:
$ source translog.txt
Now you have access to the values of the variables defined inside the file:
$ echo $TIME
"2013-02-19 15:31:06"
3 Easiest solution
Another approach was mentioned by #user2086768. Put these lines to `test2.sh:
#!/bin/bash
eval $(cat translog.txt)
And as a result you would have assigned the two variables within the test2.sh script:
ONE="000012"
TIME="2013-02-19 15:31:06"
you can easily check that adding:
echo $ONE
echo $TIME
Check also these links:
how to get value of variable config in bash?
Here's a more general overview: Loading data into bash variables
As translog.txt is valid bash code, you could do:
source translog.txt
in test2.sh, ONE and TWO would be available in test2.sh.
A word of warning, that does open you up to shell script injection attacks if the values for ONE and TWO were to come from an untrusted source.
If your translog.txt is as you say, then this will work
#!/bin/sh
while read aa
do
eval "$aa"
done < translog.txt
eval should work for you.
Try to use this version of test2.sh:
test2.sh
#!/bin/bash
eval $(cat translog.txt)
echo $ONE
echo $TIME
This outputs:
000012
2013-02-19 15:31:06

How can one store a variable in a file using bash?

I can redirect the output and then cat the file and grep/awk the variable, but I would like to use this file for multiple variables.
So If it was one variable say STATUS then i could do some thing like
echo "STATUS $STATUS" >> variable.file
#later perhaps in a remote shell where varible.file was copied
NEW_VAR=`cat variable.file | awk print '{$2}'`
I guess some inline editing with sed would help. The smaller the code the better.
One common way of storing variables in a file is to just store NAME=value lines in the file, and then just source that in to the shell you want to pick up the variables.
echo 'STATUS="'"$STATUS"'"' >> variable.file
# later
. variable.file
In Bash, you can also use source instead of ., though this may not be portable to other shells. Note carefully the exact sequence of quotes necessary to get the correct double quotes printed out in the file.
If you want to put multiple variables at once into the file, you could do the following. Apologies for the quoting contortions that this takes to do properly and portably; if you restrict yourself to Bash, you can use $"" to make the quoting a little simpler:
for var in STATUS FOO BAR
do
echo "$var="'"'"$(eval echo '$'"$var")"'"'
done >> variable.file
The declare builtin is useful here
for var in STATUS FOO BAR; do
declare -p $var | cut -d ' ' -f 3- >> filename
done
As Brian says, later you can source filename
declare is great because it handles quoting for you:
$ FOO='"I'"'"'m here," she said.'
$ declare -p FOO
declare -- FOO="\"I'm here,\" she said."
$ declare -p FOO | cut -d " " -f 3-
FOO="\"I'm here,\" she said."

How can I use bash to parse out only a section of a variable with different delimiters?

I have a loop in a bash file to show me all of the files in a directory, each as its own variable. I need to take that variable (filename) and parse out only a section of it.
Example:
92378478234978ehbWHATIWANT#98712398712398723
Now, assuming "ehb" and the pound symbol never change, how can I just capture WHATIWANT into its own variable?
So far I have:
#!/bin/bash
for FILENAME in `dir -d *` ; do
done
You can use sed to edit out the parts you don't want.
want=$(echo "$FILENAME" | sed -e 's/.*ehb\(.*\)#.*/\1/')
Or you can use Bash's parameter expansion to strip out the tail and head.
want=${FILENAME%#*}; want=${want#*ehb}
One possibility:
for i in '92378478234978ehbWHATIWANT#98712398712398723' ; do
j=$(echo $i | sed -e 's/^.*ehb//' -e 's/#.*$//')
echo $j
done
produces:
WHATIWANT
using only the bash shell, no need external tools
$ string=92378478234978ehbWHATIWANT#98712398712398723
$ echo ${string#*ehb}
WHATIWANT#98712398712398723
$ string=${string#*ehb}
$ echo ${string%#*}
WHATIWANT

Resources