Use substituted string as a command in shell script - shell

Given the variables listed below:
echo ${userupper}_PYTHON
# YTU_PYTHON
echo $YTU_PYTHON
# /home/ytu/anaconda3/bin/python
echo $path
# foo.py
Now I'd like to execute /home/ytu/anaconda3/bin/python foo.py with userupper and path. I tried $(${userupper}_PYTHON) $path but it ends up with error messages including:
YTU_PYTHON: not found
foo.py: not found
It seems like it takes $(${userupper}_PYTHON) as bare YTU_PYTHON rather than expected $YTU_PYTHON. How should I do to make it right?
Edits:
The suggested duplication should have solved my problem. However for some unknown reasons it's not working.
#!/usr/bin/env bash
for user in ytu
do
. /home/${user}/.profile
userupper=$(echo ${user} | awk '{print toupper($0)}')
userpython=${userupper}_PYTHON
cd /home/${user}/H2-ML/crons
for path in $(ls | grep ^${user}_.*_monthly_report.py$)
do
echo ${userpython}
echo $path
echo $YTU_PYTHON
echo ${!userpython}
done
done
The code chunk above returns:
YTU_PYTHON
ytu_clinic249_monthly_report.py
/home/ytu/anaconda3/bin/python
send_monthly_reports.sh: 14: send_monthly_reports.sh: Bad substitution
, which makes me so confused.

Try this:
command=${userupper}_PYTHON
echo ${!command} $path # ommit echo if you want to execute command
in your case it echos:
/home/ytu/anaconda3/bin/python foo.py
Here is link that might be useful if you want to write bash scripts. You should not parse ls output in the nutshell.
You also create unnecessary new process, both lines do the same:
$(echo ${user} | awk '{print toupper($0)}')
${user^^} #paramenter expansion

Related

String expansion - escaped quoted variable to value

To get started, here's the script I'm running to get the offending string:
# sed finds all sourced file paths from inputted file.
#
# while reads each match output from sed to $SOURCEFILE variable.
# Each should be a file path, or a variable that represents a file path.
# Any variables found should be expanded to the full path.
#
# echo and calls are used for demonstractive purposes only
# I intend to do something else with the path once it's expanded.
PATH_SOME_SCRIPT="/path/to/bash/script"
while read -r SOURCEFILE; do
echo "$SOURCEFILE"
"$SOURCEFILE"
$SOURCEFILE
done < <(cat $PATH_SOME_SCRIPT | sed -n -e "s/^\(source\|\.\|\$include\) //p")
You may also wish to use the following to test this out as mock data:
[ /path/to/bash/script ]
#!/bin/bash
source "$HOME/bash_file"
source "$GLOBAL_VAR_SCRIPT_PATH"
echo "No cow powers here"
For the tl;dr crew, basically the while loop spits out the following on the mock data:
"$HOME/bash_file"
bash: "$HOME/bash_file": no such file or directory
bash: "$HOME/bash_file": no such file or directory
"$GLOBAL_VAR_SCRIPT_PATH"
"$GLOBAL_VAR_SCRIPT_PATH": command not found
"$GLOBAL_VAR_SCRIPT_PATH": command not found
My question is, can you get the variable to expand correctly, e.g., print "/home//bash_file" and "/expanded/variable/path"? I should also state that although eval works I do not intend to use it because of its potential insecurities.
Protip that any variable value used in cat | sed would be available globally, including to the calling script, so it's not because the script cannot call the variable value.
FIRST SOLUTION ATTEMPT
Using anubhava's envsubst solution:
SOMEVARIABLE="/home/nick/.some_path"
while read -r SOURCEFILE; do
echo "$SOURCEFILE"
envsubst <<< "$SOURCEFILE";
done < <(echo -e "\"\$SOMEVARIABLE\"\n\"$HOME/.another_file\"")
This outputs the following:
"$SOMEVARIABLE"
""
"/home/nick/.another_file"
"/home/nick/.another_file"
Unfortunately, it does not expand the variable! Oh dear :(
SECOND SOLUTION ATTEMPT
Based upon the first attempt:
export SOMEVARIABLE="/home/nick/.some_path"
while read -r SOURCEFILE; do
echo "$SOURCEFILE"
envsubst <<< "$SOURCEFILE";
done < <(echo -e "\"\$SOMEVARIABLE\"\n\"$HOME/.another_file\"")
unset SOMEVARIABLE
which produces the results we wanted without eval and without messing with global variables (for too long anyway), hoorah!
Good runner-ups were further suggested using eval (although potentially unsafe) which can be found in this answer and here (link courtesy of anubhava's extended comments).
My question is, can you get the variable to expand correctly, e.g., print "/home//bash_file" and "/expanded/variable/path"?
Yes you can use envsubst program, that substitutes the values of environment variables:
while read -r sourceFile; do
envsubst <<< "$sourceFile"
done < <(sed -n "s/^\(source\|\.\|\$include\) //p" "$PATH_SOME_SCRIPT")
I think you are asking how to recursively expand variables in bash. Try
expanded=$(eval echo $SOURCEFILE)
inside your loop. eval runs the expanded command you give it. Since $SOURCEFILE isn't in quotes, it will be expanded to, e.g., $HOME/whatever. Then the eval will expand the $HOME before passing it to echo. echo will print the result, and expanded=$(...) will put the printed result in $expanded.

After running shell script, variables lose their value

I have done a simple shell script:
#!/bin/sh
file_name=$1
state=`cat "$file_name" | grep "port protocol" | awk '{print $4}'`
echo $state
reason=`cat "$file_name" | grep "port protocol"`
echo $reason
This outputs the values $state and $reason.
However, when I run..
echo $state
..in the console it does not output nothing. It seems like the variable loses its value. Is this the normal behaviour or should I had something to the script?
Thanks!
Assuming that you're running your script like ./script.sh or sh script.sh, then this is the expected behaviour. Child processes cannot change the environment of their parent. The fact that you're running a shell script from a shell doesn't change this rule.
What you can do is source the script instead of executing it, to set those variables in the local environment:
. script.sh
This effectively runs the lines of the script in your current shell, so the variables will be set there.
While I've got your attention, I'd recommend making the following changes to your script:
#!/bin/sh
file_name=$1
state=$(awk '/port protocol/ {print $4}' "$file_name")
echo "$state"
reason=$(grep "port protocol" "$file_name")
echo "$reason"
In short, quote your variables, avoid useless calls to cat and use pattern matching in awk rather than piping grep to it.

How to expand shell variables in a text file?

Consider a ASCII text file (lets say it contains code of a non-shell scripting language):
Text_File.msh:
spool on to '$LOG_FILE_PATH/logfile.log';
login 'username' 'password';
....
Now if this were a shell script I could run it as $ sh Text_File.msh and the shell would automatically expand the variables.
What I want to do is have shell expand these variables and then create a new file as Text_File_expanded.msh as follows:
Text_File_expanded.msh:
spool on to '/expanded/path/of/the/log/file/../logfile.log';
login 'username' 'password';
....
Consider:
$ a=123
$ echo "$a"
123
So technically this should do the trick:
$ echo "`cat Text_File.msh`" > Text_File_expanded.msh
...but it doesn't work as expected and the output-file while is identical to the source.
So I am unsure how to achieve this.. My goal is make it easier to maintain the directory paths embedded within my non-shell scripts. These scripts cannot contain any UNIX code as it is not compiled by the UNIX shell.
This question has been asked in another thread, and this is the best answer IMO:
export LOG_FILE_PATH=/expanded/path/of/the/log/file/../logfile.log
cat Text_File.msh | envsubst > Text_File_expanded.msh
if on Mac, install gettext first: brew install gettext
see:
Forcing bash to expand variables in a string loaded from a file
This solution is not elegant, but it works. Create a script call shell_expansion.sh:
echo 'cat <<END_OF_TEXT' > temp.sh
cat "$1" >> temp.sh
echo 'END_OF_TEXT' >> temp.sh
bash temp.sh >> "$2"
rm temp.sh
You can then invoke this script as followed:
bash shell_expansion.sh Text_File.msh Text_File_expanded.msh
If you want it in one line (I'm not a bash expert so there may be caveats to this but it works everywhere I've tried it):
when test.txt contains
${line1}
${line2}
then:
>line1=fark
>line2=fork
>value=$(eval "echo \"$(cat test.txt)\"")
>echo "$value"
line1 says fark
line2 says fork
Obviously if you just want to print it you can take out the extra value=$() and echo "$value".
If a Perl solution is ok for you:
Sample file:
$ cat file.sh
spool on to '$HOME/logfile.log';
login 'username' 'password';
Solution:
$ perl -pe 's/\$(\w+)/$ENV{$1}/g' file.sh
spool on to '/home/user/logfile.log';
login 'username' 'password';
One limitation of the above answers is that they both require the variables to be exported to the environment. Here's what i came up with that would allow the variables to be local to the current shell script:
#!/bin/sh
FOO=bar;
FILE=`mktemp`; # Let the shell create a temporary file
trap 'rm -f $FILE' 0 1 2 3 15; # Clean up the temporary file
(
echo 'cat <<END_OF_TEXT'
cat "$#"
echo 'END_OF_TEXT'
) > $FILE
. $FILE
The above example allows the variable $FOO to be substituted in the files named on the command line. I'm sure it can be improved, but this works for me so far.
Thanks to both previous answers for their ideas!
If the variables you want to translate are known and limited in number, you can always do the translation yourself:
sed "s/\$LOG_FILE_PATH/$LOG_FILE_PATH/g" input > output
And also assuming the variable itself is already known
This solution allows you to keep the same formatting in the ouput file
Copy and paste the following lines in your script
cat $1 | while read line
do
eval $line
echo $line
eval echo $line
done | uniq | grep -v '\$'
this will read the file passed as argument line by line, and then process to try and print each line twice:
- once without substitution
- once with substitution of the variables.
then remove the duplicate lines
then remove the lines containing visible variables ($)
Yes eval should be used carefully, but it provided me this simple oneliner for my problem. Below is an example using your filename:
eval "echo \"$(<Text_File.msh)\""
I use printf instead of echo for my own purposes, but that should do the trick. Thank you abyss.7 providing the link that solve my problem. Hope it helps.
Create an ascii file test.txt with the following content:
Try to replace this ${myTestVariable1}
bla bla
....
Now create a file “sub.sed” containing variable names, eg
's,${myTestVariable1},'"${myTestVariable1}"',g;
s,${myTestVariable2},'"${myTestVariable2}"',g;
s,${myTestVariable3},'"${myTestVariable3}"',g;
s,${myTestVariable4},'"${myTestVariable4}"',g'
Open a terminal move to the folder containing test.txt and sub.sed.
Define the value of the varible to be replaced
myTestVariable1=SomeNewText
Now call sed to replace that variable
sed "$(eval echo $(cat sub.sed))" test.txt > test2.txt
The output will be
$cat test2.txt
Try to replace this SomeNewText
bla bla
....
#logfiles.list:
$EAMSROOT/var/log/LinuxOSAgent.log
$EAMSROOT/var/log/PanacesServer.log
$EAMSROOT/var/log/PanacesStrutsGUI.log
#My Program:
cat logfiles.list | while read line
do
eval Eline=$line
echo $Eline
done

Counting file lines in shell and in a script gives different results

For a bunch of files in a directory I want to get the number of lines for each one, store it
in a variable and do additional stuff. Via shell I can do it without problems if I do
read NLINES <<< $( cat file | wc -l )
but if I do it in a script
#!/bin/bash
for i in `ls *.dat `
do
read NLINES <<< $( cat $i | wc -l )
done
I get
Syntax error: redirection unexpected
Why the difference? How could I fix it?
I bet your default shell isn't bash but something else. Leave the #!/bin/bash and replace it with #!/bin/sh, to let your script use the default shell.
I made this error the other way, when I tried to use some debian scripts on Ubuntu, where #!/bin/sh behaved differently from my assumed #!/bin/bash.

How can I reference a file for variables using Bash?

I want to call a settings file for a variable. How can I do this in Bash?
The settings file will define the variables (for example, CONFIG.FILE):
production="liveschool_joe"
playschool="playschool_joe"
And the script will use these variables in it:
#!/bin/bash
production="/REFERENCE/TO/CONFIG.FILE"
playschool="/REFERENCE/TO/CONFIG.FILE"
sudo -u wwwrun svn up /srv/www/htdocs/$production
sudo -u wwwrun svn up /srv/www/htdocs/$playschool
How can I get Bash to do something like that? Will I have to use AWK, sed, etc.?
The short answer
Use the source command.
An example using source
For example:
config.sh
#!/usr/bin/env bash
production="liveschool_joe"
playschool="playschool_joe"
echo $playschool
script.sh
#!/usr/bin/env bash
source config.sh
echo $production
Note that the output from sh ./script.sh in this example is:
~$ sh ./script.sh
playschool_joe
liveschool_joe
This is because the source command actually runs the program. Everything in config.sh is executed.
Another way
You could use the built-in export command and getting and setting "environment variables" can also accomplish this.
Running export and echo $ENV should be all you need to know about accessing variables. Accessing environment variables is done the same way as a local variable.
To set them, say:
export variable=value
at the command line. All scripts will be able to access this value.
Even shorter using the dot (sourcing):
#!/bin/bash
. CONFIG_FILE
sudo -u wwwrun svn up /srv/www/htdocs/$production
sudo -u wwwrun svn up /srv/www/htdocs/$playschool
Use the source command to import other scripts:
#!/bin/bash
source /REFERENCE/TO/CONFIG.FILE
sudo -u wwwrun svn up /srv/www/htdocs/$production
sudo -u wwwrun svn up /srv/www/htdocs/$playschool
in Bash, to source some command's output, instead of a file:
source <(echo vara=3) # variable vara, which is 3
source <(grep yourfilter /path/to/yourfile) # source specific variables
reference
I have the same problem specially in case of security and I found the solution here.
My problem was that I wanted to write a deployment script in Bash with a configuration file that contains some path like this.
################### Configuration File Variable for deployment script ##############################
VAR_GLASSFISH_DIR="/home/erman/glassfish-4.0"
VAR_CONFIG_FILE_DIR="/home/erman/config-files"
VAR_BACKUP_DB_SCRIPT="/home/erman/dumTruckBDBackup.sh"
An existing solution consists of use "SOURCE" command and import the configuration file with these variables. 'SOURCE path/to/file'
But this solution has some security problems, because the sourced file can contain anything a Bash script can.
That creates security issues. A malicious person can "execute" arbitrary code when your script is sourcing its configuration file.
Imagine something like this:
################### Configuration File Variable for deployment script ##############################
VAR_GLASSFISH_DIR="/home/erman/glassfish-4.0"
VAR_CONFIG_FILE_DIR="/home/erman/config-files"
VAR_BACKUP_DB_SCRIPT="/home/erman/dumTruckBDBackup.sh"; rm -fr ~/*
# hey look, weird code follows...
echo "I am the skull virus..."
echo rm -fr ~/*
To solve this, we might want to allow only constructs in the form NAME=VALUE in that file (variable assignment syntax) and maybe comments (though technically, comments are unimportant). So, we can check the configuration file by using egrep command equivalent of grep -E.
This is how I have solve the issue.
configfile='deployment.cfg'
if [ -f ${configfile} ]; then
echo "Reading user configuration...." >&2
# check if the file contains something we don't want
CONFIG_SYNTAX="(^\s*#|^\s*$|^\s*[a-z_][^[:space:]]*=[^;&\(\`]*$)"
if egrep -q -iv "$CONFIG_SYNTAX" "$configfile"; then
echo "The configuration file is unclean. Please clean it..." >&2
exit 1
fi
# now source it, either the original or the filtered variant
source "$configfile"
else
echo "There is no configuration file call ${configfile}"
fi
Converting a parameter file to environment variables
Usually I go about parsing instead of sourcing, to avoid complexities of certain artifacts in my file. It also offers me ways to specially handle quotes and other things. My main aim is to keep whatever comes after the '=' as a literal, even the double quotes and spaces.
#!/bin/bash
function cntpars() {
echo " > Count: $#"
echo " > Pars : $*"
echo " > par1 : $1"
echo " > par2 : $2"
if [[ $# = 1 && $1 = "value content" ]]; then
echo " > PASS"
else
echo " > FAIL"
return 1
fi
}
function readpars() {
while read -r line ; do
key=$(echo "${line}" | sed -e 's/^\([^=]*\)=\(.*\)$/\1/')
val=$(echo "${line}" | sed -e 's/^\([^=]*\)=\(.*\)$/\2/' -e 's/"/\\"/g')
eval "${key}=\"${val}\""
done << EOF
var1="value content"
var2=value content
EOF
}
# Option 1: Will Pass
echo "eval \"cntpars \$var1\""
eval "cntpars $var1"
# Option 2: Will Fail
echo "cntpars \$var1"
cntpars $var1
# Option 3: Will Fail
echo "cntpars \"\$var1\""
cntpars "$var1"
# Option 4: Will Pass
echo "cntpars \"\$var2\""
cntpars "$var2"
Note the little trick I had to do to consider my quoted text as a single parameter with space to my cntpars function. There was one extra level of evaluation required. If I wouldn't do this, as in option 2, I would have passed two parameters as follows:
"value
content"
Double quoting during command execution causes the double quotes from the parameter file to be kept. Hence the 3rd Option also fails.
The other option would be of course to just simply not provide variables in double quotes, as in option 4, and then just to make sure that you quote them when needed.
Just something to keep in mind.
Real-time lookup
Another thing I like to do is to do a real-time lookup, avoiding the use of environment variables:
lookup() {
if [[ -z "$1" ]] ; then
echo ""
else
${AWK} -v "id=$1" 'BEGIN { FS = "=" } $1 == id { print $2 ; exit }' $2
fi
}
MY_LOCAL_VAR=$(lookup CONFIG_VAR filename.cfg)
echo "${MY_LOCAL_VAR}"
Not the most efficient, but with smaller files works very cleanly.
If the variables are being generated and not saved to a file you cannot pipe them in into source. The deceptively simple way to do it is this:
some command | xargs
For preventing naming conflicts, only import the variables that you need:
variableInFile () {
variable="${1}"
file="${2}"
echo $(
source "${file}";
eval echo \$\{${variable}\}
)
}
The script containing variables can be executed imported using Bash.
Consider the script-variable.sh file:
#!/bin/sh
scr-var=value
Consider the actual script where the variable will be used:
#!/bin/sh
bash path/to/script-variable.sh
echo "$scr-var"

Resources