Shell Script Variable Scope with commnd - bash

I came across an interesting thing in Shell Scripting and not 100% sure why the behaviour is like this
I tried the below script:
#!/bin/sh
CMD="curl -XGET http://../endpoint";
var1=eval $CMD | sed -e 's/find/replace/g';
echo $var1; # Output: printed the value on this line
echo $var1; # Output: blank/no data printed (Why it is blank?)
I had to change the command in variable enclosing with back-tick ` to print the variable as many time as I wanted.
CMD="curl -XGET http://../endpoint";
var1=`eval $CMD | sed -e 's/find/replace/g'`;
echo $var1; # Output: printed the value on this line
echo $var1; # Output: printed the value on this line
Why I have to surround my command with ` to assign it's o/p to the variable in subsequent variable usage?
I have a feeling that it has something to do with the variable-command scope.
Shedding light on my understanding will be appreciated!
UPDATE:
I tried the below command and it is working in my env.
#!/bin/sh
CMD="curl -XGET http://www.google.com/";
var1=eval $CMD | sed -e 's/find/replace/g';
echo $var1; # Output: printed the value on this line
echo "######";
echo $var1; # Output: blank/no data printed (Why it is blank?)

sh/bash allows you to run a command with a variable in its environment, without permanently modifying the variable in the shell. This is great, because you can e.g. run a command in a certain language just one time without having to change your entire user's or system's language:
$ LC_ALL=en_US.utf8 ls foo
ls: cannot access foo: No such file or directory
$ LC_ALL=nb_NO.utf8 ls foo
ls: cannot access foo: Ingen slik fil eller filkatalog
However, this means that when you try to do
var=this is some command
you're trigger this syntax.
It means "run the command is a command and tell it that the variable var is set to this"
It does not assign "this is my string" to the variable, and it definitely does not evaluate "this is a string" as a command, and then assign its output to var.
Given this, we can look at what actually happened:
CMD="curl -XGET http://../endpoint";
var1=eval $CMD | sed -e 's/find/replace/g'; # No assignment, output to screen
echo $var1; # Output: blank/no data printed
echo $var1; # Output: blank/no data printed
There is no scope issue and no inconsistency: the variable is never assigned, and is never written by an echo statement.
var=`some command` (or preferably, var=$(some command)) works because this is valid syntax to assign output from a program to a variable.

The first example isn't doing what you think it is.
Neither echo is printing anything. Make them echo "[$var1]" to see that.
You need the backticks to run the command and capture its output.
Your first attempt was running the $CMD | sed -e 's/find/replace/g'; pipeline with the environment of $CMD containing var1 set to a value of eval.
You also shouldn't be putting commands inside strings (or using eval in general). See http://mywiki.wooledge.org/BashFAQ/001 for more on why.

Related

Set a variable equal to the output of a command containing non-command words

I'm writing a small script in which I want to set the value of a variable equal to the output of a command. However, the command in question is a call to another script with command-line arguments. I'm using backticks as you normally should in this scenario, but the problem is that the the computer gives an error, in which it tries to interpret the command-line arguments as commands.
#!/bin/bash
filename="$1"
while read p; do
echo "This is the gene we are looking at: ""$p"
lookIn= `./findGeneIn "$p" burgdorferi afzelii garinii hermsii miyamotoi parkeri`
echo "$lookIn"
#grep "$p" "$lookIn""/""prokka_""$lookIn""/*.tsv" | awk '{print $1}'
done < $filename
I'm trying to set variable lookIn equal to the output of ./findGeneIn "$p" burgdorferi afzelii garinii hermsii miyamotoi parkeri, where ./findGeneIn is a script, and the words burgdorferi,...,parkeri are command line arguments for ./findGeneIn.
The issue, is that I get an error saying "burgdorferi: command not found". So it's trying to interpret those arguments as commands. How do I get it to not do that?
lookIn= `./findGeneIn "$p" burgdorferi afzelii garinii hermsii miyamotoi parkeri`
^
Delete the extra space. Assignments must not have spaces around the equal sign.
With the space there, Bash parses the line as var=value command, which runs a command with $var temporarily set to "value". Or in this case, it interprets result of the backticks as a command name and lookIn= as an empty variable assignment.

Use substituted string as a command in shell script

Given the variables listed below:
echo ${userupper}_PYTHON
# YTU_PYTHON
echo $YTU_PYTHON
# /home/ytu/anaconda3/bin/python
echo $path
# foo.py
Now I'd like to execute /home/ytu/anaconda3/bin/python foo.py with userupper and path. I tried $(${userupper}_PYTHON) $path but it ends up with error messages including:
YTU_PYTHON: not found
foo.py: not found
It seems like it takes $(${userupper}_PYTHON) as bare YTU_PYTHON rather than expected $YTU_PYTHON. How should I do to make it right?
Edits:
The suggested duplication should have solved my problem. However for some unknown reasons it's not working.
#!/usr/bin/env bash
for user in ytu
do
. /home/${user}/.profile
userupper=$(echo ${user} | awk '{print toupper($0)}')
userpython=${userupper}_PYTHON
cd /home/${user}/H2-ML/crons
for path in $(ls | grep ^${user}_.*_monthly_report.py$)
do
echo ${userpython}
echo $path
echo $YTU_PYTHON
echo ${!userpython}
done
done
The code chunk above returns:
YTU_PYTHON
ytu_clinic249_monthly_report.py
/home/ytu/anaconda3/bin/python
send_monthly_reports.sh: 14: send_monthly_reports.sh: Bad substitution
, which makes me so confused.
Try this:
command=${userupper}_PYTHON
echo ${!command} $path # ommit echo if you want to execute command
in your case it echos:
/home/ytu/anaconda3/bin/python foo.py
Here is link that might be useful if you want to write bash scripts. You should not parse ls output in the nutshell.
You also create unnecessary new process, both lines do the same:
$(echo ${user} | awk '{print toupper($0)}')
${user^^} #paramenter expansion

String expansion - escaped quoted variable to value

To get started, here's the script I'm running to get the offending string:
# sed finds all sourced file paths from inputted file.
#
# while reads each match output from sed to $SOURCEFILE variable.
# Each should be a file path, or a variable that represents a file path.
# Any variables found should be expanded to the full path.
#
# echo and calls are used for demonstractive purposes only
# I intend to do something else with the path once it's expanded.
PATH_SOME_SCRIPT="/path/to/bash/script"
while read -r SOURCEFILE; do
echo "$SOURCEFILE"
"$SOURCEFILE"
$SOURCEFILE
done < <(cat $PATH_SOME_SCRIPT | sed -n -e "s/^\(source\|\.\|\$include\) //p")
You may also wish to use the following to test this out as mock data:
[ /path/to/bash/script ]
#!/bin/bash
source "$HOME/bash_file"
source "$GLOBAL_VAR_SCRIPT_PATH"
echo "No cow powers here"
For the tl;dr crew, basically the while loop spits out the following on the mock data:
"$HOME/bash_file"
bash: "$HOME/bash_file": no such file or directory
bash: "$HOME/bash_file": no such file or directory
"$GLOBAL_VAR_SCRIPT_PATH"
"$GLOBAL_VAR_SCRIPT_PATH": command not found
"$GLOBAL_VAR_SCRIPT_PATH": command not found
My question is, can you get the variable to expand correctly, e.g., print "/home//bash_file" and "/expanded/variable/path"? I should also state that although eval works I do not intend to use it because of its potential insecurities.
Protip that any variable value used in cat | sed would be available globally, including to the calling script, so it's not because the script cannot call the variable value.
FIRST SOLUTION ATTEMPT
Using anubhava's envsubst solution:
SOMEVARIABLE="/home/nick/.some_path"
while read -r SOURCEFILE; do
echo "$SOURCEFILE"
envsubst <<< "$SOURCEFILE";
done < <(echo -e "\"\$SOMEVARIABLE\"\n\"$HOME/.another_file\"")
This outputs the following:
"$SOMEVARIABLE"
""
"/home/nick/.another_file"
"/home/nick/.another_file"
Unfortunately, it does not expand the variable! Oh dear :(
SECOND SOLUTION ATTEMPT
Based upon the first attempt:
export SOMEVARIABLE="/home/nick/.some_path"
while read -r SOURCEFILE; do
echo "$SOURCEFILE"
envsubst <<< "$SOURCEFILE";
done < <(echo -e "\"\$SOMEVARIABLE\"\n\"$HOME/.another_file\"")
unset SOMEVARIABLE
which produces the results we wanted without eval and without messing with global variables (for too long anyway), hoorah!
Good runner-ups were further suggested using eval (although potentially unsafe) which can be found in this answer and here (link courtesy of anubhava's extended comments).
My question is, can you get the variable to expand correctly, e.g., print "/home//bash_file" and "/expanded/variable/path"?
Yes you can use envsubst program, that substitutes the values of environment variables:
while read -r sourceFile; do
envsubst <<< "$sourceFile"
done < <(sed -n "s/^\(source\|\.\|\$include\) //p" "$PATH_SOME_SCRIPT")
I think you are asking how to recursively expand variables in bash. Try
expanded=$(eval echo $SOURCEFILE)
inside your loop. eval runs the expanded command you give it. Since $SOURCEFILE isn't in quotes, it will be expanded to, e.g., $HOME/whatever. Then the eval will expand the $HOME before passing it to echo. echo will print the result, and expanded=$(...) will put the printed result in $expanded.

Passing a variable into awk within a shell script

I have a shell script that I'm writing to search for a process by name and return output if that process is over a given value.
I'm working on finding the named process first. The script currently looks like this:
#!/bin/bash
findProcessName=$1
findCpuMax=$2
#echo "parameter 1: $findProcessName, parameter2: $findCpuMax"
tempFile=`mktemp /tmp/processsearch.XXXXXX`
#echo "tempDir: $tempFile"
processSnapshot=`ps aux > $tempFile`
findProcess=`awk -v pname="$findProcessName" '/pname/' $tempFile`
echo "process line: "$findProcess
`rm $tempFile`
The error is occuring when I try to pass the variable into the awk command. I checked my version of awk and it definitely does support the -v flag.
If I replace the '/pname/' portion of the findProcess variable assignment the script works.
I checked my syntax and it looks right. Could anyone point out where I'm going wrong?
The processSnapshot will always be empty: the ps output is going to the file
when you pass the pattern as a variable, use the pattern match operator:
findProcess=$( awk -v pname="$findProcessName" '$0 ~ pname' $tempFile )
only use backticks when you need the output of a command. This
`rm $tempFile`
executes the rm command, returns the output back to the shell and, it the output is non-empty, the shell attempts to execute that output as a command.
$ `echo foo`
bash: foo: command not found
$ `echo whoami`
jackman
Remove the backticks.
Of course, you don't need the temp file at all:
pgrep -fl $findProcessName

Echo newline in Bash prints literal \n

How do I print a newline? This merely prints \n:
$ echo -e "Hello,\nWorld!"
Hello,\nWorld!
Use printf instead:
printf "hello\nworld\n"
printf behaves more consistently across different environments than echo.
Make sure you are in Bash.
$ echo $0
bash
All these four ways work for me:
echo -e "Hello\nworld"
echo -e 'Hello\nworld'
echo Hello$'\n'world
echo Hello ; echo world
echo $'hello\nworld'
prints
hello
world
$'' strings use ANSI C Quoting:
Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.
You could always do echo "".
For example,
echo "Hello,"
echo ""
echo "World!"
On the off chance that someone finds themselves beating their head against the wall trying to figure out why a coworker's script won't print newlines, look out for this:
#!/bin/bash
function GET_RECORDS()
{
echo -e "starting\n the process";
}
echo $(GET_RECORDS);
As in the above, the actual running of the method may itself be wrapped in an echo which supersedes any echos that may be in the method itself. Obviously, I watered this down for brevity. It was not so easy to spot!
You can then inform your comrades that a better way to execute functions would be like so:
#!/bin/bash
function GET_RECORDS()
{
echo -e "starting\n the process";
}
GET_RECORDS;
Simply type
echo
to get a new line
POSIX 7 on echo
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html
-e is not defined and backslashes are implementation defined:
If the first operand is -n, or if any of the operands contain a <backslash> character, the results are implementation-defined.
unless you have an optional XSI extension.
So I recommend that you should use printf instead, which is well specified:
format operand shall be used as the format string described in XBD File Format Notation [...]
the File Format Notation:
\n <newline> Move the printing position to the start of the next line.
Also keep in mind that Ubuntu 15.10 and most distros implement echo both as:
a Bash built-in: help echo
a standalone executable: which echo
which can lead to some confusion.
str='hello\nworld'
$ echo | sed "i$str"
hello
world
You can also do:
echo "hello
world"
This works both inside a script and from the command line.
On the command line, press Shift+Enter to do the line break inside the string.
This works for me on my macOS and my Ubuntu 18.04 (Bionic Beaver) system.
For only the question asked (not special characters etc) changing only double quotes to single quotes.
echo -e 'Hello,\nWorld!'
Results in:
Hello,
World!
There is a new parameter expansion added in Bash 4.4 that interprets escape sequences:
${parameter#operator} - E operator
The expansion is a string that is the value of parameter with
backslash escape sequences expanded as with the $'…' quoting
mechanism.
$ foo='hello\nworld'
$ echo "${foo#E}"
hello
world
I just use echo without any arguments:
echo "Hello"
echo
echo "World"
To print a new line with echo, use:
echo
or
echo -e '\n'
This could better be done as
x="\n"
echo -ne $x
-e option will interpret backslahes for the escape sequence
-n option will remove the trailing newline in the output
PS: the command echo has an effect of always including a trailing newline in the output so -n is required to turn that thing off (and make it less confusing)
My script:
echo "WARNINGS: $warningsFound WARNINGS FOUND:\n$warningStrings
Output:
WARNING : 2 WARNINGS FOUND:\nWarning, found the following local orphaned signature file:
On my Bash script I was getting mad as you until I've just tried:
echo "WARNING : $warningsFound WARNINGS FOUND:
$warningStrings"
Just hit Enter where you want to insert that jump. The output now is:
WARNING : 2 WARNINGS FOUND:
Warning, found the following local orphaned signature file:
If you're writing scripts and will be echoing newlines as part of other messages several times, a nice cross-platform solution is to put a literal newline in a variable like so:
newline='
'
echo "first line${newline}second line"
echo "Error: example error message n${newline}${usage}" >&2 #requires usage to be defined
If the previous answers don't work, and there is a need to get a return value from their function:
function foo()
{
local v="Dimi";
local s="";
.....
s+="Some message here $v $1\n"
.....
echo $s
}
r=$(foo "my message");
echo -e $r;
Only this trick worked on a Linux system I was working on with this Bash version:
GNU bash, version 2.2.25(1)-release (x86_64-redhat-linux-gnu)
You could also use echo with braces,
$ (echo hello; echo world)
hello
world
This got me there....
outstuff=RESOURCE_GROUP=[$RESOURCE_GROUP]\\nAKS_CLUSTER_NAME=[$AKS_CLUSTER_NAME]\\nREGION_NAME=[$REGION_NAME]\\nVERSION=[$VERSION]\\nSUBNET-ID=[$SUBNET_ID]
printf $outstuff
Yields:
RESOURCE_GROUP=[akswork-rg]
AKS_CLUSTER_NAME=[aksworkshop-804]
REGION_NAME=[eastus]
VERSION=[1.16.7]
SUBNET-ID=[/subscriptions/{subidhere}/resourceGroups/makeakswork-rg/providers/Microsoft.Network/virtualNetworks/aks-vnet/subnets/aks-subnet]
Sometimes you can pass multiple strings separated by a space and it will be interpreted as \n.
For example when using a shell script for multi-line notifcations:
#!/bin/bash
notify-send 'notification success' 'another line' 'time now '`date +"%s"`
With jq:
$ jq -nr '"Hello,\nWorld"'
Hello,
World
Additional solution:
In cases, you have to echo a multiline of the long contents (such as code/ configurations)
For example:
A Bash script to generate codes/ configurations
echo -e,
printf might have some limitation
You can use some special char as a placeholder as a line break (such as ~) and replace it after the file was created using tr:
echo ${content} | tr '~' '\n' > $targetFile
It needs to invoke another program (tr) which should be fine, IMO.

Resources