How can I disable * expansion in a script? - bash

I have a strange problem - possibly I'm just going blind. I have this short script, which replaces the string #qry# in the here-document with a select statement in a file and then pipes it to mysql:
#!/bin/bash
if [[ "$1" == "-h" ]]
then
echo "sqljob [sqlfile] [procnm] [host] [database] [config file]"
echo " sqlfile: text file containing an SQL statement"
echo " procnm: name that will given to the new, stored procedure"
echo " host: hostname of IP address of the database server"
echo " database: the procedure will be created here"
echo " config file: default configuration file with username and password"
exit
fi
infile=$1
procnm=$2
hn=$3
pn=$4
db=$5
mycfg=$6
{
set -o noglob
sed -e "s/#qry#/$(echo $(cat $infile))/g" <<!
drop procedure if exists $procnm;
delete from jobs where jobname="$procnm";
insert into jobs
set
notes="SQL job $procnm",
jobname="$procnm",
parm_tmpl='int';
delimiter //
create procedure $procnm(vqid int)
begin
call joblogmsg(vqid,0,"$procnm","","Executing #qry#");
drop table if exists ${procnm}_res;
create table ${procnm}_res as
#qry#
end//
delimiter ;
!
} | mysql --defaults-file=$mycfg -h $hn -P $pn $db
However, when the select contains *, it expands to whatever is in the directory even though I use noglob. However, it works from the command line:
$ set -o noglob
$ ls *
What am I doing wrong?
Edit
Block Comments in a Shell Script has been suggested as a duplicate, but as you will notice, I need to expand ${procnm} in the here-doc; I just need to avoid the same happening to select *.

I suspect it is because the construct echo (cat). The echo command gets the * from the cat command and the shell in which it runs expands it. In that shell set noglob is not active.
Try leaving the echo away: /$(cat $infile)/, in the end that is the data you need; then there is no extra glob expansion by a shell.

Related

why result set value not stored in arraylist in shell script

sample code below
psql -h $host -U postgres -d postgres -At -c "select partner_country_id as country , case when (threshold is null) then global_threshold else threshold end as threshold from ra_country_day_threshold " \
| while read -a Record
do
arrIN=(${Record[0]//|/ })
col1=${arrIN[0]}
col2=${arrIN[1]}
country_array["$col1"]="$col2"
echo "Col1:$col1 Col2:$col2"
done
echo "Elements:${country_array[#]}"
echo "length: ${#country_array[#]}"
Result
empty elements and length 0
The answer is simple, while command create a subprocess with its own context, if you create a new variable in that context, it will not be accessible outside of it.
Meaning the variable will not be accessible when you are outside the loop.
My suggestion is that you store the result inside a temporary file that will be available within all your script, then outside your loop, read that file.

Parameterizing to dynamically generate the extract file from Oracle table using Shell Script

I have a requirement where I need to parameterize to generate one extract file from multiple Oracle tables through the UNIX shell script.
Here is the script which I have written to generate one tab delimited file which will fetch all the data from EMPLOYEE table.
I need to parameterize the TABLE_NAME,OWNER_NAME,USERNAME,PASSWORD and HOST to generate from 12 more tables.
So, I would like to have only one SQL to dyngenerate the extract for 12 tables by passing these parameters values when executing the scripts.
Could you please give me show me how we can modify the below script and how to pass the parameter during the script execution.
Second Requirement is to generate the file incrementally based on a column for example, ETL_UPDATE_TS. can you please show me this also.
Sample Scripts
#!/usr/bin/ksh
TD=/mz/mz01/TgtFiles
MD=/mz/mz01/Scripts
#CAQH_Server=sftp.org
#UN=user
#PWD=password
#RD=Incoming
#RD=/home/
cd $TD
FILE="EMPLOYEE.TXT"
sqlplus -s scott/tiger#db <<EOF
SET PAGES 999
SET COLSEP " "
SET LINES 999
SET FEEDBACK OFF
SPOOL $FILE
SELECT * FROM EMP;
SPOOL OFF
EXIT
EOF
Handling your parameters in a similar way you did for $FILE variable and passing them as options to the script
#!/usr/bin/ksh
TD=/mz/mz01/TgtFiles
MD=/mz/mz01/Scripts
cd $TD
FILE="undefined"
TABLE="undefined"
while getopts :f:t: opt
do
case $opt in
f) FILE=${OPTARG} ;;
t) TABLE=${OPTARG} ;;
*) echo "invalid flag" ;;
esac
done
if [ "$TABLE" == "undefined" ]; then
echo "ERROR. TABLE is undefined, use -f option."
exit 1
fi
# More required variables checks here
# create more options to parameterize connection
sqlplus -s scott/tiger#db <<EOF
SET PAGES 999
SET COLSEP " "
SET LINES 999
SET FEEDBACK OFF
SPOOL $FILE
SELECT * FROM $TABLE;
SPOOL OFF
EXIT
EOF
An execute it as
my_script.sh -f "EMPLOYEE.TXT" -t "EMP"

How to print each element of Multi-line String Parameter by bash shell in Jenkins Job

It is strange while I want to use Multi-line String Parameter in Jenkins job.
The parameter name is PRS_INFO, and its default text is:
PT54321:file xxx not exit
PT74231:xxx reboot
After running this job, the parameter is set to environment:
PRS_INFO PT54321:file xxx not exit>PT74231:xxx reboot
But when I want to print it line by line using bash shell code like:
set -x
IFS_OLD=$IFS
IFS='>'
for i in $PRONTO_INFO; do
echo $i
done
IFS=$IFS_OLD
It prints the log in console:
+ IFS_OLD='
'
+ IFS='>'
+ for i in '$PRS_INFO'
+ echo 'PT54321:file xxx not exit
PT74231:xxx reboot'
PT54321:file xxx not exit
PT74231:xxx reboot
+ IFS='
Why is the echo just called once to print all lines, not twice to print two lines?
PS. Jenkins 2.32.3 is the Jenkins version we use.
There does not seem to be a > character in your PRS_INFO variable;
Try:
set -x
OLD_IFS="$IFS"
IFS=$'\n'
for line in $PRS_INFO; do
echo "$line"
done
IFS="$OLD_IFS"

Echo printing variables in a completely wrong order

I am trying to create a string with a query that will be save / send to another location, this string contains different variables.
The issue that I am having is that the echo of the variables are completely upside down and mix.
See code below:
tokenID=$(docker exec -ti $dockerContainerID /bin/sh -c "cat /tempdir/tokenfile.txt")
serverName="asdasd"
attQuery="$tokenID $serverName"
agentRegQuery="$./opt/mule/bin/amc_setup -H $attQuery"
echo TOKEN ID $tokenID
echo SERVER NAME $serverName
echo $attQuery
echo $agentRegQuery
Find below the output I am receiving:
TOKEN ID 29a6966f-fa0e-4f08-87eb-418722872d80---46407
SERVER NAME asdasd
asdasdf-fa0e-4f08-87eb-418722872d80---46407
asdasdmule/bin/amc_setup -H 29a6966f-fa0e-4f08-87eb-418722872d80---46407
There's a carriage return character at the end of the tokenID variable, probably because /tempdir/tokenfile.txt is in DOS/Windows format (lines end with carriage return+linefeed), not unix (lines end with just linefeed). When you print tokenID by itself, it looks ok, but if you print something else after that on the same line, it winds up overwriting the first part of the line. So when you print $attQuery, it prints this:
29a6966f-fa0e-4f08-87eb-418722872d80---46407[carriage return]
asdasd
...but with the second line printed on top of the first, so it comes out as:
asdasdf-fa0e-4f08-87eb-418722872d80---46407
The solution is to either convert the file to unix format (dos2unix will do this if you have it), or remove the carriage return in your script. You can do it like this:
tokenID=$(docker exec -ti $dockerContainerID /bin/sh -c "cat /tempdir/tokenfile.txt" | tr -d '\r')
I think everything works as it should
echo TOKEN ID $tokenID -> TOKEN ID 29a6966f-fa0e-4f08-87eb-418722872d80---46407
echo SERVER NAME $serverName -> SERVER NAME asdasd
echo $attQuery -> asdasdf-fa0e-4f08-87eb-418722872d80---46407
echo $agentRegQuery -> asdasdmule/bin/amc_setup -H 29a6966f-fa0e-4f08-87eb-418722872d80---46407
Why do you think something is wron here?
Best regards, Georg

Run a string as a command within a Bash script

I have a Bash script that builds a string to run as a command
Script:
#! /bin/bash
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
teamAComm="`pwd`/a.sh"
teamBComm="`pwd`/b.sh"
include="`pwd`/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
illcommando="$serverbin include='$include' server::team_l_start = '${teamAComm}' server::team_r_start = '${teamBComm}' CSVSaver::save='true' CSVSaver::filename = 'out.csv'"
echo "running: $illcommando"
# $illcommando > server-output.log 2> server-error.log
$illcommando
which does not seem to supply the arguments correctly to the $serverbin.
Script output:
running: /usr/local/bin/rcssserver include='/home/joao/robocup/runner_workdir/server_official.conf' server::team_l_start = '/home/joao/robocup/runner_workdir/a.sh' server::team_r_start = '/home/joao/robocup/runner_workdir/b.sh' CSVSaver::save='true' CSVSaver::filename = 'out.csv'
rcssserver-14.0.1
Copyright (C) 1995, 1996, 1997, 1998, 1999 Electrotechnical Laboratory.
2000 - 2009 RoboCup Soccer Simulator Maintenance Group.
Usage: /usr/local/bin/rcssserver [[-[-]]namespace::option=value]
[[-[-]][namespace::]help]
[[-[-]]include=file]
Options:
help
display generic help
include=file
parse the specified configuration file. Configuration files
have the same format as the command line options. The
configuration file specified will be parsed before all
subsequent options.
server::help
display detailed help for the "server" module
player::help
display detailed help for the "player" module
CSVSaver::help
display detailed help for the "CSVSaver" module
CSVSaver Options:
CSVSaver::save=<on|off|true|false|1|0|>
If save is on/true, then the saver will attempt to save the
results to the database. Otherwise it will do nothing.
current value: false
CSVSaver::filename='<STRING>'
The file to save the results to. If this file does not
exist it will be created. If the file does exist, the results
will be appended to the end.
current value: 'out.csv'
if I just paste the command /usr/local/bin/rcssserver include='/home/joao/robocup/runner_workdir/server_official.conf' server::team_l_start = '/home/joao/robocup/runner_workdir/a.sh' server::team_r_start = '/home/joao/robocup/runner_workdir/b.sh' CSVSaver::save='true' CSVSaver::filename = 'out.csv' (in the output after "runnning: ") it works fine.
You can use eval to execute a string:
eval $illcommando
your_command_string="..."
output=$(eval "$your_command_string")
echo "$output"
I usually place commands in parentheses $(commandStr), if that doesn't help I find bash debug mode great, run the script as bash -x script
don't put your commands in variables, just run it
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
PWD=$(pwd)
teamAComm="$PWD/a.sh"
teamBComm="$PWD/b.sh"
include="$PWD/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
$serverbin include=$include server::team_l_start = ${teamAComm} server::team_r_start=${teamBComm} CSVSaver::save='true' CSVSaver::filename = 'out.csv'
./me casts raise_dead()
I was looking for something like this, but I also needed to reuse the same string minus two parameters so I ended up with something like:
my_exe ()
{
mysql -sN -e "select $1 from heat.stack where heat.stack.name=\"$2\";"
}
This is something I use to monitor openstack heat stack creation. In this case I expect two conditions, an action 'CREATE' and a status 'COMPLETE' on a stack named "Somestack"
To get those variables I can do something like:
ACTION=$(my_exe action Somestack)
STATUS=$(my_exe status Somestack)
if [[ "$ACTION" == "CREATE" ]] && [[ "$STATUS" == "COMPLETE" ]]
...
Here is my gradle build script that executes strings stored in heredocs:
current_directory=$( realpath "." )
GENERATED=${current_directory}/"GENERATED"
build_gradle=$( realpath build.gradle )
## touch because .gitignore ignores this folder:
touch $GENERATED
COPY_BUILD_FILE=$( cat <<COPY_BUILD_FILE_HEREDOC
cp
$build_gradle
$GENERATED/build.gradle
COPY_BUILD_FILE_HEREDOC
)
$COPY_BUILD_FILE
GRADLE_COMMAND=$( cat <<GRADLE_COMMAND_HEREDOC
gradle run
--build-file
$GENERATED/build.gradle
--gradle-user-home
$GENERATED
--no-daemon
GRADLE_COMMAND_HEREDOC
)
$GRADLE_COMMAND
The lone ")" are kind of ugly. But I have no clue how to fix that asthetic aspect.
To see all commands that are being executed by the script, add the -x flag to your shabang line, and execute the command normally:
#! /bin/bash -x
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
teamAComm="`pwd`/a.sh"
teamBComm="`pwd`/b.sh"
include="`pwd`/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
$serverbin include="$include" server::team_l_start="${teamAComm}" server::team_r_start="${teamBComm}" CSVSaver::save='true' CSVSaver::filename='out.csv'
Then if you sometimes want to ignore the debug output, redirect stderr somewhere.
For me echo XYZ_20200824.zip | grep -Eo '[[:digit:]]{4}[[:digit:]]{2}[[:digit:]]{2}'
was working fine but unable to store output of command into variable.
I had same issue I tried eval but didn't got output.
Here is answer for my problem:
cmd=$(echo XYZ_20200824.zip | grep -Eo '[[:digit:]]{4}[[:digit:]]{2}[[:digit:]]{2}')
echo $cmd
My output is now 20200824

Resources