my script refuses to work under cron, but works fine while executing manually
#!/bin/bash
LOGFILE=/opt/xxx/scripts/rc.log
fUpMail() {
echo -e "Hello!\n\n$1 xx\n\nBest regards,\n\nCheck LLC 2k18" | mailx -s "$1 Rates were not imported" smone#smth.com
}
curDate=`date +%Y-%m-%d`
#postgres expression output being assigned to a variable
rateQ=`PGPASSWORD=xxxxxx psql -t -h xxx.xxx.228.134 -p 5433 -d axx2 -U axxx2bo << EOF
SELECT COUNT(id) FROM quote WHERE f_date = '$curDate'
EOF`
#same for demodb
rateDemo=`PGPASSWORD=xxx psql -t -h xx.xxx.42.14 -p 5432 -d axxxo -U acxxxxbo << EOF
SELECT COUNT(id) FROM quote WHERE f_date = '$curDate'
EOF`
#logging
printf "\n`date +%H:%M:%S` $curDate $rateQ $rateDemo\n" >> $LOGFILE
#check if rate value is not null
if [[ $(($rateQ)) != 0 ]] && [[ $(($rateDemo)) != 0 ]];
then
#posting a commentary into jira
curl -u xxx-support-bot:Rzq-xxx-xxx-gch -X POST --data '{"body": "'"$rateQ"' LIVE rates for '"$curDate"' were imported automatically'"\n"''"$rateDemo"' DEMO rates for '"$curDate"' were imported automatically"}' -H "Content-type: application/json" https://jira.in.xxx.com:443/rest/api/2/issue/xxxxxx-1024/comment >> $LOGFILE
else
#if rates were not imported
if [[ $(($rateQ)) == 0 ]];
then
echo "looks like LIVE rates for $curDate were not imported, please check manually!"
#sending a letter
fUpMail 'LIVE'
fi
if [[ $(($rateDemo)) == 0 ]];
then
echo "looks like DEMO rates for $curDate were not imported, please check manually!"
fUpMail 'DEMO'
fi
fi
cron sends following message:
/opt/xxx/scripts/ratecheck.sh: line 25: Timing is on.
6543
Time: 4.555 ms: syntax error: invalid arithmetic operator (error token is ".
6543
Time: 4.555 ms")
line 25 is
if [[ $(($rateQ)) != 0 ]] && [[ $(($rateDemo)) != 0 ]];
Could please someone help explaining what's wrong here?
You're getting more than a plain number back from psql and this is interfering with the type conversion you're doing. I think you can remove the extra output like this:
rateQ=$(PGPASSWORD=xxxxxx psql -t -h xxx.xxx.228.134 -p 5433 -d axx2 -U axxx2bo -q -c "SELECT COUNT(id) FROM quote WHERE f_date = '$curDate'")
rateDemo=$(PGPASSWORD=xxx psql -t -h xx.xxx.42.14 -p 5432 -d axxxo -U acxxxxbo -q -c "SELECT COUNT(id) FROM quote WHERE f_date = '$curDate'")
Note the addition of the -q flag:
-q
--quiet
Specifies that psql should do its work quietly. By default, it prints welcome messages and various informational output. If this option is used, none of this happens. This is useful with the -c option. This is equivalent to setting the variable QUIET to on.
https://www.postgresql.org/docs/9.0/app-psql.html
I also replaced your old-fashioned backticks with $() and put the SQL query into an argument.
If that doesn’t silence the additional output, you may also need to edit ~/.psqlrc for the user running the cron job and ensure there is no \timing line.
Related
I'm using mosquitto on an openWRT device to receive some data from a server and then send this same data to a local printer to print this data.
I'm using this script to receive the data
mosquitto_sub -h "${HOST}" -k 30 -c -p 8883 -t "${TOPIC}" -u "${USERNAME}" -P "${PASSWORD}" --id "${ID}" | /bin/sh /bin/printer_execute "${TOPIC}" "${PRINTER}" "${USERNAME}" "${PASSWORD}"
And the printer_execute code:
#!/bin/sh
TOPIC="${1}"
PRINTER="${2}"
USERNAME="${3}"
PASSWORD="${4}"
while read MSG
do
echo "input: ${MSG}"
echo "INPUT MSG: " "${MSG}" >> /root/log
RES=`curl -m 2 --header "Content-Type: text/xml;charset=UTF-8" --header "SOAPAction: ''" --header "If-Modified-Since: Thu, 01 Jan 1970 00:00:00 GMT" --data "${MSG}" "http://${PRINTER}/cgi-bin/epos/service.cgi?devid=local_printer&timeout=5000"`
mosquitto_pub -h ${HOST_PLACEHOLDER} -p 8883 -t "${TOPIC}-response" -m "${RES}" -u "${USERNAME}" -P "${PASSWORD}"
echo "RESULT CURL: " "${RES}" >> /root/log
done
This solution works with a relatively low messages per second, but when the volume is too high the printer_execute code stop working. I'm pretty new to shell scripting and I guess the problem could be caused by the pipe and while read pattern or by the while exit condition, but i'm not really sure.
Anyone has some idea or found a similar problem and know how to solve this?
EDIT:
In light of the answers i have tried to do this:
EDIT2: Sorry in the first edit i just added what i modified but the entire script is like that and the scope should be correct for the variables.
#!/bin/sh
TOPIC="${1}"
PRINTER="${2}"
USERNAME="${3}"
PASSWORD="${4}"
PrintOne(){
MSG="${1}"
RES=$(curl [params])
mosquitto_pub -h [host] -p 8883 -d -t "${TOPIC}-response" -m "${RES}" -u "${USERNAME}" -P "${PASSWORD}"
echo "RESULT CURL: " "${RES}" >> /root/log
}
while read msg ; do
PrintOne "$msg" &
done
With the printone and the appersand this take one message and stop working, without the & it's just like it was before.
You could try making a function to handle one message and calling that in the background (by appending an ampersand) so that you can respond quickly and in parallel - and that will allow you to take longer to handle each message... for a period. If your messages continually arrive faster than you can handle them, there will inevitably be a backlog.
Something like this:
#!/bin/bash
PrintOne(){
echo "Received $1"
curl ...
mosquitto_pub ...
echo $RESULT
}
while read msg ; do
PrintOne "$msg" &
done
If you want a little example. change the code to this and save it as go, and make it executable with chmod +x go
#!/bin/bash
PrintOne(){
echo "Received $1"
sleep 2
echo "Finished $1"
}
while read msg ; do
PrintOne "$msg" &
done
Now send it 10 lines:
seq 10 | ./go
Then remove the ampersand, and do exactly the same thing again and you will see the difference.
A more complete version of my answer is as follows:
#!/bin/bash
PrintOne(){
TOPIC="${1}"
PRINTER="${2}"
USERNAME="${3}"
PASSWORD="${4}"
curl ...
mosquitto_pub ...
echo $RESULT
}
while read msg ; do
PrintOne "${1}" "${2}" "${3}" "${4}" &
done
I have a below shell script which reads a file and updates the table. But the problem with the below code is that a connection is established for running each sql statements. I am looking for recommendation to enhance this code.
#!/bin/bash
input="/Users/temp/newfile.txt"
while IFS= read -r var
do
echo "update keyspace.tableName set randomColumn='randomText' where random_id='$var'; exit" | ./cqlsh serverName -u username -p password
if [ $? -eq 0 ]; then
echo SUCCESS
echo "select random_id,randomColumn from keyspace.tableName where random_id='$var'; exit" | ./cqlsh serverName -u username -p password
else
echo FAIL
fi
done < "$input"
I suggest doing this with the Python driver if you'd like better performance.
This example should be roughly equivalent:
#! /usr/bin/env python3
from cassandra.cluster import Cluster
from cassandra.auth import PlainTextAuthProvider
cluster = Cluster(auth_provider=PlainTextAuthProvider(username='cassandra', password='cassandra'))
session = cluster.connect()
with open("/Users/temp/newfile.txt") as f:
for line in f:
try:
rows = session.execute("UPDATE keyspace.tableName SET randomColumn='randomText' WHERE random_id=%(var)s", {'var'=line})
print("SUCCESS")
for r in rows:
print(r)
except Exception:
print("FAIL")
edit: you can even go a step further and use prepared statements and async queries, which will provide very significant performance increases if you are making a large volume of queries.
cqlsh has a -f option which allows to run cql statement from a file. You could generate the cql statements by parsing your newfile.txt and then run cqlsh with -f option
At the very basic level, you could do something like this:
#!/bin/bash
input="newfile.txt"
while IFS= read -r var
do
echo "update ks.t1 set name='randomText' where id=$var;" >> result
done < "$input"
./cqlsh serverName -u username -p password -f result
if [ $? -eq 0 ]; then
echo SUCCESS
echo "select * from keyspace.tableName; exit" | ./cqlsh serverName -u username -p password
else
echo FAIL
fi
I suggest to use cqlsh -e with xargs :
#!/bin/bash
input="/Users/temp/newfile.txt"
while IFS= read -r var
do
echo "update keyspace.tableName set randomColumn='randomText' where random_id='$var';" | xargs cqlsh serverName -u 'username' -p 'password' -e
if [ $? -eq 0 ]; then
echo SUCCESS
echo "select random_id,randomColumn from keyspace.tableName where random_id='$var';" | xargs cqlsh serverName -u 'username' -p 'password' -e
else
echo FAIL
fi
done < "$input"
Running in a docker container, I am trying to determine if a table exits in the mssql database:
RESULT='/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P $SA_PASSWORD -d master -i "SELECT name FROM master.sys.databases WHERE name = N'MyTable'"'
if [ "$RESULT" == "MyTable" ]; then
echo YES
fi
echo "$RESULT"
echo "$RESULT" always just outputs the entire command as a string, its not getting executed. So its just assigning it as a sting...
How can I execute it and assign the result?
Bash does not execute commands defined in single quotes. Single quotes are used for string definitions.
What you try to do is called command substitution. And it can be done in two different ways:
RESULT=`command`
RESULT=$(command)
So your example should look like this:
RESULT=`/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P $SA_PASSWORD -d master -i "SELECT name FROM master.sys.databases WHERE name = 'MyTable'`
if [ "$RESULT" == "MyTable" ]; then
echo YES
fi
echo "$RESULT"
You can find more information regarding command substitution here:
https://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html
I am coding a script to select some IPs from a table in a DB, and then use IPTables rules to ban theses IPs, last step is to notify by e-mail, but I am getting 2 errors:
#!/bin/bash
Now=$(date +"%d-%m-%Y %T")
fileLocation="/var/lib/mysql/DBName/"
fileName="ip2ban.txt"
filePath=$fileLocation$fileName
curLocation=$(pwd)
#Connect to DB and select ban_ip
mysql -u root -pPASSWORD -D DBName -e 'SELECT ip INTO OUTFILE "'$filePath'" FROM ban_ip WHERE ip_tables = "0"' >> banIP.log 2>&1
selRes=$?
# If the command was successful
if [ $selRes -eq "0" ]
then
# We need to check if the file exists on the saved location
#find $fileLocation -type f -iname "ip2ban.txt" -empty => To check if file empty or not
if [ -f $filePath ]
then
mv $filePath $curLocation'/ip2ban.txt'
# Connect to DB and update the ban_ip
mysql -u root -pPASSWORD -D DBName -e 'UPDATE ban_ip SET ip_tables = "1" WHERE ip_tables = "0"' >> banIP.log 2>&1
upRes=$?
if [ $upRes -eq "0" ]
then
# Send message for succesful result
echo -e "Database updated with new banned IPs on $Now \nThank you for using this script" 2>&1 | sed '1!b;s/^/To: myID#gmail.com\nSubject: New banned IPs[Success]\n\n/' | sendmail -t
else
# Send message for failure result
echo -e "We cannot update the ban_ip table on $Now \nThank you for using this script" 2>&1 | sed '1!b;s/^/To: myID#gmail.com\nSubject: [Failure] New banned IPs\n\n/' | sendmail -t
fi
fi
else
echo 'Something wrong with Select statment on' $Now >> banIP.log
fi
# Save IPTables rules
iptables-save > /root/Scripts/IPTables/BannedIPs.conf // LIGNE 53
I am getting 2 errors:
line 53: iptables-save: command not found
line 37: sendmail: command not found
However the sendamil is already installed, with mail, postfix:
# which sendmail
/usr/sbin/sendmail
# which mail
/usr/bin/mail
# which postfix
/usr/sbin/postfix
Thanks for your usual support
According to the crontab(5) man page for Linux:
PATH is set to "/usr/bin:/bin".
Meaning your shell script will not be able to find anything under /usr/sbin or /sbin. Change this with by adding the following near the top of your script:
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
Also the environment can be set from within the crontab. See https://stackoverflow.com/a/14694543/5766144 for how to do this.
We have a script we use to promote databases from dev to QA. Currently the script has a select statement we use to print out all the available databases and then takes your selection and does the various bits to promote the db.
I want to add a parameter to bypass that select statement for scripting purposes.
Here's how the script works currently:
# GET LIST OF DATABASES
DATABASES=`echo "show databases;"|/usr/bin/mysql -u root -h $DEVHOST|grep -v "information_schema"|grep -v "Database"|grep -v "performance_schema"|grep -v "mysql"`
eval set $DATABASES
select DEVDB in "$#"
do
if [[ "$DEVDB" =~ "dev1" ]]; then
QADB=`echo $DEVDB|sed -e s/_dev1_/_qa1_/`
fi
# Does various bits after this
Here's what I'm trying to do:
arg1=$1
if [ ! -z "$arg1" ]; then
DEVDB=$arg1
else
# GET LIST OF DATABASES
DATABASES=`echo "show databases;"|/usr/bin/mysql -u root -h $DEVHOST|grep -v "information_schema"|grep -v "Database"|grep -v "performance_schema"|grep -v "mysql"`
eval set $DATABASES
select DEVDB in "$#"
do
fi
if [[ "$DEVDB" =~ "dev1" ]]; then
QADB=`echo $DEVDB|sed -e s/_dev1_/_qa1_/`
fi
I'm getting an error at fi, I assume it's related to the select... do. How do I wrap these up in my if statements?
You need to do something in the body of the select. In this case it's sufficient to check that the variable has a value:
PS3="Select a database: "
select DEVDB; do
[[ $DEVDB ]] && break
done
echo "you selected $DEVDB"
Definitely don't need eval here -- it's always a code smell.
set -- $(
/usr/bin/mysql -u root -h $DEVHOST <<< "show databases;" |
grep -v -e information_schema -e Database -e performance_schema -e mysql
)