I'm trying to run a script (automation.sh) automated in crontab.
(I'm on Ubuntu 14.04).
#!/usr/bin/env bash
day=$(date +%F -d'yesterday')
cat /home/tomi/logs/$day |grep registration > /home/tomi/registrations/$day
cat /home/tomi/logs/$day |grep free_tree > /home/tomi/free_tree/$day
cat /home/tomi/logs/$day |grep super_tree > /home/tomi/super_tree/$day
psql -U hello -d postgres -c "\COPY registrations FROM '/home/tomi/registrations/$day' DELIMITER ' '";
psql -U hello -d postgres -c "\COPY free_tree FROM '/home/tomi/free_tree/$day' DELIMITER ' '";
psql -U hello -d postgres -c "\COPY super_tree FROM '/home/tomi/super_tree/$day' DELIMITER ' '";
psql -U hello -d postgres -f daily_active_users.sql > /home/tomi/tmp1
psql -U hello -d postgres -f daily_revenue.sql > /home/tomi/tmp2
If I run this script normally from the command line, then the last two lines generate tmp1 and tmp2 with data in them. (That's the expected result.)
However, if I run this very same script in crontab, everything works, but the last two lines generate empty files (tmp1 and tmp2).
The tricky thing is that when I break this script into two scripts (eg. automated.sh and automated2.sh) and run the last two lines in crontab 5 minutes later (via this automated2.sh script), tmp1 and tmp2 are generated correctly, with data in them.
Any idea, what can cause this?
Answer is in the comments:
full path is missing on last two lines in automation.sh!
Thanks!
Related
I run a few scripts 1 by 1
cat 001.sh
sh /home/mysqldom/da-cron/f_mysqldom_nrd/5_change_nrd_tld.sh
sh /home/mysqldom/da-cron/f_mysqldom_nrd/5_proxy_removed.sh
sh /home/mysqldom/da-cron/f_mysqldom_nrd/6_sync_nrd.sh
The last script wont work... if I run manually it work very well...
the script is
cat 6_sync_nrd.sh
source /home/mysqldom/da-cron/var.sh
cd /home/mysqldom/da-cron/f_mysqldom_nrd/
mysql -u mysqldom_fnrd -p$mysqldom_fnrd_password -D mysqldom_fnrd -e "UPDATE \`$yesterday\` SET sync='$yesterday';"
mysql -u mysqldom_fnrd -p$mysqldom_fnrd_password -D mysqldom_fnrd -e "DELETE FROM \`$yesterday\` WHERE domain_name = 'domain_name';"
sed s/change_database/$yesterday/g update.conf > $yesterday.conf
/usr/share/logstash/bin/logstash -f $yesterday.conf --path.data /var/lib/logstash108
rm -rf nohup.out
The 6 has to be run after 5
any idea whats worn in it
I have been tasked with replacing ISQL in a lot of our bash scripts with sqlcmd. ISQL allows piping a variable in it's execution.
An example would be:
SQL_STATEMENT="SELECT TOP 1 SYS_USER_NAME FROM SYS_USER"
echo $SQL_STATEMENT | isql -b -d, $DSN $DBUID $DBPWD >> setupdb_test.txt
From what I can tell this is not viable in sqlcmd. How can I do this? What flags does sqlcmd have to allow this to happen?
Here is what I have tried and have had a good result BUT I really do not want to create the file sql_command.sql every time a particular script runs:
echo "SELECT TOP 1 SYS_USER_NAME FROM SYS_USER" > sql_command.sql
sqlcmd -S $DB -U $DBUID -P $DBPWD -d $DSN -i sql_command.sql >> setupdb_test.txt
Programs originating on Windows can be picky about how they handle non-regular files and I don't have the opportunity to test, but you can try the typical Unix tricks for providing a "file" with data from an echo.
Either /dev/stdin:
echo "SELECT TOP 1 SYS_USER_NAME FROM SYS_USER" | sqlcmd -S "$DB" -U "$DBUID" -P "$DBPWD" -d "$DSN" -i /dev/stdin
or process substitution:
sqlcmd -S "$DB" -U "$DBUID" -P "$DBPWD" -d "$DSN" -i <(echo "SELECT TOP 1 SYS_USER_NAME FROM SYS_USER")
I want to export a random entry of my database into a file with the command
SELECT * FROM my_table ORDER BY RANDOM() LIMIT 1 \g /path/file;
This query works if I enter it in my db terminal, but I want to us this query with a bash script but then I get the error: syntax error at or near "\g"
My bash script looks like this:
PGPASSWORD=*** psql -U user -d db_name -h localhost -p port -t -c "SELECT * FROM my_table ORDER BY RANDOM() LIMIT 1 \g /path/file"
Bash is interpreting the string and trying to interpolate it. Probably, escaping the backslash will solve your problem.
PGPASSWORD=*** psql -U user -d db_name -h localhost -p port -t -c "SELECT * FROM my_table ORDER BY RANDOM() LIMIT 1 \\g /path/file"
A SQL statement terminated by \g is not supported by the -c command switch. Per documentation of -c:
-c command
...
command must be either a command string that is completely parsable by the server (i.e., it contains no psql-specific features), or a single backslash command. Thus you cannot mix SQL
and psql meta-commands with this option
To redirect the results to a file, there are several options:
shell redirection: psql [other options] -Atc 'SELECT...' >/path/to/data.txt
-A is to switch to unaligned mode (no space fillers to align columns).
put the SQL part in a heredoc text instead of the command line:
psql [options] <<EOF
SELECT ... \g /path/to/file
EOF
This form has the advantage that multiline statements or multiple statements are supported directly.
\copy of the query. Be aware that COPY to a FILE is different: it creates the file on the server with the permissions of postgres and requires being a database superuser. COPY TO STDOUT works too but is not better than SELECT concerning the redirection.
I found a solution for my script, and now it works.
#!/bin/bash
RANDOM_NUMBER=0
while true
do
for i in `seq 1`
do
RANDOM_NUMBER=$(($RANDOM % 100000))
echo $RANDOM_NUMBER
PGPASSWORD=*** psql -U user_name -d db_name -h localhost -p PORT -c
"INSERT INTO numbers (number) VALUES ('$RANDOM_NUMBER');"
done
sleep 10
for i in `seq 1`
do
PGPASSWORD=*** psql -U user_name -d db_name -h localhost -p PORT -c
"DELETE FROM numbers WHERE id = (SELECT id FROM numbers ORDER BY RANDOM() LIMIT 1);"
done
done
I can successfully run SQL (Postgres) files from command line following instructions here:
Run a PostgreSQL .sql file using command line arguments
In particular, I use something like
psql -d DBPASSWORD -a -f FILENAME
Problem is that this (and specifically, I believe the -a) prints the sql code out to the terminal. This is annoying because I am running a lot of files in sequence within a Python script using subprocess, and I would rather not have the SQL code print out in terminal. Is there a way to not print the SQL code out to terminal?
EDIT: I've tried adding the -q option like people said, but the code in the SQL file is still being printed out to terminal.
What I tried was
psql -q -d DBPASSWORD -a -f FILENAME
psql -d DBPASSWORD -q -a -f FILENAME
psql -d DBPASSWORD -a -q -f FILENAME
psql -d DBPASSWORD -a -f FILENAME -q
And in each of those cases, the code in FILENAME is being printed to terminal
You may want to redirect STDOUT, STDERR or both to a log file.
Something like one of these
psql ... > out.log
psql ... 2> err.log
psql ... &> out_and_err.log
I'm getting 7: Syntax error: "(" unexpected error while running bellow code on Ubuntu. But It's run on centos without any issues.
#!/bin/sh
#
TODATE=`date '+%Y-%b-%d'`
#
# Backup Creation for Databases
#
databases=(`echo 'show databases;' | mysql -u root -ppaSSword | grep -v ^Database$`)
for DB in "${databases[#]}"; do
mysqldump --force --opt --user=root --password=paSSword $DB | gzip > /mnt/Backup/DB/${DB}_${TODATE}.sql.gz
done
#
Please help me to solve this.
I can't figure out problem. But,
I'm using bellow code for backup. It's working fine with Ubuntu
#!/bin/bash
#
TODATE=`date '+%Y-%b-%d'`
databases="$(mysql -u root -ppaSSword -Bse 'show databases')"
for DB in $databases
do
mysqldump -u root -psqlMYadmin $DB | gzip > /mnt/Backup/DB/${DB}_${TODATE}.sql.gz
done
You can redirect the 'show databases' output to dump.txt file, if done then try.
#!/bin/bash
da=$(date +"%d-%m-%y")
for db in `cat dump.txt` ; do mysqldump --force --opt --user=root --password=paSSword $db | gzip /path/to/backup/$db_"$da".sql.gz ; done
You need to escape the last '$' on the line databases= ...
There's only one ( in the script and you have the shebang line #!/bin/sh. My best guess is that the program /bin/sh does not recognize array assignment, whereas /bin/bash would.
Change your shebang to #!/bin/bash.
You'd probably do better to use $(...) in place of the back ticks.) Also, as Sami Laine points out in his answer, it would be better if you quoted the regex to the grep command (though it is not the cause of your problem):
databases=( $(echo 'show databases;' | mysql -u root -ppaSSword | grep -v '^Database$') )