TAR doesn't work properly with the crontab - bash

First of all, I'm saying that it doesn't work properly with the crontab because when I run the script manually it works fine.
The problem is that when I run the backup script with the cronjob and... it's coming to tar up the mysql dump, the tar archive has only 16 bytes size (and its empty, so it looks like there were no files to pack into the archive), the strange thing about that is that when I run the script manually, it runs almost 5~ minutes, and the tar package size is ~1.8GB.
Here is my bash code:
#!/usr/local/bin/bash
# Configuration
BACKUPD="/backup/mysql"
MySQLuser='root'
MySQLpass='xxxx'
# End configuration
ROK=`date +%Y`
MIESIAC=`date +%m`
DZIEN=`date +%d`
GIM=`date +%H-%M`
if [ -d $BACKUPD/$ROK/$MIESIAC/$DZIEN ]
then
echo
else
mkdir -p $BACKUPD/$ROK/$MIESIAC/$DZIEN
fi
for db in $(echo "SHOW DATABASES;" | mysql --user=$MySQLuser --password=$MySQLpass | grep -v -e "Database" -e "information_schema")
do
mysqldump --skip-lock-tables --ignore-table=log.log --user="$MySQLuser" --password="$MySQLpass" $db >$BACKUPD/$ROK/$MIESIAC/$DZIEN/$db.sql
done
cd $BACKUPD/$ROK/$MIESIAC/$DZIEN && tar jcPf $BACKUPD/$ROK/$MIESIAC/$DZIEN/mysql-$GIM.tar.bz2 *.sql && rm -rf *.sql
Where is the problem? Did anyone experienced a problem like this before?
Regards.

Can you try with full path name for mysqldump and mysql inside your script.
So:
if which mysql is equal to /usr/local/mysql/bin/mysql
and
if which mysqldump is equal to /usr/local/mysql/bin/mysqldump
Modify your script to:
for db in $(echo "SHOW DATABASES;" | /usr/local/mysql/bin/mysql --user=$MySQLuser --password=$MySQLpass | grep -v -e "Database" -e "information_schema")
do
/usr/local/mysql/bin/mysqldump --skip-lock-tables --ignore-table=log.log --user="$MySQLuser" --password="$MySQLpass" $db >$BACKUPD/$ROK/$MIESIAC/$DZIEN/$db.sql
done

My guess is that the last line is your problem. The shell glob (*.sql) in:
cd $BACKUPD/$ROK/$MIESIAC/$DZIEN && tar jcPf $BACKUPD/$ROK/$MIESIAC/$DZIEN/mysql-$GIM.tar.bz2 *.sql && rm -rf *.sql
is expanded in the current directory and not after the cd as you might expect. Try the following instead, it is safer.
old_dir=`pwd`
cd "$BACKUPD/$ROK/$MIESIAC/$DZIEN"
tar jcPf mysql-$GIM.tar.bz2 *.sql
rm -fr *.sql
cd "$old_dir"
There still might not be any .sql files to tar ball. I don't have mysql installed but I suspect that the for loop is messed up as well. Try something like the following instead:
mysqlshow | \
xargs mysqldump --databases | \
bzip2 > $BACKUPD/$ROK/$MIESIAC/$DZIEN/mysql-$GIM.bz2
You will probably beed to insert other arguments for the mysqlshow and mysqldump commands. Of course this won't create a tarball but it will give you a compressed backup.

Related

bash script didn't work in sequence by running in cron

I run a few scripts 1 by 1
cat 001.sh
sh /home/mysqldom/da-cron/f_mysqldom_nrd/5_change_nrd_tld.sh
sh /home/mysqldom/da-cron/f_mysqldom_nrd/5_proxy_removed.sh
sh /home/mysqldom/da-cron/f_mysqldom_nrd/6_sync_nrd.sh
The last script wont work... if I run manually it work very well...
the script is
cat 6_sync_nrd.sh
source /home/mysqldom/da-cron/var.sh
cd /home/mysqldom/da-cron/f_mysqldom_nrd/
mysql -u mysqldom_fnrd -p$mysqldom_fnrd_password -D mysqldom_fnrd -e "UPDATE \`$yesterday\` SET sync='$yesterday';"
mysql -u mysqldom_fnrd -p$mysqldom_fnrd_password -D mysqldom_fnrd -e "DELETE FROM \`$yesterday\` WHERE domain_name = 'domain_name';"
sed s/change_database/$yesterday/g update.conf > $yesterday.conf
/usr/share/logstash/bin/logstash -f $yesterday.conf --path.data /var/lib/logstash108
rm -rf nohup.out
The 6 has to be run after 5
any idea whats worn in it

Bash cron job on hpanel not locating directory

I have the following code on cron job, it runs but the code does not really do what it supposed to. It does not create the directory plus is does not do anything in the code. Please help check if the way I pointed to the directory is wrong.
#!/bin/bash
NAMEDATE=`date +%F_%H-%M`_`whoami`
NAMEDATE2=`date `
mkdir ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE -m 0755
mysqldump -u u3811*****_boss -p"*******" u3811*****_data | gzip ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE/db.sql.gz
echo "This is the database backup for website.com on $NAMEDATE2" |
mailx -a ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE/db.sql.gz -s "website.com Database attached" -- mail#gmail.com
chmod -R 0644 ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE/*
exit 0
Your NAMEDATE variable needs to be modified a bit, as shown below, for more information about variables in bash see this link
NAMEDATE=$(date +%F_%H-%M"_"$(whoami))
When you issue the mkdir command you will need to pass the -p option to create the complete directory structure if it doesn't exists.
mkdir -p ~/home/u3811numbers/domains/website.com/public_html/cron/backup/files/$NAMEDATE -m 0755
Also, the ~ character on Linux based distributions is used as a shortcut for the home directory of the user that invokes it so, in the line below the result is /home//home/u3811*****/domains/website.com/public_html/cron/backup/files/2020-09-04_23-13_ you can read more about it in here
In you last command before the exit, you might need to pass a wildcard (*) to avoid removing the executable bit on the directory, see below
chmod -R 0644 ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE/
The final version of your script will look something like this.
#!/bin/bash
NAMEDATE=$(date +%F_%H-%M"_"$(whoami))
NAMEDATE2=date
mkdir -p ~/home/u3811******/domains/website.com/public_html/cron/backup/files/$NAMEDATE -m 0755
mysqldump -u u3811*****_boss -p"******" u3811*****_data | gzip > ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE/db.sql.gz
echo "This is the database backup for website.com on $NAMEDATE2" | mailx -a ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE/db.sql.gz -s "website.com Database attached" -- mail#gmail.com
chmod -R 0644 ~/home/u3811*****/domains/website.com/public_html/cron/backup/files/$NAMEDATE/*
To debug a bash script you can always pass the -x flag for more information take a look at this article

Adding shell if statement inside lftp

I'm trying to use SFTP to copy some files from one server to another, this task should run everyweek. The script I use :
HOST='sftp://my.server.com'
USER='user1'
PASSWORD='passwd'
DIR=$HOSTNAME
REMOTE_DIR='/home/remote'
LOCAL_DIR='/home/local'
# LFTP via SFTP connexion
lftp -u "$USER","$PASSWORD" $HOST <<EOF
# changing directory
cd "$REMOTE_DIR"
$(if [ ! -d "$DIR" ]; then
mkdir $DIR
fi)
put -O "$REMOTE_DIR"/$DIR "$LOCAL_DIR"/uploaded.txt
EOF
My issue is that put is executed without taking in consideration the result of if statment.
PS : The error message I got is the following :
put: Access failed: No such file (/home/backups/myhost/upload.txt)
LFTP has no if statement!
What you are doing here?
lftp -u "$USER","$PASSWORD" $HOST <<EOF
cd "$REMOTE_DIR"
$(if [ ! -d "$DIR" ]; then
mkdir $DIR
fi)
put -O "$REMOTE_DIR"/$DIR "$LOCAL_DIR"/uploaded.txt
EOF
You call a sub command in a here document. The sub command is executed locally before lftp is started and its output is pasted in the here document, which gets passed to lftp. This works just, because mkdir has no output. You do not call mkdir on the ftp server. You call the mkdir of your local shell. Effectively it is the same as if you put the if statement before the lftp execution.
if [ ! -d "$DIR" ]; then
mkdir $DIR
fi
lftp -u "$USER","$PASSWORD" $HOST <<EOF
cd "$REMOTE_DIR"
put -O "$REMOTE_DIR"/$DIR "$LOCAL_DIR"/uploaded.txt
EOF
What you are trying to do, does not work. You have to think about a different solution.
Right now I have no FTP server to test it, but it might be possible to use the -f option of LFTP's mkdir. I assume that it may work like the -f option of the Unix rm command. Try this:
lftp -u "$USER","$PASSWORD" $HOST <<EOF
cd "$REMOTE_DIR"
mkdir -f "$DIR"
put -O "$REMOTE_DIR"/$DIR "$LOCAL_DIR"/uploaded.txt
EOF
Update: It works as supposed. The creation of a directory, which exist already, throws no error, if you use the option -f:
lftp anonymous#localhost:/pub> mkdir -f dir
mkdir ok, `dir' created
lftp anonymous#localhost:/pub> mkdir -f dir
lftp anonymous#localhost:/pub> ls
drwx------ 2 116 122 4096 Aug 10 12:04 dir
Maybe you lftp client is outdated. I tested it with Debian 9.

Permission Issue: Creating postgres back as postgres user

I have following postgres backup script, its a shell script and written to run ans postgres user.
But the problem is postgres user doesn't have permission to write these directories. I as a user don't have sudo on these machines but I have changed the directory to has 755 and added to one of the group that has major permission to do read-write-execute. Since postgres user isn't part of the unix user group I guess I am running into this issue.
My goal is to put this in the cron-tab but prior to that I need to get the script running with proper permission:
#!/bin/bash
# location to store backups
backup_dir="/location/to/dir"
# name of the backup file has the date
backup_date=`date +%d-%m-%Y`
# only keep the backup for 30 days (maintain low storage)
number_of_days=30
databases=`psql -l -t | cut -d'|' -f1 | sed -e 's/ //g' -e '/^$/d'`
for i in $databases; do
if [ "$i" != "template0" ] && [ "$i" != "template1" ]; then
echo Dumping $i to $backup_dir$i\_$backup_date
pg_dump -Fc $i > $backup_dir$i\_$backup_date
fi
done
find $backup_dir -type f -prune -mtime +$number_of_days -exec rm -f {} \;
Before doing this be sure to login as a super user (sudo su) and try executing these:
useradd -G unix postgres (Add postgres user to unix group)
su postgres (Login as postgres user)
mkdir folder (Go to the directory where postgres needs to write files)
***From this line down is my answer to #find-missing-semicolon question
Just to illustrate an example with a shell script, you can capture the password using the read command and put it to a variable. Here I stored the password in password and echoed it afterwards. I hope this helps.
`#!/bin/bash`
`read -s -p "Password: " password`
`echo $password`

Downloading and automatically installing a tgz file

#!/bin/bash
mkdir /tmp
curl -O http://www.mucommander.com/download/nightly/mucommander-current.app.tar.gz /tmp/mucommander.tgz
tar -xvzf /tmp/mucommander.tgz */mucommander.app/*
cp -r /tmp/mucommander.app /Applications
rm -r /tmp
I'm trying to create a shell script to download and extract muCommander to my applications directory on a Mac.
I tried cd into the tmp dir, but then the script stops when I do that.
I can extract all using the -C argument, but the current tgz path is muCommander-0_9_0/mucommander.app, which could change on later builds, so I'm trying to keep it generic.
Can anyone give me pointers where I'm going wrong?
Thanks in advance.
Strip the first path component when you untar the archive, from tar(1):
--strip-components count
(x mode only) Remove the specified number of leading path ele-
ments. Pathnames with fewer elements will be silently skipped.
Note that the pathname is edited after checking inclusion/exclu-
sion patterns but before security checks.
Update
Here is a working bash example of how to, fairly generically, copy the contents of the tgz file to /Applications.
shopt -s nocaseglob
TMPDIR=/tmp
APP=mucommander
TMPAPPDIR=$TMPDIR/$APP
mkdir -p $TMPAPPDIR
curl -o $TMPDIR/$APP.tgz http://www.mucommander.com/download/nightly/mucommander-current.app.tar.gz
tar --strip-components=1 -xvzf $APP.tgz -C $TMPAPPDIR
mv $TMPAPPDIR/${APP}* /Applications
# rm -rf $TMPAPPDIR $TMPDIR/$APP
The rm command is commented out for now, verify that it does no harm before you use it.
The following will update your muCommander.
#for the safety, remove old temporary extraction from the /tmp
rm -rf /tmp/muCommander.app
#kill the running mucommander - you dont want replace the runnung app
ps -ef | grep ' /Applications/muCommander.app/' | grep -v grep | awk '{print $2}' | xargs kill
#download, extract, remove old, move new, open
#each command run only when the previous ended with success
curl http://www.mucommander.com/download/nightly/mucommander-current.app.tar.gz |\
tar -xzf - -C /tmp --strip-components=1 '*/muCommander.app' && \
rm -rf /Applications/muCommander.app && \
mv /tmp/muCommander.app /Applications && \
open /Applications/muCommander.app
Beware, after the '\' must following new line, and not any spaces...

Resources