I would like to ask you for a help. I'm a scripting noob and I need to make a script for backing up our webserver via FTPS. I made this script, but it only makes the backup file, but it does not upload that file to the FTPS server. But when I run that lftp command alone, it works. I'm looking at this for quite a long time, but I can't find out, why it's not working... Could someone help please? Thank you!
#!/bin/bash
# SETTINGS
RMDATE=$(date --iso -d '10 days ago').tar
FTPUSER=user
FTPPW=pass
FTPSERVER=my.server.com
LFTP=/usr/bin/lftp
# DELETE OLD BACKUPS
rmold () {
$LFTP << EOF
open ${FTPUSER}:${FTPPW}#${FTPSERVER}
rm -rf ${RMDATE}
bye
EOF
echo "Done."
}
# PLESK BACKUP
if /usr/local/psa/bin/pleskbackup server -v --exclude-logs >/tmp/backup-plesk.log 2>&1 --output-file=/var/www/bak/`date -I`.tar; then
if lftp -c "open ${FTPUSER}:${FTPPW}#${FTPSERVER}; put /var/www/bak/`date -I`.tar"; then
rm -f /var/www/bak/`date -I`.tar
/usr/bin/sendEmail <<< some parameters >>> # backup success message
rmold
else
/usr/bin/sendEmail <<< some parameters >>> # upload error message
exit 1
fi
else
/usr/bin/sendEmail <<< some parameters >>> # backup error message
exit 1
fi
I had a problem with FTP FEAT and certificate. I disabled FEAT in lftp config file and I specified full path to the certificate. It works now.
Related
I'm attempting to write a bash script that I can set as a cronjob to automatically upload a backup file via SFTP to a remote server.
The backup files on the local server are datestamped (e.g. backup-file-YYYY-mm-dd.tar.gz) and I'd like the script to only upload a file from the directory that has the same datestamp as the current date.
Any ideas on where I'm going wrong? I can't help but think I'm missing something basic but I can't think what it is!
Current broken script below:
#!/bin/bash
FILE=$backups/$(date+%Y-%m-%d).tar.gz *<<<<< I'm guessing this is where it's slipping up*
sshpass -p "remoteserverpassword" sftp -o StrictHostKeyChecking=no <user>#<remoteserverip)
cd /directory1/directory2/
put $FILE
exit 0
EOF
You are right about where it is slipping up, date needs to be eval'd prior to passing to here script. Reformatted for clarity but you could plug into the original script too.
#!/bin/bash
backup=/tmp
today=`date +%Y-%m-%d`
FILE=$backup/$today.tar.gz
sshpass -p "remoteserverpassword" sftp -o StrictHostKeyChecking=no <user>#<remoteserverip) <<EOF
cd /directory1/directory2/
put $FILE
exit 0
EOF
I am a new at this so please be patiant.
I have a script that I did not write (not that advance, maybe one day) and I need it to output logs in var/logs.
Can anyone please help wiht it please?
I need to know that all the actions are completed in that script.
Here is the script:
#!/bin/bash
# copy the old database to a timestamped copy
TIMESTAMP=$(date +%d)
REPORT_EMAIL=user#domain.com
# backup Spamassassin bayes db
sa-learn -p /usr/mailcleaner/etc/mailscanner/spam.assassin.prefs.conf --siteconfigpath /usr/mailcleaner/share/spamassassin --backup >/var/mailcleaner/spool/spamassassin/spamass_rules.bak
# backup Bogofilter bayes db
cp -a /root/.bogofilter/wordlist.db "/root/.bogofilter/$TIMESTAMP-wordlist.db"
if [ -f "/root/.bogofilter/$TIMESTAMP-wordlist.db.gz" ]
then
rm -f "/root/.bogofilter/$TIMESTAMP-wordlist.db.gz"
fi
gzip "/root/.bogofilter/$TIMESTAMP-wordlist.db"
# get the spam and ham from the imap mailbox for the spamassassin and bogofilter db's
/opt/mailcleaner/scripts/imap-sa-learn.pl
if [ $? -ne 0 ]
then
(
echo "Subject: Bogofilter database update $(hostname) failed"
ls -l /var/mailcleaner/spool/bogofilter/database/
) | /usr/sbin/sendmail $REPORT_EMAIL
exit 1
fi
# copy the database to the right location
cp /root/.bogofilter/wordlist.db /var/mailcleaner/spool/bogofilter/database/wordlist.db
# If slave(s) Mailcleaner exists, ssh copy dbs to the slave(s)
#scp /root/.bogofilter/wordlist.db mailcleaner2.domain.com:/var/mailcleaner/spool/bogofilter/database/
#scp /var/mailcleaner/spool/spamassassin/spamass_rules.bak mailcleaner2.domain.com:/var/mailcleaner/spool/spamassassin/
# get the spam and ham counts from bogofilter - this just prints how many spam and ham you collected so far...
/opt/bogofilter/bin/bogoutil -w /var/mailcleaner/spool/bogofilter/database/wordlist.db .MSG_COUNT
Again thanks for the help.
Raj
It's easiest just to use output redirection when you run the script:
./the_script.sh > /var/log/the_script.log 2>&1
The 2>&1 is to capture stderr (the error stream) as well as stdout (the output stream).
If you want to write a log file and see the output on the console, use tee:
./the_script.sh 2>&1 | tee /var/log/the_script.log
I want to create a directory if it doesn't exists after login to sftp server.
test.sh
sftp name#example.com << EOF
mkdir test
put test.xml
bye
EOF
Now i call test.sh and upload different files each time to test folder. When running this
mkdir test
First time it works and second time it throws Couldn't create directory: Failure error?
How to create a directory if doesn't exists and if exists don't create directory in sftp.
man 1 sftp (from openssh-client package):
-b batchfile
Batch mode reads a series of commands from an input
batchfile instead of stdin. Since it lacks user
interaction it should be used in conjunction with
non-interactive authentication. A batchfile of ‘-’
may be used to indicate standard input. sftp will
abort if any of the following commands fail: get,
put, reget, reput, rename, ln, rm, mkdir, chdir, ls,
lchdir, chmod, chown, chgrp, lpwd, df, symlink, and
lmkdir. Termination on error can be suppressed on a
command by command basis by prefixing the command
with a ‘-’ character (for example, -rm /tmp/blah*).
So:
{
echo -mkdir dir1
echo -mkdir dir1/dir2
echo -mkdir dir1/dir2/dir3
} | sftp -b - $user#$host
I understand this thread is old and has been marked as answered but the answer did not work in my case. The second page on google for a search regarding "sftp checking for directory" so here is an update that would have saved me a few hours.
Using an EOT you cannot capture the error code resulting from the directory not being found. The work around I found was to create a file containing instructions for the call and then capture the result of that automated call.
The example below using sshpass but my script also uses this same method authenticating with sshkeys.
Create the file containing the instructions:
echo "cd $RemoteDir" > check4directory
cat check4directory; echo "bye" >> check4directory
Set permissions:
chmod +x check4directory
Then make the connection using the batch feature:
export SSHPAA=$remote_pass
sshpass -e sftp -v -oBatchMode=no -b check4directory $remote_user#$remote_addy
Lastly check for the error code:
if [ $? -ge "1" ] ; then
echo -e "The remote directory was not found or the connection failed."
fi
At this point you can exit 1 or initiate some other action. Note that if the SFTP connection fails for another reason like password or the address is incorrect the error will trip the action.
Another variant is to split the SFTP session into two.
First SFTP session simply issues the MKDIR command.
Second SFTP session can then assume existence of the directory and put the files.
You can use the SSH access of your account to first verify if the directory exists at all (using the "test" command). If it returns exit code 0, the dir exists, otherwise it doesn't. You can act on that accordingly.
# Both the command and the name of your directory are "test"
# To avoid confusion, I just put the directory in a separate variable
YOURDIR="test"
# Check if the folder exists remotely
ssh name#example.com "test -d $YOURDIR"
if [ $? -ne 0 ]; then
# Directory does not exist
sftp name#example.com << EOF
mkdir test
put test.xml
bye
EOF
else
# Directory already exists
sftp name#example.com << EOF
put test.xml
bye
EOF
fi
Try this to ignore errors if directory already exists.
# Turn OFF error
set +e
# Create remote dirs
sftp -P 22 -o StrictHostKeyChecking=no -oIdentityFile=key.pem -v $user#$host <<EOF
mkdir <remote_path> # create remote directory
bye
EOF
# Turn ON error
set -e
# Do upload to SFTP
sftp -P 22 -o StrictHostKeyChecking=no -oIdentityFile=key.pem -v $user#$host <<EOF
cd <remote_path> # remote_path
put <local_file_path> # local_path
quit
EOF
I'm writing a bash script to send files from a linux server to a remote Windows FTP server.
I would like to check using FTP if the folder where the file will be stored exists before attempting to create it.
Please note that I cannot use SSH nor SCP and I cannot install new scripts on the linux server. Also, for performance issues, I would prefer if checking and creating the folders is done using only one FTP connection.
Here's the function to send the file:
sendFile() {
ftp -n $FTP_HOST <<! >> ${LOCAL_LOG}
quote USER ${FTP_USER}
quote PASS ${FTP_PASS}
binary
$(ftp_mkdir_loop "$FTP_PATH")
put ${FILE_PATH} ${FTP_PATH}/${FILENAME}
bye
!
}
And here's what ftp_mkdir_loop looks like:
ftp_mkdir_loop() {
local r
local a
r="$#"
while [[ "$r" != "$a" ]]; do
a=${r%%/*}
echo "mkdir $a"
echo "cd $a"
r=${r#*/}
done
}
The ftp_mkdir_loop function helps in creating all the folders in $FTP_PATH (Since I cannot do mkdir -p $FTP_PATH through FTP).
Overall my script works but is not "clean"; this is what I'm getting in my log file after the execution of the script (yes, $FTP_PATH is composed of 5 existing directories):
(directory-name) Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
To solve this, do as follows:
To ensure that you only use one FTP connection, you create the input (FTP commands) as an output of a shell script
E.g.
$ cat a.sh
cd /home/test1
mkdir /home/test1/test2
$ ./a.sh | ftp $Your_login_and_server > /your/log 2>&1
To allow the FTP to test if a directory exists, you use the fact that "DIR" command has an option to write to file
# ...continuing a.sh
# In a loop, $CURRENT_DIR is the next subdirectory to check-or-create
echo "DIR $CURRENT_DIR $local_output_file"
sleep 5 # to leave time for the file to be created
if (! -s $local_output_file)
then
echo "mkdir $CURRENT_DIR"
endif
Please note that "-s" test is not necessarily correct - I don't have acccess to ftp now and don't know what the exact output of running DIR on non-existing directory will be - cold be empty file, could be a specific error. If error, you can grep the error text in $local_output_file
Now, wrap the step #2 into a loop over your individual subdirectories in a.sh
#!/bin/bash
FTP_HOST=prep.ai.mit.edu
FTP_USER=anonymous
FTP_PASS=foobar#example.com
DIRECTORY=/foo # /foo does not exist, /pub exists
LOCAL_LOG=/tmp/foo.log
ERROR="Failed to change directory"
ftp -n $FTP_HOST << EOF | tee -a ${LOCAL_LOG} | grep -q "${ERROR}"
quote USER ${FTP_USER}
quote pass ${FTP_PASS}
cd ${DIRECTORY}
EOF
if [[ "${PIPESTATUS[2]}" -eq 1 ]]; then
echo ${DIRECTORY} exists
else
echo ${DIRECTORY} does not exist
fi
Output:
/foo does not exist
If you want to suppress only the messages in ${LOCAL_LOG}:
ftp -n $FTP_HOST <<! | grep -v "Cannot create a file" >> ${LOCAL_LOG}
this topic has been discussed at length, however, I have a variant on the theme that I just cannot crack. Two days into this now and decided to ping the community. THx in advance for reading..
Exec. summary is I have a script in OS X that runs fine and executes without issue or error when done manually. When I put the script in the crontab to run daily it still runs but it doesnt run all of the commands (specifically SFTP).
I have read enough posts to go down the path of environment issues, so as you will see below, I hard referenced the location of the SFTP in the event of a PATH issue...
The only thing that I can think of is the IdentityFile. NOTE: I am putting this in the crontab for my user not root. So I understand that it should pickup on the id_dsa.pub that I have created (and that has already been shared with the server)..
I am not trying to do any funky expect commands to bypass the password, etc. I dont know why when run from the cron that it is skipping the SFTP line.
please see the code below.. and help is greatly appreciated.. thx
#!/bin/bash
export DATE=`date +%y%m%d%H%M%S`
export YYMMDD=`date +%y%m%d`
PDATE=$DATE
YDATE=$YYMMDD
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
FEED="~/Dropbox/"
USER="user"
HOST="host.domain.tld"
A="/tmp/5nPR45bH"
>${A}.file1${PDATE}
>${A}.file2${PDATE}
BYEbye ()
{
rm ${A}.file1${PDATE}
rm ${A}.file2${PDATE}
echo "Finished cleaning internal logs"
exit 0
}
echo "get -r *" >> ${A}.file1${PDATE}
echo "quit" >> ${A}.file1${PDATE}
eval mkdir ${FEED}${YDATE}
eval cd ${FEED}${YDATE}
eval /usr/bin/sftp -b ${A}.file1${PDATE} ${USER}#${HOST}
BYEbye
exit 0
Not an answer, just comments about your code.
The way to handle filenames with spaces is to quote the variable: "$var" -- eval is not the way to go. Get into the habit of quoting all variables unless you specifically want to use the side effects of not quoting.
you don't need to export your variables unless there's a command you call that expects to see them in the environment.
you don't need to call date twice because the YYMMDD value is a substring of the DATE: YYMMDD="${DATE:0:6}"
just a preference: I use $HOME over ~ in a script.
you never use the "file2" temp file -- why do you create it?
since your sftp batch file is pretty simple, you don't really need a file for it:
printf "%s\n" "get -r *" "quit" | sftp -b - "$USER#$HOST"
Here's a rewrite, shortened considerably:
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
FEED_DIR="$HOME/Dropbox/$(date +%Y%m%d)"
USER="user"
HOST="host.domain.tld"
mkdir "$FEED_DIR" || { echo "could not mkdir $FEED_DIR"; exit 1; }
cd "$FEED_DIR"
{
echo "get -r *"
echo quit
} |
sftp -b - "${USER}#${HOST}"