I'm running CentOS 6.
I need to upload some files every hour to another server.
I have SSH access with password to the server. But ssh-keys etc. is not an option.
Can anyone help me out with a .sh script that uploads the files via scp and delete the original after a successful upload?
For this, I'd suggest to use rsync rather than scp, as it is far more powerful. Just put the following in an executable script. Here, I assume that all the files (and nothing more) is in the directory pointed to by local_dir/.
#!/bin/env bash
rsync -azrp --progress --password-file=path_to_file_with_password \
local_dir/ remote_user#remote_host:/absolute_path_to_remote_dir/
if [ $? -ne 0 ] ; then
echo "Something went wrong: don't delete local files."
else
rm -r local_dir/
fi
The options are as follows (for more info, see, e.g., http://ss64.com/bash/rsync.html):
-a, --archive Archive mode
-z, --compress Compress file data during the transfer
-r, --recursive recurse into directories
-p, --perms Preserve permissions
--progress Show progress during transfer
--password-file=FILE Get password from FILE
--delete-after Receiver deletes after transfer, not during
Edit: removed --delete-after, since that's not the OP's intent
Be careful when setting the permissions for the file containing the password. Ideally only you should have access tot he file.
As usual, I'd recommend to play a bit with rsync in order to get familiar with it. It is best to check the return value of rsync (using $?) before deleting the local files.
More information about rsync: http://linux.about.com/library/cmd/blcmdl1_rsync.htm
Related
I'm trying to rsync a large directory of around 200 GB from a server to my local external hard drive. I can ssh onto the server and see the directory fine. I can also cd into the external hard drive fine. When I try and rsync the file across, I don't get an error, but the last line of the rsync output is 'total size is 0 speedup is 0.00', and there are no files in the destination directory.
Here's how I ssh onto the server successfully:
ssh skgtmdf#live.rd.ucl.ac.uk
Here's my rsync command:
rsync -avrt -progress -e "ssh skgtmdf#live.rd.ucl.ac.uk:/mnt/gpfs/live/rd01__/ritd-ag-project-rd012x-smmca23/" "/Volumes/DUAL DRIVE/ONP/2022.08.10_results/"
And here's the rsync output:
sending incremental file list
drwxrwxrwx 65,536 2022/08/10 21:32:06 .
sent 57 bytes received 64 bytes 242.00 bytes/sec
total size is 0 speedup is 0.00
What am I doing wrong?
The way you have it quoted, the source path is part of the remote shell option (-e value) rather than a separate argument as it should be.
rsync -avrt -progress -e "ssh skgtmdf#live.rd.ucl.ac.uk:/mnt/gpfs/live/rd01__/ritd-ag-project-rd012x-smmca23/" "/Volumes/DUAL DRIVE/ONP/2022.08.10_results/"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This is all part of the `-e` option value
This means rsync doesn't see that as a sync source at all, but just part of the command it'll use to connect to the remote system. I'm not sure why this doesn't lead to an error. In any case, the fix is simple: don't include ssh with the source path.
As I noticed later (see comments) the --progress option needs a double-dash or it'll be wildly misparsed. Fixing both of these things gives:
rsync -avrt --progress -e ssh "skgtmdf#live.rd.ucl.ac.uk:/mnt/gpfs/live/rd01__/ritd-ag-project-rd012x-smmca23/" "/Volumes/DUAL DRIVE/ONP/2022.08.10_results/"
In fact, since ssh is the default command for making a remote connection, you can leave off -e ssh entirely:
rsync -avrt --progress "skgtmdf#live.rd.ucl.ac.uk:/mnt/gpfs/live/rd01__/ritd-ag-project-rd012x-smmca23/" "/Volumes/DUAL DRIVE/ONP/2022.08.10_results/"
rsync -azve ssh user#host:/src/ target/
Normally, you don't need to wrap -e flag with ". It's probably messing the connection string
This is my first post, and actually first steps into bash.
I wanted to create automated ftp server backup feature to make my life easier. Since I've felt good doing this script I've started working on improvements for it and currently I ran out of ideas.
I wanted to make script backup download all files from ftp server and defined credentials as variables (which I am proud of lol) but then I've realized that I can use same script for all my servers but I will have to store login credentials in separate file.
The problem is that I can't make my script to read this values for backup processing and they remain blank as I see from output.
If anyone have any ideas how can I make it read my credentials and server name from separate file I would be grateful!
Below you can find my files and scripts in order that they are being processed:
'start_backup.sh'
#!/bin/sh
# Load credentials
/home/user/scripts/credentials.sh
# Launch backup script
/home/user/scripts/my_ovh_backup.sh
credentials.sh
#!/bin/sh
my_ovh_server="(server address goes here)"
my_ovh_login="(login goes here)"
my_ovh_password="(password goes here)"
my_ovh_backup.sh
# Path that need to be copied
backup_files="/(path on server)"
# Where to backup
dest="/home/user/ftp_backup"
data=$(date +%y-%m-%d)
# Create folder with current date
mkdir $dest/in_progress_$data
# Copy data
cd $dest/in_progress_$data
wget -r -nc ftp://$my_ovh_login:$my_ovh_password#$my_ovh_server/$backup_files
# Rename folder after download completion
mv $dest/in_progress_$data $dest/$data
# Cleanup
rm -r $dest/$data
Output that I recieve:
./start_backup.sh
ftp://:#//(remote folder)/: Invalid host name.
adding: home/user/ftp_backup/22-01-02/ (stored 0%)
I have set of 10 files each can be of a varied size ranging from 1mb to 10mb. I want to transfer these files to the remote server via SFTP key based authentication (as per the requirement). I have written a simple shell script to pick the files from local directory and connect to remote server and then put all the files.
I want to know whether there is a way to check below
File transfer failed in between(out of 10 files, 5 got transferred and 5 didn't).
Transfer of partial files.
Aborting the script when the transfer is happening.
Sample code :
cd local_directory
sftp -i privatekey username#ip_address << EOF 2>> TMP_LOG
cd /data
pwd
put *
bye
EOF
if [[ $? != 0 ]]
then
echo "Failure"
else
echo "fine"
fi
But this doesn't seem to be working fine:
When script is aborted.
Transfer is partial.
SFTP connection getting lost.
Any suggestion on this please?
Yeah, sftp isn't working very well with exit status reporting.
You can use diff to find if there is any differences, by default, diff gives exit status 1 if there is a difference, which means that you can use it in script.
To compare local file and remote file with diff:
ssh user_name#remotehost ls -R /remote/dir > remotedirfiles.txt
ls -R /local/dir > localdirfiles.txt
diff remotedirfiles.txt localdirfiles.txt
And if ssh prompt you for password, that will not work, you will need to redirect both results to text files and compare them with diff.
I'm trying to automate the download of a subdirectories in a directory. However, when executing the script, the last one or two directories cannot be found by the script - "No such file or directory". All others do fine and can be downloaded. This occurs for all directories I've tried this on which is strange to me. Why would it always not find the last two directories?
Can anyone help with this? Is it due to the loop? I've tried changing it to loop over the last ones only and this doesn't help. Or maybe it's due to the conversion of array=($l)?
Here's my script:
dirServer=/dir/to/location/in/server
dirLocal=/dir/to/location/in/local/pc
l=`ssh -t username#server 'ls' ${dirServer}`
#array of folders that should be copied to local machine
array=($l)
for folder in ${array[#]}
do
echo ${folder}
#if directory doesn't exist, creat it
mkdir -p ${dirLocal}${folder}
scp -r username#server:${dirServer}${folder}/analysis/ ${dirLocal}${folder}
done
Try change the following line including this "-o LogLevel=QUIET"
l=`ssh -o LogLevel=QUIET -t username#server 'ls' ${dirServer}`
Explanation:
That is coming from SSH. You see it because you gave the -t switch, which forces SSH to allocate a pseudo-terminal for the connection. Traditionally, SSH displays that message to make it clear that you are no longer interacting with the shell on the remote host, which is normally only a question when SSH has a pseudo-terminal allocated.
I am new to FTP configuration. What I am trying to do is as follows:
I am running a shell script on my localhost and downloading some files to my machine. Now I want a functionality where the files which I downloaded should be stored in a temporary directory, and then it should be transferred to a location(other directory) which I specify. I feel this mechanism is achievable by FTP communication and will be helpful when I host this on a domain, but I am not getting resources from where I can teach myself how to set this up.
OK, having visited many sites, here are some resources you might find handy:
For configuring vsftpd, here's a manual of how to install, configure and use.
About receiving many files recursively via FTP, you can use wget (extracted from this site):
cd /tmp/ftptransfer
wget --mirror --username=foo --password=bar ftp://ftp.originsite.com/path/to/folder
About sending many files recursively, many people find the only way of doing so by tar-n-send; the only problem is that the files will remain tarred until you extract them by going to the other machine (remotely or via ssh) to extract the manually. There is an alternative, not using FTP, but using ssh and pipes which lets you have files extracted on target machine:
tar -cf - /tmp/ftptransfer | ssh geek#targetsite "cd target_dir; tar -xf -"
Explained:
tar is the application to make tar files
-c: create file
-f -: file name is "stdout"
/tmp/ftptransfer: include this folder and all subdirectories in the tar
|: Make a pipe to the next program (connect stdout to stdin)
ssh: Secure Shell program
geek#targetsite: username # machinename where you want to connect to
"..." command to send to the remote host
cd target_dir: changes the dir of output
tar -xf -: extracts the file received by "stdin"
For configuring SSH on Ubuntu, have a look here.
If you need more help, don't be afraid to ask! :)