I am trying to create upload.sh that lets me upload multiple files via SFTP doing something like this:
upload.sh file1 file2 file3
upload.sh subfolder/*
In each case, the user should be prompted to enter the remoteFS password.
It might look like this:
ls -1 $* > tmp.txt
sftp -b tmp.txt root#1.2.3.4:/remotefs/path/to/html/
The above isn't correct, because tmp.txt would contain a list of filenames, and it should be containing a list of sftp put foo/file.bar commands.
But even if I get that working (at the moment I'm simulating by manually creating a tmp.txt file), I am running into this problem: How to send password using sftp batch file
i.e. Rather than sftp requesting a password, it is just failing.
Is there any way to force it to request a password?
Storing the password inside the .sh would be bad practice, and I don't want to involve SSH keys at this point (I'm using this scenario to demonstrate shell-scripting to a maths student, and I don't want to escalate the complexity).
Try this:
#!/bin/bash
[[ ${#} -eq 0 ]] && exit 1
(
echo "cd /remotefs/path/to/html";
for file in "$#"; do
echo "put '$file'"
done
) | sftp root#1.2.3.4
Related
I am currently using the below command in a ,sh to gather a list of contents of a specific folder, on a list of servers, depicted by list.txt (contains IPs)
for f in `cat serverlist.txt`; do
echo "### $f ###";
sshpass -p PASSWORD ssh USER#$f ls /usr/local/folder >>list.txt;
done
Whilst this works, its only half of my problem, I am a total novice with BASH
What I am trying to obtain is a list formatted as such
file1.HOSTNAMEOFSERVER1
file2.HOSTNAMEOFSERVER1
file3.HOSTNAMEOFSERVER1
file1.HOSTNAMEOFNEXTSERVER2
file2.HOSTNAMEOFNEXTSERVER2
file3.HOSTNAMEOFNEXTSERVER2
file1.HOSTNAMEOFNEXTSERVER3
Is any one able to help?
Untested:
while read host; do
sshpass -p PASSWORD ssh USER#"$host" ls /usr/local/folder |
sed 's/$/.'"$host"/;
done < serverlist.txt
DO NOT ACTUALLY PUT YOUR PASSWORD in a script like this. Set up your ssh keys instead.
Just format the ls output into columns with the option -C1: ls -C1 /usr/local/folder.
I have file containing a list of files separated by end of lines
$ cat file_list
file1
file2
file3
I want to copy this list of files with FTP
How can I do that ? Do I have to write a script ?
You can turn your list of files into list of ftp commands easily enough:
(echo open hostname.host;
echo user username;
cat filelist | awk '{ print "put " $1; }';
echo bye) > script.ftp
Then you can just run:
ftp -s script.ftp
Or possibly (with other versions of ftp)
ftp -n < script.ftp
Something along these lines - the somecommand depends on what you want to do - I don't get that from your question, sorry.
#!/bin/bash
# Iterate through lines in file
for line in `cat file.txt`;do
#your ftp command here do something
somecommand $line
done
edit: If you really want to persue this route for multiple files (you shouldn't!), you can use the following command in place of somecommand $line:
ncftpput -m -u username -p password ftp.server.com /remote/folder $line
ncftpput propably also takes an arbitrary number of files to upload in one go, but I havn't checked it. Notice that this approach will connect and disconnect for every single file!
Thanks for the very helpful example of how to feed a list of files to ftp. This worked beautifully for me.
After creating my ftp script in Linux (CentOs 5.5), I ran the script with:
ftp –n < ../script.ftp
My script (with names changed to protect the innocent) starts with:
open <ftpsite>
user <userid> <passwd>
cd <remote directory>
bin
prompt
get <file1>
get <file2>
And ends with:
get <filen-1>
get <filen>
bye
The script uses scp to upload a file. That works.
Now I want to log in with ssh, cd to the directory that holds the uploaded file, do an md5sum on the file. The script keeps telling me that md5sum cannot find $LOCAL_FILE. I tried escaping: \$LOCAL_FILE. Tried quoting the EOI: <<'EOI'. I'm partially understanding this, that no escaping means everything happens locally. echo pwd unescaped gives the local path. But why can I do "echo $MD5SUM > $LOCAL_FILE.md5sum", and it creates the file on the remote machine, yet "echo md5sum $LOCAL_FILE > md5sum2" does not work? And if it the local md5sum, how do I tell it to work on the remote?
scp "files/$LOCAL_FILE" "$i#$i.567.net":"$REMOTE_FILE_PATH"
ssh -T "$i#$i.567.net" <<EOI
touch I_just_logged_in
cd $REMOTE_DIRECTORY_PATH
echo `date` > I_just_changed_directories
echo `whoami` >> I_just_changed_directories
echo `pwd` >> I_just_changed_directories
echo "$MD5SUM" >> I_just_changed_directories
echo $MD5SUM > $LOCAL_FILE.md5sum
echo `md5sum $LOCAL_FILE` > md5sum2
EOI
You have to think about when $LOCAL_FILE is being interpreted. In this case, since you've used double-quotes, it's being interpreted on the sending machine. You need instead to quote the string in such a way that $LOCAL_FILE is in the command line on the receiving machine. You also need to get your "here document" correct. What you show just sends the output to touch to the ssh.
What you need will look something like
ssh -T address <'EOF'
cd $REMOTE_DIRECTORY_PATH
...
EOF
The quoting rules in bash are somewhat arcane. You might want to read up on them in Mendel Cooper's Advanced Guide to Bash Scripting.
I am writing a bash script to copy a file from a remote server, to my local machine. I need to check to see if the file is available, so I can take an alternative action if it is not there.
I know how to test if a local file exists, however, using scp complicates things a bit. Common sense tells me that one way would be to try to scp the file anyhow, and check the return code from the scp command. Is this the correct way to go about it?
If yes, how do I test the return code from the scp invocation?
using ssh + some shell code embedded in the cmd line; use this method when you need to take a decision before the file transfer will fail;
ssh remote-host 'sh -c "if [ -f ~/myfile ] ; then gzip -c ~/myfile ; fi" ' | gzip -dc > /tmp/pkparse.py
if you want to transfer directories you may want to "tar"-it first
if you want to use scp you can check the return code like this:
if scp remote-host:~/myfile ./ >&/dev/null ; then echo "transfer OK" ; else echo "transfer failed" ; fi
it really depends on when its important for you to know if the file is there or not; before the transfer starts (use ssh+sh) or after its finished.
well, since you can use scp you can try using ssh to list and see if the file is their or not before proceeding.
I'm trying to write a Bash script that uploads a file to a server. How can I achieve this? Is a Bash script the right thing to use for this?
Below are two answers. First is a suggestion to use a more secure/flexible solution like ssh/scp/sftp. Second is an explanation of how to run ftp in batch mode.
A secure solution:
You really should use SSH/SCP/SFTP for this rather than FTP. SSH/SCP have the benefits of being more secure and working with public/private keys which allows it to run without a username or password.
You can send a single file:
scp <file to upload> <username>#<hostname>:<destination path>
Or a whole directory:
scp -r <directory to upload> <username>#<hostname>:<destination path>
For more details on setting up keys and moving files to the server with RSYNC, which is useful if you have a lot of files to move, or if you sometimes get just one new file among a set of random files, take a look at:
http://troy.jdmz.net/rsync/index.html
You can also execute a single command after sshing into a server:
From man ssh
ssh [...snipped...] hostname [command] If command is specified, it is
executed on the remote host instead of a login shell.
So, an example command is:
ssh username#hostname.example bunzip file_just_sent.bz2
If you can use SFTP with keys to gain the benefit of a secured connection, there are two tricks I've used to execute commands.
First, you can pass commands using echo and pipe
echo "put files*.xml" | sftp -p -i ~/.ssh/key_name username#hostname.example
You can also use a batchfile with the -b parameter:
sftp -b batchfile.txt ~/.ssh/key_name username#hostname.example
An FTP solution, if you really need it:
If you understand that FTP is insecure and more limited and you really really want to script it...
There's a great article on this at http://www.stratigery.com/scripting.ftp.html
#!/bin/sh
HOST='ftp.example.com'
USER='yourid'
PASSWD='yourpw'
FILE='file.txt'
ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
binary
put $FILE
quit
END_SCRIPT
exit 0
The -n to ftp ensures that the command won't try to get the password from the current terminal. The other fancy part is the use of a heredoc: the <<END_SCRIPT starts the heredoc and then that exact same END_SCRIPT on the beginning of the line by itself ends the heredoc. The binary command will set it to binary mode which helps if you are transferring something other than a text file.
You can use a heredoc to do this, e.g.
ftp -n $Server <<End-Of-Session
# -n option disables auto-logon
user anonymous "$Password"
binary
cd $Directory
put "$Filename.lsm"
put "$Filename.tar.gz"
bye
End-Of-Session
so the ftp process is fed on standard input with everything up to End-Of-Session. It is a useful tip for spawning any process, not just ftp! Note that this saves spawning a separate process (echo, cat, etc.). It is not a major resource saving, but it is worth bearing in mind.
The ftp command isn't designed for scripts, so controlling it is awkward, and getting its exit status is even more awkward.
Curl is made to be scriptable, and also has the merit that you can easily switch to other protocols later by just modifying the URL. If you put your FTP credentials in your .netrc, you can simply do:
# Download file
curl --netrc --remote-name ftp://ftp.example.com/file.bin
# Upload file
curl --netrc --upload-file file.bin ftp://ftp.example.com/
If you must, you can specify username and password directly on the command line using --user username:password instead of --netrc.
Install ncftpput and ncftpget. They're usually part of the same package.
Use this to upload a file to a remote location:
#!/bin/bash
#$1 is the file name
#usage:this_script <filename>
HOST='your host'
USER="your user"
PASSWD="pass"
FILE="abc.php"
REMOTEPATH='/html'
ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
cd $REMOTEPATH
put $FILE
quit
END_SCRIPT
exit 0
The command in one line:
ftp -in -u ftp://username:password#servername/path/to/ localfile
#/bin/bash
# $1 is the file name
# usage: this_script <filename>
IP_address="xx.xxx.xx.xx"
username="username"
domain=my.ftp.domain
password=password
echo "
verbose
open $IP_address
USER $username $password
put $1
bye
" | ftp -n > ftp_$$.log
Working example to put your file on root...see, it's very simple:
#!/bin/sh
HOST='ftp.users.qwest.net'
USER='yourid'
PASSWD='yourpw'
FILE='file.txt'
ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
put $FILE
quit
END_SCRIPT
exit 0
There isn't any need to complicate stuff. This should work:
#/bin/bash
echo "
verbose
open ftp.mydomain.net
user myusername mypassword
ascii
put textfile1
put textfile2
bin
put binaryfile1
put binaryfile2
bye
" | ftp -n > ftp_$$.log
Or you can use mput if you have many files...
If you want to use it inside a 'for' to copy the last generated files for an everyday backup...
j=0
var="`find /backup/path/ -name 'something*' -type f -mtime -1`"
# We have some files in $var with last day change date
for i in $var
do
j=$(( $j + 1 ))
dirname="`dirname $i`"
filename="`basename $i`"
/usr/bin/ftp -in >> /tmp/ftp.good 2>> /tmp/ftp.bad << EOF
open 123.456.789.012
user user_name passwd
bin
lcd $dirname
put $filename
quit
EOF # End of ftp
done # End of 'for' iteration
echo -e "open <ftp.hostname>\nuser <username> <password>\nbinary\nmkdir New_Folder\nquit" | ftp -nv