I have a script which will run on remote servers,
df_command.sh:-
#!/bin/bash
if [[ $1 == "" ]]; then
echo -e "No Argument passed:- Showing default disk usage\n"
df -k > /tmp/Global_df_tmp 2>&1
cat /tmp/Global_df_tmp
else
df -k "$1" > /tmp/Global_df_tmp 2>&1
cat /tmp/Global_df_tmp
fi
This is how i run this script:-
$ssh -oConnectTimeout=5 -oBatchMode=yes -l group servername 'bash -s -- /some/directory' < ./df_command.sh
This works fine and gives me correct output in every scenario, means if user passes any valid directory it gives me disk usage of that directory and if he/she passes some invalid directory the script gives me the proper error message back.
Problem rises when more than one user starts using the script at the same time for the same server and passes two different directories
e.g
User A:-
$ssh -oConnectTimeout=5 -oBatchMode=yes -l group servername 'bash -s -- /user_A/directory' < ./df_command.sh
user B:-
$ssh -oConnectTimeout=5 -oBatchMode=yes -l group servername 'bash -s -- /user_B/directory' < ./df_command.sh
Now since the tmp file for the script is same(/tmp/Global_df_tmp), whoever starts the script first will get the correct output while the second user will get the same output as first user got.
I know one solution would be to generate random number and use that instead of hardcoded tmp file, but if 100 users will use the same script then i'll end up having huge number of temporary files on the remote servers
Any other ideas?
Thank you!
The obvious solution is to not use a fixed name for temporary files. One common way to do that is to use the process identifier in a suffix, like /tmp/Global_df_tmp.$$.
Also, if you're using bash you could $RANDOM to get a random number.
Or just use the mktemp command to create a temporary file with a randomized name.
You could use the standard mktemp command to generate temporary filenames:
# Create a tempfile (in a BSD- and Linux-friendly way)
my_mktemp () {
mktemp || mktemp -t my_prefix
} 2> /dev/null
declare -r tempfile=$(my_mktemp) || echo "Cannot create temp file" >&2
df -k > "$tempfile" 2>&1
Just a little explanation: the my_mktemp is a bash function (you don't need to use that, just use mktemp if you are on Linux.
The declare -r tempfile=$(cmd) runs the command cmd and sets it as the value of a new read-only variable called temp file.
Afterwards you can use "$tempfile" to refer to the temp file name.
Related
I am currently using the below command in a ,sh to gather a list of contents of a specific folder, on a list of servers, depicted by list.txt (contains IPs)
for f in `cat serverlist.txt`; do
echo "### $f ###";
sshpass -p PASSWORD ssh USER#$f ls /usr/local/folder >>list.txt;
done
Whilst this works, its only half of my problem, I am a total novice with BASH
What I am trying to obtain is a list formatted as such
file1.HOSTNAMEOFSERVER1
file2.HOSTNAMEOFSERVER1
file3.HOSTNAMEOFSERVER1
file1.HOSTNAMEOFNEXTSERVER2
file2.HOSTNAMEOFNEXTSERVER2
file3.HOSTNAMEOFNEXTSERVER2
file1.HOSTNAMEOFNEXTSERVER3
Is any one able to help?
Untested:
while read host; do
sshpass -p PASSWORD ssh USER#"$host" ls /usr/local/folder |
sed 's/$/.'"$host"/;
done < serverlist.txt
DO NOT ACTUALLY PUT YOUR PASSWORD in a script like this. Set up your ssh keys instead.
Just format the ls output into columns with the option -C1: ls -C1 /usr/local/folder.
i need to exchange public key between two systems A and B.
These are the steps am following
copy the content of id_rsa.pub from /root/.ssh directory and save it in variable 'key'
ssh to B as ubuntu user . ssh -i key_file ubuntu#B
Move to root login by sudo su
Append the variable $key to /root/.ssh/authorized_keys
But the file authorized_keys is owned by root. Hence i get the permission error.
I cannot directory connect to system B as root. Only way is to connect as ubuntu and change to root.
I tried the following shell script
# Get all the Ips from the source file
sudo grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' $1 | sort -u > /tmp/list_of_servers.txt
# Get the public key
pubkey=$(sudo cat /root/.ssh/id_rsa.pub)
# For each server
while read ip;
do
(echo "$ip"
# ssh to the server
ssh -i $2 $3#$ip
# append key to autorized_keys file
sudo -c "echo $pubkey >> /root/.ssh/authorized_keys" root
echo "done $ip" )
done < /tmp/list_of_servers.txt
but i didnt work. its giving me permission error.
Can someone help me in the last step.
A fully paranoid approach to the mechanics of the SSH connection might be something like this:
# generate a shell-escaped version of the public key (spaces, wildcards, etc)
printf -v pubkey_q '%q' "$pubkey"
# generate a shell command using that quoted form
cmd="echo $pubkey_q >>/root/.ssh/authorized_keys"
# generate a shell-quoted sudo command invoking the above in a shell
printf -v cmd_q '%q ' sudo bash -c "$cmd"
# ...and execute it on the other end of a ssh connection.
ssh -i "$2" "$3#$ip" "$cmd_q"
printf %q is a bash extension which escapes a string in such a way that being parsed by a shell -- whether in a string that's eval'd, passed to ssh with bash as the remote shell, or passed to bash -c -- evaluates back to the original data. (For regular whitespace its output is safe for sh -c as well, but for any content where bash prefers $'' to escape nonprintable characters, this output may not be POSIX compliant).
This code doesn't do what you think it does:
# ssh to the server
ssh -i $2 $3#$ip
# append key to autorized_keys file
sudo -c "echo $pubkey >> /root/.ssh/authorized_keys" root
The ssh command there would normally open an interactive remote shell, but since we are in a script, an interactive shell is not possible. So the remote shell immediately exits, without actually doing anything at all.
The sudo command that follows is incorrect syntax, it cannot work that way with the -c flag. Check the man page of sudo. And since you are not actually in the remote she'll as you may have believed, the command is running in your local system, not the remote one where you want to append your key.
To run sudo remotely, use something like this:
ssh -i $2 $3#$ip sudo echo hello
The echo is just an example for testing of course.
However, this whole attempt of appending a public key to the authorized list of root is deeply flawed in terms of security. Sudo should be configured to ask for the password of the user, and there is no good way to do that in a script. Or if the user can run sudo without entering a password, that's just unacceptable from a security perspective.
I want to create a directory if it doesn't exists after login to sftp server.
test.sh
sftp name#example.com << EOF
mkdir test
put test.xml
bye
EOF
Now i call test.sh and upload different files each time to test folder. When running this
mkdir test
First time it works and second time it throws Couldn't create directory: Failure error?
How to create a directory if doesn't exists and if exists don't create directory in sftp.
man 1 sftp (from openssh-client package):
-b batchfile
Batch mode reads a series of commands from an input
batchfile instead of stdin. Since it lacks user
interaction it should be used in conjunction with
non-interactive authentication. A batchfile of ‘-’
may be used to indicate standard input. sftp will
abort if any of the following commands fail: get,
put, reget, reput, rename, ln, rm, mkdir, chdir, ls,
lchdir, chmod, chown, chgrp, lpwd, df, symlink, and
lmkdir. Termination on error can be suppressed on a
command by command basis by prefixing the command
with a ‘-’ character (for example, -rm /tmp/blah*).
So:
{
echo -mkdir dir1
echo -mkdir dir1/dir2
echo -mkdir dir1/dir2/dir3
} | sftp -b - $user#$host
I understand this thread is old and has been marked as answered but the answer did not work in my case. The second page on google for a search regarding "sftp checking for directory" so here is an update that would have saved me a few hours.
Using an EOT you cannot capture the error code resulting from the directory not being found. The work around I found was to create a file containing instructions for the call and then capture the result of that automated call.
The example below using sshpass but my script also uses this same method authenticating with sshkeys.
Create the file containing the instructions:
echo "cd $RemoteDir" > check4directory
cat check4directory; echo "bye" >> check4directory
Set permissions:
chmod +x check4directory
Then make the connection using the batch feature:
export SSHPAA=$remote_pass
sshpass -e sftp -v -oBatchMode=no -b check4directory $remote_user#$remote_addy
Lastly check for the error code:
if [ $? -ge "1" ] ; then
echo -e "The remote directory was not found or the connection failed."
fi
At this point you can exit 1 or initiate some other action. Note that if the SFTP connection fails for another reason like password or the address is incorrect the error will trip the action.
Another variant is to split the SFTP session into two.
First SFTP session simply issues the MKDIR command.
Second SFTP session can then assume existence of the directory and put the files.
You can use the SSH access of your account to first verify if the directory exists at all (using the "test" command). If it returns exit code 0, the dir exists, otherwise it doesn't. You can act on that accordingly.
# Both the command and the name of your directory are "test"
# To avoid confusion, I just put the directory in a separate variable
YOURDIR="test"
# Check if the folder exists remotely
ssh name#example.com "test -d $YOURDIR"
if [ $? -ne 0 ]; then
# Directory does not exist
sftp name#example.com << EOF
mkdir test
put test.xml
bye
EOF
else
# Directory already exists
sftp name#example.com << EOF
put test.xml
bye
EOF
fi
Try this to ignore errors if directory already exists.
# Turn OFF error
set +e
# Create remote dirs
sftp -P 22 -o StrictHostKeyChecking=no -oIdentityFile=key.pem -v $user#$host <<EOF
mkdir <remote_path> # create remote directory
bye
EOF
# Turn ON error
set -e
# Do upload to SFTP
sftp -P 22 -o StrictHostKeyChecking=no -oIdentityFile=key.pem -v $user#$host <<EOF
cd <remote_path> # remote_path
put <local_file_path> # local_path
quit
EOF
I'm writing a bash script to send files from a linux server to a remote Windows FTP server.
I would like to check using FTP if the folder where the file will be stored exists before attempting to create it.
Please note that I cannot use SSH nor SCP and I cannot install new scripts on the linux server. Also, for performance issues, I would prefer if checking and creating the folders is done using only one FTP connection.
Here's the function to send the file:
sendFile() {
ftp -n $FTP_HOST <<! >> ${LOCAL_LOG}
quote USER ${FTP_USER}
quote PASS ${FTP_PASS}
binary
$(ftp_mkdir_loop "$FTP_PATH")
put ${FILE_PATH} ${FTP_PATH}/${FILENAME}
bye
!
}
And here's what ftp_mkdir_loop looks like:
ftp_mkdir_loop() {
local r
local a
r="$#"
while [[ "$r" != "$a" ]]; do
a=${r%%/*}
echo "mkdir $a"
echo "cd $a"
r=${r#*/}
done
}
The ftp_mkdir_loop function helps in creating all the folders in $FTP_PATH (Since I cannot do mkdir -p $FTP_PATH through FTP).
Overall my script works but is not "clean"; this is what I'm getting in my log file after the execution of the script (yes, $FTP_PATH is composed of 5 existing directories):
(directory-name) Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
To solve this, do as follows:
To ensure that you only use one FTP connection, you create the input (FTP commands) as an output of a shell script
E.g.
$ cat a.sh
cd /home/test1
mkdir /home/test1/test2
$ ./a.sh | ftp $Your_login_and_server > /your/log 2>&1
To allow the FTP to test if a directory exists, you use the fact that "DIR" command has an option to write to file
# ...continuing a.sh
# In a loop, $CURRENT_DIR is the next subdirectory to check-or-create
echo "DIR $CURRENT_DIR $local_output_file"
sleep 5 # to leave time for the file to be created
if (! -s $local_output_file)
then
echo "mkdir $CURRENT_DIR"
endif
Please note that "-s" test is not necessarily correct - I don't have acccess to ftp now and don't know what the exact output of running DIR on non-existing directory will be - cold be empty file, could be a specific error. If error, you can grep the error text in $local_output_file
Now, wrap the step #2 into a loop over your individual subdirectories in a.sh
#!/bin/bash
FTP_HOST=prep.ai.mit.edu
FTP_USER=anonymous
FTP_PASS=foobar#example.com
DIRECTORY=/foo # /foo does not exist, /pub exists
LOCAL_LOG=/tmp/foo.log
ERROR="Failed to change directory"
ftp -n $FTP_HOST << EOF | tee -a ${LOCAL_LOG} | grep -q "${ERROR}"
quote USER ${FTP_USER}
quote pass ${FTP_PASS}
cd ${DIRECTORY}
EOF
if [[ "${PIPESTATUS[2]}" -eq 1 ]]; then
echo ${DIRECTORY} exists
else
echo ${DIRECTORY} does not exist
fi
Output:
/foo does not exist
If you want to suppress only the messages in ${LOCAL_LOG}:
ftp -n $FTP_HOST <<! | grep -v "Cannot create a file" >> ${LOCAL_LOG}
I'm trying to write a Bash script that uploads a file to a server. How can I achieve this? Is a Bash script the right thing to use for this?
Below are two answers. First is a suggestion to use a more secure/flexible solution like ssh/scp/sftp. Second is an explanation of how to run ftp in batch mode.
A secure solution:
You really should use SSH/SCP/SFTP for this rather than FTP. SSH/SCP have the benefits of being more secure and working with public/private keys which allows it to run without a username or password.
You can send a single file:
scp <file to upload> <username>#<hostname>:<destination path>
Or a whole directory:
scp -r <directory to upload> <username>#<hostname>:<destination path>
For more details on setting up keys and moving files to the server with RSYNC, which is useful if you have a lot of files to move, or if you sometimes get just one new file among a set of random files, take a look at:
http://troy.jdmz.net/rsync/index.html
You can also execute a single command after sshing into a server:
From man ssh
ssh [...snipped...] hostname [command] If command is specified, it is
executed on the remote host instead of a login shell.
So, an example command is:
ssh username#hostname.example bunzip file_just_sent.bz2
If you can use SFTP with keys to gain the benefit of a secured connection, there are two tricks I've used to execute commands.
First, you can pass commands using echo and pipe
echo "put files*.xml" | sftp -p -i ~/.ssh/key_name username#hostname.example
You can also use a batchfile with the -b parameter:
sftp -b batchfile.txt ~/.ssh/key_name username#hostname.example
An FTP solution, if you really need it:
If you understand that FTP is insecure and more limited and you really really want to script it...
There's a great article on this at http://www.stratigery.com/scripting.ftp.html
#!/bin/sh
HOST='ftp.example.com'
USER='yourid'
PASSWD='yourpw'
FILE='file.txt'
ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
binary
put $FILE
quit
END_SCRIPT
exit 0
The -n to ftp ensures that the command won't try to get the password from the current terminal. The other fancy part is the use of a heredoc: the <<END_SCRIPT starts the heredoc and then that exact same END_SCRIPT on the beginning of the line by itself ends the heredoc. The binary command will set it to binary mode which helps if you are transferring something other than a text file.
You can use a heredoc to do this, e.g.
ftp -n $Server <<End-Of-Session
# -n option disables auto-logon
user anonymous "$Password"
binary
cd $Directory
put "$Filename.lsm"
put "$Filename.tar.gz"
bye
End-Of-Session
so the ftp process is fed on standard input with everything up to End-Of-Session. It is a useful tip for spawning any process, not just ftp! Note that this saves spawning a separate process (echo, cat, etc.). It is not a major resource saving, but it is worth bearing in mind.
The ftp command isn't designed for scripts, so controlling it is awkward, and getting its exit status is even more awkward.
Curl is made to be scriptable, and also has the merit that you can easily switch to other protocols later by just modifying the URL. If you put your FTP credentials in your .netrc, you can simply do:
# Download file
curl --netrc --remote-name ftp://ftp.example.com/file.bin
# Upload file
curl --netrc --upload-file file.bin ftp://ftp.example.com/
If you must, you can specify username and password directly on the command line using --user username:password instead of --netrc.
Install ncftpput and ncftpget. They're usually part of the same package.
Use this to upload a file to a remote location:
#!/bin/bash
#$1 is the file name
#usage:this_script <filename>
HOST='your host'
USER="your user"
PASSWD="pass"
FILE="abc.php"
REMOTEPATH='/html'
ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
cd $REMOTEPATH
put $FILE
quit
END_SCRIPT
exit 0
The command in one line:
ftp -in -u ftp://username:password#servername/path/to/ localfile
#/bin/bash
# $1 is the file name
# usage: this_script <filename>
IP_address="xx.xxx.xx.xx"
username="username"
domain=my.ftp.domain
password=password
echo "
verbose
open $IP_address
USER $username $password
put $1
bye
" | ftp -n > ftp_$$.log
Working example to put your file on root...see, it's very simple:
#!/bin/sh
HOST='ftp.users.qwest.net'
USER='yourid'
PASSWD='yourpw'
FILE='file.txt'
ftp -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PASSWD
put $FILE
quit
END_SCRIPT
exit 0
There isn't any need to complicate stuff. This should work:
#/bin/bash
echo "
verbose
open ftp.mydomain.net
user myusername mypassword
ascii
put textfile1
put textfile2
bin
put binaryfile1
put binaryfile2
bye
" | ftp -n > ftp_$$.log
Or you can use mput if you have many files...
If you want to use it inside a 'for' to copy the last generated files for an everyday backup...
j=0
var="`find /backup/path/ -name 'something*' -type f -mtime -1`"
# We have some files in $var with last day change date
for i in $var
do
j=$(( $j + 1 ))
dirname="`dirname $i`"
filename="`basename $i`"
/usr/bin/ftp -in >> /tmp/ftp.good 2>> /tmp/ftp.bad << EOF
open 123.456.789.012
user user_name passwd
bin
lcd $dirname
put $filename
quit
EOF # End of ftp
done # End of 'for' iteration
echo -e "open <ftp.hostname>\nuser <username> <password>\nbinary\nmkdir New_Folder\nquit" | ftp -nv