Automate FTP using Shell Script - shell

I am Using a shell script to transfer files via FTP, my shell works fine.
But the problem is my shell script hangs and does not exits if the FTP connection drops down in between the transfer.
this is how my shell script looks like.
echo "open $ip" > ${cmd_File}
echo "user $usrnm $psswd" >> ${cmd_File}
echo "cd $location" >> ${cmd_File}
echo "binary" >> ${cmd_File}
echo "put $filename" >> ${cmd_File}
echo "bye" >> ${cmd_File}
progress=$(ftp -vin < ${cmd_File} 2>&1) 1> /dev/null
I would be glad if someone can help me to handle the error, my code works really fine unless connection drops in between.
this code does hangs up there only, I need to exit the code when such a thing happens.
Thanks,
Abhijit

Consider rewriting your script using "expect" where you can set a timeout. An example is here. Another example is here.
EDITED:
Alternatively, you could do error checking pretty easily in Perl, like this.
Ok, you can do it in the shell using something along these lines:
YOURTFPCMD & PID=$! ; (sleep $TIMEOUT && kill $PID 2> /dev/null & ) ; wait $PID
which starts your FTP command and saves its PID. It them immediately starts a subshell which will kill your FTP command after $TIMEOUT seconds if it hasn't finished, then waits for your FTP command to exit.

I resolved it using lftp instead of ftp.
In my case I was trying to upload files on GoDaddy Online Storage FTP. For some reason the transfer of the biggest file (500 MB) was hanging forever.
Install it as usual (present in main distros):
yum install lftp (CentOS)
zypper install lftp (openSuse)
...
Then create your script:
#!/bin/sh
echo FTP begin at : $(date)
lftp -u myUser,myPassword myFTPSite <<EOF
put myfile.gz
bye
EOF
echo $(date) : FTP ended
echo Validating RAID
cat /proc/mdstat
exit 0

Use -q quittime option in ftp command:
As per mn ftp:
-q quittime
Quit if the connection has stalled for quittime seconds.
Try this command e.g.:
progress=$(ftp -q 30 -vin < ${cmd_File} 2>&1) 1> /dev/null

Related

Automatize the cert creation OpenVPN

I do not know why I am getting an error when I run my script with SSH, but when I run the bash from my CA server everything works fine.
I installed my VPN server based on this article https://www.digitalocean.com/community/tutorials/how-to-set-up-an-openvpn-server-on-ubuntu-18-04
I wrote a bash for the VPN creation but when I try to run it I need to SSH to the other server at some point. If I start the script with SSH in it I got an error message:
>./easyrsa: 341: set: Illegal option -o echo
My bash contain this and run from my VPN server:
sshpass -p $PASSWORD ssh username#"CA server IP" "/home/username/makevpn.sh $NAME $PASSWORD"
And makevpn.sh contain this:
>./easyrsa sign-req client $NAME
After this run it seems okay but give that error above.
I tried to read after this error and found nothing. :( Hope someone can help because I am hopeless after 4 days of troubleshooting.
Code of VPN script
#!/bin/sh
clear
read -p "Please enter the name of the new certificate : " NAME
read -p "Please enter the Password : " PASSWORD
cd /home/username/EasyRSA-3.0.7/
./easyrsa gen-req $NAME nopass
echo "gen-req done"
cp /home/username/EasyRSA-3.0.7/pki/private/$NAME.key /home/username/client-configs/keys/
echo "cp done"
sshpass -p $PASSWORD scp /home/username/EasyRSA-3.0.7/pki/reqs/$NAME.req username#192.168.1.105:/tmp
echo "scp done"
sshpass -p $PASSWORD ssh username#192.168.1.105 "/home/username/makevpn.sh $NAME $PASSWORD"
echo "ssh done"
cp /tmp/$NAME.crt /home/username/client-configs/keys/
echo "last CP done"
sudo /home/username/client-configs/make_config.sh $NAME
echo "All Done"
Code on CA server
#!/bin/sh
NAME=$1
PASSWORD=$2
cd /home/username/EasyRSA-3.0.7/
echo "CD Done"
./easyrsa import-req /tmp/$NAME.req $NAME
echo "Import-req done"
./easyrsa sign-req client $NAME
echo "Sign-req done"
sshpass -p $PASSWORD scp /home/username/EasyRSA-3.0.7/pki/issued/$NAME.crt username#192.168.1.103:/tmp
echo "Scp done"
I was just browsing the code of that easyrsa script here. This one is likely different from yours given the line for the error is 341. On the Github page, it is line 352 and it is part of a function called cleanup. It appears that this function is only attached as a trap (line 2744). Traps are used to catch signals like sigint (interrupt) which is normally sent on the terminal with ctrl+c (and may display a character like ^C). The reason the error only displays in your script is it likely causes a signal to be emitted that you would not normally receive if you ran it manually over ssh.
The error itself is really not an issue.
Code from Github:
Line 352:
(stty echo 2>/dev/null) || { (set -o echo 2>/dev/null) && set -o echo; }
Line 2744:
trap "cleanup" EXIT
It appears that line is just trying to turn terminal output of your typed characters back on (via stty echo). Sometimes programs will disable terminal output somewhere, and then re-enable it when the program finishes. However, if you were to kill the program mid way through (e.g. with ctrl+c), your program would terminate with the terminal output still disabled. This would make the terminal appear to be frozen. It would still work, but would not display the characters you type with your keyboard. The point of the trap is to ensure that terminal output is re-enabled no matter how the program exits.
More info...
At line 567 there is a function that disables echo. Looks like the point is to not show a password to the screen. If you were to kill the program during password reading, echo would remain disabled on the terminal. Likely the reason for the error has more to do with the way you are running the script. For whatever reason it causes stty echo to fail. Line 352 is assuming that the failure is due to stty echo not being a valid command. So on failure ( || ), it tries a different method (set -o echo) of enabling echo. If I try to run that on my terminal, I also get an error (bash 4.2):
-bash: set: echo: invalid option name

Script stuck during read line when script is executed remotely

I want to have one script which starts a services in another server.
I have tested that the script works as expected in the server where the server is going to run.
This is the code which starts the service and monitors the log until it is in the startup process:
pkill -f "$1"
nohup java -jar -Dspring.profiles.active=$PROFILE $1 &
tail -n 0 -f nohup.out | while read LOGLINE
do
echo $LOGLINE
[[ "${LOGLINE}" == *"$L_LOG_STRING"* ]] && pkill -P $$ tail
done
This works fine as long as I execute that from that machine.
Now I want to call that script from another server:
#!/usr/bin/env bash
DESTINATION_SERVER=$1
ssh root#$DESTINATION_SERVER /bin/bash << EOF
echo "Restarting first service..."
/usr/local/starter.sh -s parameter
echo "Restarting second service..."
/usr/local/starter.sh -s parameter2
EOF
Well, everytime I try that the script of the remote server gets stuck in the "while READ" loop. But as I said, when I execute it locally from the server works fine, and in my "not simplified script" I´m not using any system variable or similar.
Update: I just tried to simplify the code even more with the following lines in the first scenario:
pkill -f "$1"
nohup java -jar -Dspring.profiles.active=$PROFILE $1 &
tail -n 0 -f nohup.out | sed "/$L_LOG_STRING/ q"
I'd say the problem is some how in the "|" through ssh, but I still can find why.
it seems that the problem comes from not having an interactive console when you execute the ssh command, therefore the nohup command behaves strangly.
I could solve it in two ways, outputing the code to the file explicitly:
"nohup java -jar -Dspring.profiles.active=test &1 >> nohup.out &"
instead of:
"nohup java -jar -Dspring.profiles.active=test &1&"
Or changing the way I access via ssh adding the tt option (just one did not work):
ssh -tt root#$DESTINATION_SERVER /bin/bash << EOF
But this last solution could lead to other problems with some character, so unless someone suggests another solution that is my patch which makes it work.

stop bash script from outputting in terminal

I believe I have everything setup correctly for my if else statement however it keeps outputting content into my shell terminal as if i ran the command myself. is there anyway i can escape this so i can run these commands without it populating my terminal with text from the results?
#!/bin/bash
ps cax | grep python > /dev/null
if [ $? -eq 0 ]; then
echo "Process is running." &
echo $!
else
echo "Process is not running... Starting..."
python likebot.py &
echo $!
fi
Here is what the output looks like a few minutes after running my bash script
[~]# sh check.sh
Process is not running... Starting...
12359
[~]# Your account has been rated. Sleeping on kranze for 1 minute(s). Liked 0 photo(s)...
Your account has been rated. Sleeping on kranze for 2 minute(s). Liked 0 photo(s)...
If you want to redirect output from within the shell script, you use exec:
exec 1>/dev/null 2>&1
This will redirect everything from now on. If you want to output to a log:
exec 1>/tmp/logfile 2>&1
To append a log:
exec 1>>/tmp/logfile 2>&1
To backup your handles so you can restore them:
exec 3>&1 4>&2
exec 1>/dev/null 2>&1
# Do some stuff
# Restore descriptors
exec 1>&3 2>&4
# Close the descriptors.
exec 3>&- 4>&-
If there is a particular section of a script you want to silence:
#!/bin/bash
echo Hey, check me out, I can make noise!
{
echo Thats not fair, I am being silenced!
mv -v /tmp/a /tmp/b
echo Me too.
} 1>/dev/null 2>&1
If you want to redirect the "normal (stdout)" output use >/dev/null if you also want to redirect the error output as well use 2>&1 >/dev/null
eg
$ command 2>&1 >/dev/null
I think you have to redirect STDOUT (and may be STDERR) of the python interpreter:
...
echo "Process is not running... Starting..."
python likebot.py >/dev/null 2>&1 &
...
For further details, please have a look at Bash IO-Redirection.
Hope that helped a bit.
*Jost
You have two options:
You can redirect standard output to a log file using > /path/to/file
You can redirect standard output to /dev/null to get rid of it completely using > /dev/null
If you want error output redirected as well use &>
See here
Also, not relevant to this particular example, but some bash commands support a 'quiet' or 'silent' flag.
Append >> /path/to/outputfile/outputfile.txt to the end of every echo statement
echo "Process is running." >> /path/to/outputfile/outputfile.txt
Alternatively, send the output to the file when you run the script from the shell
[~]# sh check.sh >> /path/to/outputfile/outputfile.txt

Get the PID of a process started with nohup via ssh

I want to start a process using nohup on a remote machine via ssh. The problem is how to get the PID of the process started with nohup, so the "process actually doing something", not some outer shell instance or the like. Also, I want to store stdout and stderr in files, but that is not the issue here...
Locally, it works flawlessly using
nohup sleep 30 > out 2> err < /dev/null & echo $!
It is echoing me the exact PID of the command "sleep 30", which I can also see using "top" or "ps aux|grep sleep".
But I'm having trouble doing it remotely via ssh. I tried something like
ssh remote_machine 'nohup bash -c "( ( sleep 30 ) & )" > out 2> err < /dev/null'
but I cannot figure out where to place the "echo $!" so that it is displayed in my local shell. It is always showing me wrong PIDs, for example the one of the "bash" instance etc.
Has somebody an idea how to solve this?
EDIT:
OK, the "bash -c" might not be needed here. Like Lotharyx pointed out, I get the right PID just fine using
ssh remote 'nohup sleep 30 > out 2> err < /dev/null & echo $!'
but then the problem is that if you substitute "sleep 30" with something that produces output, say, "echo Hello World!", that output does not end up in the file "out", neither on the local nor on remote side. Anybody got an idea why?
EDIT2: My fault! There was just no space left on the other device, that's why the files "out" and "err" stayed empty!
So this is working. In addition, if one wants to call multiple commands in a row, separated by a semicolon (;), one can still use "bash -c", like so:
ssh remote 'nohup bash -c "echo bla;sleep 30;echo blupp" > out 2> err < /dev/null & echo $!'
Then it prints out the PID of the "bash -c" on the local side, which is just fine. (It is impossible to get the PID of the "innermost" or "busy" process, because every program itself can spawn new subprocesses, there is no way to find out...)
I tried the following (the local machine is Debian; the remote machine is CentOS), and it worked exactly as I think you're expecting:
~# ssh someone#somewhere 'nohup sleep 30 > out 2> err < /dev/null & echo $!'
someone#somewhere's password:
14193
~#
On the remote machine, I did ps -e, and saw this line:
14193 ? 00:00:00 sleep
So, clearly, on my local machine, the output is the PID of "sleep" executing on the remote machine.
Why are you adding bash to your command when sending it across an SSH tunnel?

Starting a process over ssh using bash and then killing it on sigint

I want to start a couple of jobs on different machines using ssh. If the user then interrupts the main script I want to shut down all the jobs gracefully.
Here is a short example of what I'm trying to do:
#!/bin/bash
trap "aborted" SIGINT SIGTERM
aborted() {
kill -SIGTERM $bash2_pid
exit
}
ssh -t remote_machine /foo/bar.sh &
bash2_pid=$!
wait
However the bar.sh process is still running the remote machine. If I do the same commands in a terminal window it shuts down the process on the remote host.
Is there an easy way to make this happen when I run the bash script? Or do I need to make it log on to the remote machine, find the right process and kill it that way?
edit:
Seems like I have to go with option B, killing the remotescript through another ssh connection
So no I want to know how do I get the remotepid?
I've tried a something along the lines of :
remote_pid=$(ssh remote_machine '{ /foo/bar.sh & } ; echo $!')
This doesn't work since it blocks.
How do I wait for a variable to print and then "release" a subprocess?
It would definitely be preferable to keep your cleanup managed by the ssh that starts the process rather than moving in for the kill with a second ssh session later on.
When ssh is attached to your terminal; it behaves quite well. However, detach it from your terminal and it becomes (as you've noticed) a pain to signal or manage remote processes. You can shut down the link, but not the remote processes.
That leaves you with one option: Use the link as a way for the remote process to get notified that it needs to shut down. The cleanest way to do this is by using blocking I/O. Make the remote read input from ssh and when you want the process to shut down; send it some data so that the remote's reading operation unblocks and it can proceed with the cleanup:
command & read; kill $!
This is what we would want to run on the remote. We invoke our command that we want to run remotely; we read a line of text (blocks until we receive one) and when we're done, signal the command to terminate.
To send the signal from our local script to the remote, all we need to do now is send it a line of text. Unfortunately, Bash does not give you a lot of good options, here. At least, not if you want to be compatible with bash < 4.0.
With bash 4 we can use co-processes:
coproc ssh user#host 'command & read; kill $!'
trap 'echo >&"${COPROC[1]}"' EXIT
...
Now, when the local script exits (don't trap on INT, TERM, etc. Just EXIT) it sends a new line to the file in the second element of the COPROC array. That file is a pipe which is connected to ssh's stdin, effectively routing our line to ssh. The remote command reads the line, ends the read and kills the command.
Before bash 4 things get a bit harder since we don't have co-processes. In that case, we need to do the piping ourselves:
mkfifo /tmp/mysshcommand
ssh user#host 'command & read; kill $!' < /tmp/mysshcommand &
trap 'echo > /tmp/mysshcommand; rm /tmp/mysshcommand' EXIT
This should work in pretty much any bash version.
Try this:
ssh -tt host command </dev/null &
When you kill the local ssh process, the remote pty will close and SIGHUP will be sent to the remote process.
Referencing the answer by lhunath and https://unix.stackexchange.com/questions/71205/background-process-pipe-input I came up with this script
run.sh:
#/bin/bash
log="log"
eval "$#" \&
PID=$!
echo "running" "$#" "in PID $PID"> $log
{ (cat <&3 3<&- >/dev/null; kill $PID; echo "killed" >> $log) & } 3<&0
trap "echo EXIT >> $log" EXIT
wait $PID
The difference being that this version kills the process when the connection is closed, but also returns the exit code of the command when it runs to completion.
$ ssh localhost ./run.sh true; echo $?; cat log
0
running true in PID 19247
EXIT
$ ssh localhost ./run.sh false; echo $?; cat log
1
running false in PID 19298
EXIT
$ ssh localhost ./run.sh sleep 99; echo $?; cat log
^C130
running sleep 99 in PID 20499
killed
EXIT
$ ssh localhost ./run.sh sleep 2; echo $?; cat log
0
running sleep 2 in PID 20556
EXIT
For a one-liner:
ssh localhost "sleep 99 & PID=\$!; { (cat <&3 3<&- >/dev/null; kill \$PID) & } 3<&0; wait \$PID"
For convenience:
HUP_KILL="& PID=\$!; { (cat <&3 3<&- >/dev/null; kill \$PID) & } 3<&0; wait \$PID"
ssh localhost "sleep 99 $HUP_KILL"
Note: kill 0 may be preferred to kill $PID depending on the behavior needed with regard to spawned child processes. You can also kill -HUP or kill -INT if you desire.
Update:
A secondary job control channel is better than reading from stdin.
ssh -n -R9002:localhost:8001 -L8001:localhost:9001 localhost ./test.sh sleep 2
Set job control mode and monitor the job control channel:
set -m
trap "kill %1 %2 %3" EXIT
(sleep infinity | netcat -l 127.0.0.1 9001) &
(netcat -d 127.0.0.1 9002; kill -INT $$) &
"$#" &
wait %3
Finally, here's another approach and a reference to a bug filed on openssh:
https://bugzilla.mindrot.org/show_bug.cgi?id=396#c14
This is the best way I have found to do this. You want something on the server side that attempts to read stdin and then kills the process group when that fails, but you also want a stdin on the client side that blocks until the server side process is done and will not leave lingering processes like <(sleep infinity) might.
ssh localhost "sleep 99 < <(cat; kill -INT 0)" <&1
It doesn't actually seem to redirect stdout anywhere but it does function as a blocking input and avoids capturing keystrokes.
The solution for bash 3.2:
mkfifo /tmp/mysshcommand
ssh user#host 'command & read; kill $!' < /tmp/mysshcommand &
trap 'echo > /tmp/mysshcommand; rm /tmp/mysshcommand' EXIT
doesn't work. The ssh command is not on the ps list on the "client" machine. Only after I echo something into the pipe will it appear in the process list of the client machine. The process that appears on the "server" machine would just be the command itself, not the read/kill part.
Writing again into the pipe does not terminate the process.
So summarizing, I need to write into the pipe for the command to start up, and if I write again, it does not kill the remote command, as expected.
You may want to consider mounting the remote file system and run the script from the master box. For instance, if your kernel is compiled with fuse (can check with the following):
/sbin/lsmod | grep -i fuse
You can then mount the remote file system with the following command:
sshfs user#remote_system: mount_point
Now just run your script on the file located in mount_point.

Resources