I'm having an issue but I can't see something related with this.
I'm having an issue, something is happening while I'm performing an FTP connection with a server is transferring a file, but for some reason sometimes is stuck but I would like to prevent have the connection opened, there is a way to see if the FTP connection is not transferring, close the connection?
I really don't have any code due I'm not sure if this is possible,
Any idea what can I do at this point?
If it is closing the connection while you are transferring files, then it's either your FTP/SFTP client, server, or network. First, Switch to a different FTP/SFTP client. Some have more tools for analysis than others. I have had to do this before. If that doesn't work, check the internet connection or contact your system/network administrator.
there is a way to see if the FTP connection is not transferring, close
the connection?
If you are downloading a file, you can indirectly see the FTP transferring by watching the file's size:
name=$1
size=0
while sleep 10
set -- `ls -s $name`
[ "$1" -gt $size ]
do size=$1
done
exit 1
The above script (let's call it growing) runs while the file (passed as a parameter) grows.
In your script you could write something like
growing file || pkill ftp &
before you start the FTP. If the file stops growing for ten seconds, ftp would be killed and the connection thereby closed. If ftp terminates normally, you could kill $! or just let growing end.
I connected via ssh to remoute server and started very long wget downloading. Then ssh session was broke, and after reconnect new copy of interpretator was created. Now I see wget process in ps, but is it possible to return control to old interpretator? I know that better solution is to use screen for long commands, but is there other way?
No, there is no way to reattach a process to a different terminal if it's not set up to do that (by way of screen / tmux / what have you) in the first place.
As a crude approximation, connecting a debugger to the running process may allow you to interact with it in some limited ways, but in this particular scenario, I don't think it will be beneficial.
If you want to know the progress of your currently running wget, check out the size of the downloaded file, it should be growing. If it doesn't, run killall wget and start over.
Next time, consider running wget --background to prevent the problem from happening. See the wget info page.
This command let Wget to work in the background, and write its progress to log file my.log.
Еhe number of retries 45 (-t options)
wget -t 45 -o my.log http://upload.wikimedia.org/wikipedia/commons/5/51/Google.png &
For the past few years, I have been using an rsync one-liner to back up important folders on my Mac Mini desktop (OSX 10.9, 2.5 GHz i5, 4 GB RAM) to a FreeNAS box (0.7.2 Sabanda revision 5266, Pentium D 2.66 GHz, 822MiB RAM [reported by the system, I think there's 1 GB in there]). I am running an rsync daemon on the FreeNAS box. Recently, these transfers have been hanging indefinitely. I have done the usual Google-fu and am unable to identify the source of the problem or a solution.
The one-liner is:
rsync -rvOlt --exclude '.DS_Store' \
--exclude '.com.apple.timemachine.supported' \
--delete /Volumes/Storage/Music/Albums/ 192.168.1.100::albums
I have tried enabling -vvv and --progress, but there is no pattern that I can discern between what hangs and what doesn't. Heck, if I retry, the same file might hang at a different point during the transfer or not at all. A dry run (-n) does not always succeed either. The only "success" I've had is implementing a timeout (--timeout=10) and rerunning the command over and over. Eventually, I creep along, but with no guarantee of success and at a pace that is unacceptable. I've reached a point where I have one file that I can't get past.
The Mac Mini is connected to my router via 5 GHz. The FreeNAS box is wired into that same router on a 100 mbit port. When transfers are actually going, rsync --progress reports 2.5-4 MB/s. According to --progress, a hang is literally just that—no data transfer is occurring as far as I can tell.
I need help with both the diagnostics and the solution.
I was having the same problem. Removing -v didn't work for me. My use-case is slightly different in that I'm going from source (EXT4) to ExFAT. The issue for me was that rsync was attempting to preserve device files and permissions, which ExFAT doesn't support. I was using the -hrltDvaP switches. The -D and -a switches seemed to be my problem. The -a switch translates to -rlptgoD (no -H,-A,-X). The -p, -g, and -o switches seemed to be my root cause as rsync was barfing on one or all of those during runtime. Removing -a and specifying -Prltvc switches explicitly is working for me.
bkupcmd="nice -n$nicelevel /usr/bin/rsync -Prltvc --exclude-from=/var/tmp/ignorelist "
I've been running into the same thing again and again and it seems to help if you drop the -v option (which is annoying if you need that output).
Try using --whole-file/-W.
This command disables the rsync delta-transfer algorithm.
That is what worked for us (WSL to OSX)
our full sync flags were -avWPle
(e was because we were using ssh, and that has to be the last flag)
This happened to me when the remote device ran out of space. The error wouldn't show when --verbose option was used; turning that off yielded some STDERR output that explained that the remote device was out of space. When I freed some space, I was able to run rsync again with --verbose and everything went fine.
I am using openSUSE 13.2 Linux, rsync version 3.1.1-2.4.1.x86_64, and I experienced similar problems, doing an rsync between my laptop and an external hard disk, with the destination device definitively having enough free space.
I thought I got an improvement omitting option -v, but after 10 minutes it was hanging again: strace said:
select(5, [], [4], [], {60, 0}) = 0 (Timeout)
And with "iotop" I counld see confirm that the rsync processes did no significant disk IO any more.
Neither removing the -v option nor limiting the bandwidth using --bwlimit fixed the problem.
Just had a similar problem while doing rsync from harddisk to a FAT32 USB drive. rsync froze already in less than a second in my case and did not react at all after that ... left it with CTRL+C.
Found out that the problem was a combination of usage of hardlinks on the harddisk and having FAT32 filesystem on the USB drive, which does not support hardlinks.
Formatting the USB drive with ext4 solved the problem for me.
In my situation rsync was not actually failing.
I have regular server backups which transfers large files over 500GB+ and have --append-verify or --checkusm over ssh parameters specified.
What I have found upon analysis is that once the client side completes it's file checks then the server side checks start. Which means while the server is doing it's checks the client side will appear hanged and frozen - run htop on the server to rsync working away.
This is likely a non issue if rsync is run in deamon mode on the server and using the rsync protocol instead of ssh for transfers.
On related note, this very LONG wait would trigger SSH timeout and a rsync: connection unexpectedly closed (254 bytes received so far) [sender] error message, sollution is to add ClientAliveInterval 120 and ClientAliveCountMax 720 to /etc/ssh/sshd_config.
I've seen this quite often on 3.0.9 on a directory with hardlinks, but it also happened on 3.1.3.
There is a nice analysis in Debian bug 820916: when its internal sockets are congested with errors, rsync could go into a deadlock.
This might have been fixed in a 3.2 release just a few days ago (Jun 2020):
Avoid a hang when an overabundance of messages clogs up all the I/O buffers.
The only good workaround I can think of is, if the problem is not persistent, then put timeout in front of it: timeout rsync <args> <source> <destination>, then retry. If it is persistent for you, you're the lucky one who can debug it :D
It also happens when the user on target machine has not write permissions on target folder.
You can try giving write permission to others target folder:
sudo chmod -R o+w /path/to/target-folder
In my case, it was the IPC (Intrusion Protection Component) in our firewall. It sees all the TCP SYN packets as a flood attack and kills the connection. I left a rsync over NFS session open and turned off the IPC for the servers firewall rule and it starting working again right away.
rsync -ravh /source /destination
When it happened I was not able to kill the rsync session. It locked up the NFS mount and I would have to reboot the client machine to get it to work again. The strange thing is it would copy some files over then all of a sudden stop. It always seemed to stop on the same file. So I was looking for file issues, permission issues, TCP offloading issues, tried removing the -v in the rsync call. If you are having this issue at least in my case it even happened with a simple.
cp -rp /source /destination
So I knew then to start looking at other factors. So if you have any sort of intrusion protection on a firewall or router between the servers you can try turning that off temporarily to see if it solves your issue as well.
Most likely not "your" problem, but I stumbled upon this question when I was researching a similar behavior:
I'm observing "hanging" when the target site has too much io load. e.G. on one of my small business servers, when someone is resyncing his IMAP account and downloading large batchs of data and a backup job runs that writes his data.
In this situation I notice a steep drop in performance for rsync. Noticeable in a high load value in top on the target machine, even though CPU and Mem are fine.
Waiting for the process to finish has helped every time or interrupting and attempting the rsync at a later time again.
I was having the same problem and it was because I was running out of memory during the rsync. Created a swap file and problem solved.
Had rsync hanging issue on Ubuntu 16. None of the options above helped. The problem was in the source drive (external SSD) which suddenly became faulty. I tried several disk checks, but all of them stuck. Ended up rebooting the system and disk suddenly became accessible again.
Holger Ohmacht aka h8ohmh / 8ohmh:
The problem lies in the filesystem buffer / usage of the interworking of harddisk/hw so far as I could investigate.
Temporal solution for local drives (eg. USB3<->HD) : A script which is polling the changing disk space. If no changing free disk space then rsync is stalled and has to be restarted
cmd="rsync -aW --progress --stats --preallocate --super \
<here your source dir> \
<here your dest dir>"
eval "$cmd" &
rm ./ndf.txt
rm ./odf.txt
while [[ 0 == 0 ]]; do
df > ./ndf.txt
cmp ./odf.txt ./ndf.txt
res="$?"
echo "$res"
if [[ $res == 0 ]]; then
echo "###########################################"
ls -al "./ndf.txt"
ls -al "./odf.txt"
killall rsync
eval "$cmd" &
else
cp ./ndf.txt ./odf.txt
fi
sleep 60
done
Change <source dir> etc to your paths!
In my case it is always stalling by usage of rsync's --preallocate option (normally because of better disk performance and rescueing continuous blocks), so as long as the disk and filesystem drivers not reworked there just this solution
I know these kinds of questions have been asked for years, and the answer to them are often Screen or tmux.
I surely will use screen at beginning if I know I will leave the session for a long time, or the network is too bad to maintain a reliable connection.
The main problem is when I start some session and find it must last long later, or the connection just lost accidentally. In the later case, often when I start another session immediately, I can find the previous processes are not killed at that time, but I just have no way to reconnect to their terminal.
So I wonder if it is possible to prevent normal processes from killed even long time after accidentally disconnected SSH session. And the most important is I can reconnect to their terminals with out start them in Screen in advance.
If not, is is possible to move a already started bare ssh session into a new Screen session for later reconnect?
I don't believe it's possible without something like screen. Once your pseudo-TTY is lost I'm almost certain it can't be recovered from a different shell (at least not without some narly hacks).
As far as adding an existing process to a new screen I think that is possible. Try the instructions here:
http://monkeypatch.me/blog/move-a-running-process-to-a-new-screen-shell.html
The first thing to do is to suspend the process. In my case, Irssi can be suspended by typing Ctrl + Z.
Secondly, resume the process in background:
$ bg
Now, we will detach the process from its parent (the shell). So, when the parent process will be terminated, the child (Irssi) will be able to continue. For this, we use the disown builtin:
$ disown irssi
Launch a screen session:
$ screen
As we are in a screen session, we will retrieve the irssi process. To do so, we use the reptyr command which take a pid:
$ reptyr
To avoid the tedious pid research, we can use the pgrep command:
$ reptyr $(pgrep irssi)
Now the process is in a screen shell, we can safely detach our session and no longer worry about killing our X server or close our ssh connection.
You'll need reptyr for this.
OPTION 2:
I suspect you may be trying to solve the wrong problem. If your SSH connection is dropping, why not address that? You can set SSH to be incredibly tolerant of timeouts and disconnects by tweaking your connection settings.
On your client, in $HOME/.ssh/config add:
ServerAliveInterval 60
ServerAliveCountMax 5
Now your sessions won't timeout even if the server doesn't respond for 5 minutes.
Use ssh-tmux instead of tmux:
function ssh-tmux(){
if ! command -v autossh &> /dev/null; then echo "Install autossh"; fi
autossh -M 0 $* -t 'byobu || {echo "Install byobu-tmux on server..."} && bash'
}
I worked on a text file using nano and I got disconnected. After I logged in I saw the nano process from the previous session was still running, but I couldn't switch to that nano instance. So, I killed the nano process and then it created file named filename.save. Which had my changes from the first session.
I have an expect script that I have written that at the end calls scp to copy a large file from server a to server b. The issue I have (which is the same using rsysnc and not scp) is that the expect script is terminating before the file transfer is complete. I know that I can set the timeout in expect but as the file size grows so will the timeout. Is there anyone that has come across this issue and is there a wait function that I can use in expect?
You can set the timeout to -1 to disable the timeout feature.