Okay, so I have a shell script for transferring some files to a remote host using rsync over ssh.
However, in addition to the transfer I need to do some house-keeping beforehand and afterwards, which involves an additional ssh with command, and a call to scp to transfer some extra data not included in the rsync transfer (I generate this data while the transfer is taking place).
As you can imagine this currently results in a lot of ssh sessions starting and stopping, especially since the housekeeping operations are actually very quick (usually). I've verified on the host that this is show up as lots of SSH connects and disconnects which, although minor compared to the actual transfer, seems pretty wasteful.
So what I'm wonder is; is there a way that I can just open an ssh connection and then leave it connected until I'm done with it? i.e - my initial ssh housekeeping operation would leave its connection open so that when rsync (and afterwards scp) runs it can just do its thing using that previously opened connection.
Is such a thing even possible in a shell script? If so, any pointers about how to handle errors (i.e - ensure the connection is closed once it isn't needed) would be appreciated as well!
It's possible. The easiest way doesn't even require any programming. See https://unix.stackexchange.com/questions/48632/cant-share-an-ssh-connection-with-rsync - the general idea is to use SSH connection reuse to get your multiple SSHs to share one session, then get rsync to join in as well.
The hard way is to use libssh2 and program everything yourself. This will be a lot more work, and it seems in your case has nothing to recommend it. For more complex scenarios, it's useful.
Related
Once in a while, I need to ETRN a couple of backup servers (e.g. after maintenance of my own SMTP server).
Normally, I use telnet for that. I go to that server, HELO with my own name and give the ETRN commands. I would like to automate that in a decent way. A simple (but not decent) way would be to start telnet with <<, but the problem is that I do not want to send commands out of order (and the other end may even drop the connection if I do that, haven't tested it yet, but if it works now, it may not work anymore later so I'd like a decent solution and not one that may break later). E.g., I want to wait for the 220 line from the remote SMTP server, then send my HELO, wait for the 250 reply, and only then send the various ETRN commands.
How do I do that in bash? Or do I need to use something else, such as python?
Imagine a network of several nix machines. A dedicated node stores files and periodically schedules Task A that modifies these files. Each of the other nodes schedules Task B that syncs (rsync) those files to local storage.
Task A can take considerable amount of time and the file collection needs to be in a consistent state on all nodes. Thus Task B shouldn't run while Task A is running.
A possible solution for this is to use a reader-writer lock. Task A and Task B would put a write and a read lock on the resource respectively.
I wonder how can we implement such locking mechanism with unix shell scripting.
The usual way of doing this is with the flock utility, which is part of the util-linux package. FreeBSD and NetBSD packages are also available, aiui, and probably others. (For MacOSX, see this question.)
The flock command can do both read ("shared") locks and write ("exclusive") locks. It is based on the flock(2) system call, and is consequently co-operative locking (aka advisory locking), but in most applications that will work fine (but see below for the case where the file is remote).
There are usage examples in the linked man page above. The simplest usage case is
flock /tmp/lockfile /usr/local/bin/do_the_update
flock /tmp/lockfile -s /usr/local/bin/do_the_rsync
both of obtain a lock on /tmp/lockfile, and then execute the specified command (presumably a shell script). The first command obtains an exclusive lock; I could have made that explicit with the -x option. The second command obtains a shared lock.
Since the question actually involves the need for a network lock, it is necessary to point out that flock() may not be reliable on a networked filesystem. Normally, the target file should always be local.
Even in a non-distributed application, you need to consider the possibilities of failure. Suppose you were rsync'ing locally to create a copy, for example. If the host crashes while the rsync is in process, you will end up with an incomplete or corrupt copy. rsync can recover from that, but there is no certainty that when the host restarts, the rsync will initiate before the files are modified. That shouldn't be a problem, but you definitely need to take it into account.
In a distributed application, the situation is more complex because the entire system rarely fails. You can have independent failure of the different servers or of the network itself.
Advisory locking is not persistent. If the lockfile's host crashes with the lock held and restarts, the lock will not be held after the restart. On the other hand, if one of the remote servers which holds the lock crashes and restarts, it may not be aware that it is holding the lock, in which case the lock will never be released.
If both servers were 100% aware of each other's state, this wouldn't be a problem, but it is very difficult to distinguish network failure from host failure.
You will need to evaluate the risks. As with the local case, if the fileserver crashes while an rsync is in progress, it may restart and immediately start modifying the files. If the remote rsync's did not fail while the fileserver was down, they will continue to attempt to synchronize and the resulting copy will be corrupt. With rsync, this should resolve itself on the next sync cycle, but in the interim you have a problem. You will need to decide how serious this is.
You can prevent the fileserver from starting the mutator on startup by using persistent locks. Each rsync server creates its own lockfile on the host before starting the rsync (and does not start the rsync until it is known that the file exists) and deletes the file before releasing the read lock. If an rsync server restarts and its indicator file exists, it knows that there was a crash during the rysnc, so it must delete the indicator file and restart the rsync.
This will work fine most of the time, but it can fail if an rsync server crashes during the rsync and never restarts, or restarts only after a long time. (Or, equivalently, if network failure isolates the rsync server for a long time.) In these cases, it is likely that manual intervention will be necessary. It would be useful to have a watchdog process running on the fileserver which alerts an operator if the read lock has been held for too long, for some definition of "too long".
Okay, so I have two shell scripts, a server and a client, where the server is always run as root, and the clients can be run as standard users in order to communicate with the server.
The method I've chosen to do this is with a world-accessible directory containing named pipes (fifos), one of which is world-writable (to enable initial connection requests) and the others are created per-user and writable only by them (allowing my server script to know who sent a message).
This seems to work fine but it feels like it may be over-engineered or missing a more suitable alternative. It also lacks any means of determining whether the server is currently running, besides searching for its name in the output of ps. This is somewhat problematic as it means that writing to the connection fifo will hang if the server script isn't available to read from it.
Are there better ways to do something like this for a shell script? Of course I know could use an actual program to get access to more capabilities, but this is really just intended to provide secure access to a root service for non-root users (i.e - it's a wrapper for something else).
You could use Unix domain sockets instead of fifos. Domain sockets can be created with nc -lU /path/to/wherever and connected to with nc -U /path/to/wherever. This creates a persistent object in the filesystem (like a fifo, but different). The server should be able to maintain multiple simultaneous connections over the same socket.
If you're willing to write in C (or some other "real" programming language), it's also possible to pass credentials over Unix domain sockets, unlike fifos. This makes it possible for the server to authenticate its clients without needing to rely on filesystem permissions or other indirect means. Unfortunately, I'm not aware of any widely-supported interface for doing this in a shell script.
Client send file, the size may be more than 5G, to slave server, and than slave send to master server.
Will the slave save temp file to itself? I do not want it happen because it will slow the upload speed and waste the slave's memory.
Any way to avoid this? And what is the best way to transfer a big file in golang?
Yes, there's a standard way to avoid store-and-forward approach: as soon as a client connects the slave server the latter should open a connection to the master server and then just stream the data from the client there. Typically this is done using the io.Copy() function. Thanks to Go's excellent duck typing using interfaces, this works for TCP connections and HTTP requests/responses.
(To get better explanation(s) you have to narrow your question down.)
A part of the solution does even appear in the similar questions suggested by stackoverflow—here it is.
I'm trying to upload several hundred files to 10+ different servers. I previously accomplished this using FileZilla, but I'm trying to make it go using just common command-line tools and shell scripts so that it isn't dependent on working from a particular host.
Right now I have a shell script that takes a list of servers (in ftp://user:pass#host.com format) and spawns a new background instance of 'ftp ftp://user:pass#host.com < batch.file' for each server.
This works in principle, but as soon as the connection to a given server times out/resets/gets interrupted, it breaks. While all the other transfers keep going, I have no way of resuming whichever transfer(s) have been interrupted. The only way to know if this has happened is to check each receiving server by hand. This sucks!
Right now I'm looking at wput and lftp, but these would require installation on whichever host I want to run the upload from. Any suggestions on how to accomplish this in a simpler way?
I would recommend using rsync. It's really good at only transferring just the data that's been changed during a transfer. Much more efficient than FTP! More info on how to resume interrupted connections with an example can be found here. Hope that helps!