We have an attended-upgrade script which launches apt-get update && apt-get upgrade simultaneously on all our administered systems. Ideally, we'd want to launch them all inside a screen session. When I do it like this:
File: upgrade.sh
for host in $ALLHOSTS
do
some_commands_which_take_considerable_time
screen -X screen sh -c "ssh $host \"apt-get update && apt-get upgrade\""
done
$ screen ./upgrade.sh
, it works, but as there are new windows arriving on the session, they are automatically being switched to. Instead, I'd rather have a version where the active window is fixed unless contained process quits or I switch manually using ^A n.
Bonus points if there is a possibility to preserve windows with exited processes, but keeping them separate from windows with active processes.
You can do this with tmux. For example:
# Start a session named "apt-get" and leave it running in the background.
tmux session-new -d -s apt-get
# Preserve windows with inactive processes.
tmux set-option -t apt-get set-remain-on-exit on
# Start a new window without switching to it. You can also do this from
# within a running tmux session, not just outside of it.
tmux new-window -d -t apt-get -n upgrade-$host \
"ssh $host 'apt-get update && apt-get upgrade'"
Note that you can have multiple windows with the same name, or modify the argument to the -n flag for unique names. The tmux application doesn't care.
As an example, you could name each window "upgrade," but that would make it difficult to identify your SSH sessions at a glance. Instead, this example appends the value of the host variable (populated by your for-loop) to each window name. This makes it easier to navigate, or to programmatically close windows you are no longer interested in. This is especially valuable when you have a lot of unclosed windows displaying terminated processes.
Overall, the syntax is a little cleaner and more intuitive than GNU screen's for this sort of task, but your mileage may vary.
W/r/t preserving windows after a subprocess exits, one possibility is to invoke the zombie command in your screen config file(s), which requires that you specify two keyboard characters that kill or resurrect the window, respectively. E.g.:
zombie KR
Then, K would kill a window whose subprocess has terminated, while R would attempt to relaunch the subprocess in the same window. Note that, in a zombie window, those keys are captured at the top level (i.e., do not precede them with your normal screen control character prefix sequence).
In order to prevent automatic switching to a newly created window, try altering your invocation of screen to something like the following:
screen -X eval 'screen sh -c "ssh $host \"apt-get update && apt-get upgrade\""' 'other'
Thanks to CodeGnome, I'll probably go with tmux as I believe there is no way to do it with screen (that's unfortunate, really!).
To give a better sketch of how it can be used:
#!/bin/sh
tmux new-session -d -s active
tmux new-session -d -s inactive
tmux set-option -t active set-remain-on-exit on
for host in $ALLHOSTS
do
tmux new-window -d -t active: -n upgrade-$host "
./do_upgrade_stuff.sh $host
tmux move-window -s active:upgrade-$host -t inactive:
"
done
Related
I use tmux for a program I run that sometimes crashes, so I can collect errors from it. I use this method of error collection for a few different programs and would like to know if there's a programmatic way of creating multiple tmux sessions with bash, each having their own name and having a command ran in them.
I have tried doing the following:
tmux new -ds "myname" "my command"
tmux new -ds "myname2" "my command"
however, if the program or command in the tmux session finishes/closes/crashes the tmux session is automatically exited and closed, defeating the purpose of trying to get error output.
You have to make the command not exit. Ex. run the shell after the command, so the shell will wait for you:
tmux new -d -s my-session 'sh -c "my command; sh"'
I am trying to open multiple tabs and execute a series of commands in each tab. Lets say I open 3 tabs tab1, tab2, tab3. Then in each tab I would like to execute following:
ssh user#address (PublicKey Authentication is setup and
hence no need to enter password)
Launch python scripts (python some.py)
Hold the tab open after executing the commands to see the outputs.
I went through some threads and have a rough outline for Bash script.
#!/bin/bash
echo "Script running"
gnome-terminal -e "bash -c \"ssh user#address; uname -a; exec bash\""
When I run the above script, a new terminal opens and I can see that I have ssh-ed into the target address but the other command uname -a didnot execute.
I would like to build upon this to implement the following:
Open multiple tabs and run commands. Ex : gnome-terminal --tab -e
"bash -c \"ssh user#address; python file1.py; exec bash\"" -tab -e
"bash -c \"ssh user#address; python file2.py; exec bash\""
Wait for one of the python file to start executing before opening
another tab and repeating the process for another python file.
Also is there a better way to implement the same task ?
The above code snippet was from this thread.
You should consider using screen or tmux or a similar terminal multiplexer for this.
Example usage:
screen -d -m bash -c 'ls; bash'
to initiate a screen session in which ls was executed and then a shell started, and then
screen -X screen bash -c 'date; bash'
to create a new window in the existing screen session, run date therein and then start a shell in that window.
Mind that the programs are run without you seeing their output right away on your controlling terminal. You can then attach to the screen session using
screen -x
Which attach you to the running session and will show you one of the screen windows (the virtual terminals of your two running programs). Typing Ctrl-A n will switch through the windows, Ctrl-A d will detach you again, leaving the programs running, so you can attach later with screen -x.
You can attach from several locations (e. g. from two different Gnome-terminals) to the same running windows. Both will then show the same contents.
Another advantage of using screen is that you can log out and the programs keep running. If you later login again, you can still attach to the running sessions.
Only a direct attack like a reboot, a kill-signal or an interaction (like pressing Ctrl-C while being attached) will terminate your programs then.
I use tmux in my build server. Recently I wrote a small .bashrc script that will automate attaching to tmux session if one exits. The script looks like follows
# Automate tmux Startup
if [ -z "$TMUX" ]; then
# we're not in a tmux session
if [ `ps -o comm= -p $PPID` == "sshd" ]; then
# even VNC can have $SSH_TTY and $SSH_CONNECTION set so we cant find out
# if we want to attach to tmux during ssh so we need to see if parent
# process is sshd see
# http://unix.stackexchange.com/questions/145780/linux-ssh-connection-is-set-even- without-sshing-to-the-server
# Only attach to tmux if its me
WHOAMI=$(whoami)
if tmux has-session -t $WHOAMI 2>/dev/null; then
tmux -2 attach-session -t $WHOAMI
else
echo "Start tmux with username as session name 'tmux new -s $WHOAMI' "
fi
fi #parent process check
else
echo "Inside tmux"
fi
The problem is whenever I ssh using mosh it just hangs inside the tmux window. I found that if I remove this script and then use mosh to just ssh and attach to tmux manually then I don't hit on this issue. This issue only happens if I place the above script in .bashrc.
My suspicion was that mosh is waiting for .bashrc to complete and waits for it indefinitely and in the mean time it also does not pass the mouse and keystroke controls over to tmux. I confirmed this by killing the tmux session from another terminal and found that mosh recovered and tried to execute my previously buffered keystrokes.
The strange thing is that how mosh managed to cross the ps -o comm= -p $PPID == "sshd" check. This is because for mosh the Process name of shell is bash and Parent process name of the shell is mosh-server and not sshd. Further investigation revealed that mosh executes .bashrc twice once as sshd and once as mosh-server. This is reproducible by putting ps -o comm= -p $PID -p $$ >> moshbash in .bashrc.My tmux attach was happening in sshd and hanging mosh forever.
A simple workaround I found was to do
mosh user#server -- tmux attach -t `whoami`
I can do a similar thing to my ssh as well to fire command from the client side and eliminate the .bashrc script altogether but I do not wish to spill over my server side automation to the client side.
Actually mosh does not seem to be wrong. It is sourcing a .bashrc file only once per PID .I think it is a bad design to have .bashrc kick of a blocking session to tmux since tmux also needs terminal we cant start off as a background process either so & also wont work.
Is there any other way around this problem ? I think if we can distingusih between a mosh client setting up a sshd and a ssh client setting up sshd that information can be used.
.bashrc always executes everytime an interactive non-login bash instantiates so use .bash_profile instead so it would only run once during login to ssh. If the script or processes of the script summons bash, it would cause repeated summoning.
See Bash Startup Files for more info and other startup files.
I'm using gdb to debug a parallel mpi-code 'prog'. For that I use a small number of processes, say 'M' and do something like
mpiexec -n M xterm -e gdb ./prog
This pops up M xterms with each of them running one gdb process on one of the files prog.
The resulting cluttering of the screen by individual windows can be rather cumbersome.
Is there any way, using any known split-window terminal emulator (say, terminator), such as to have the M gdb processes starting up in only one window, however split into M parts from start?
I faced a similar problem and have found tmpi which does exactly what you want: launch mpi debugging processes in M tmux terminal windows.
Clone the repository:
git clone https://github.com/Azrael3000/tmpi.git
then install with:
sudo ./tmpi/install.sh
which places the tmpi executable in /usr/local/bin
The tmpi executable and tmux must be in the path on all your servers.
Run a job with:
tmpi M gdb my_executable
where M is the number of processes that you want.
What you want is called a 'terminal multiplexer'; look into screen or tmux
EDIT: this is probably what you want; issue the following commands in your shell
tmux new-session -d bash # start a bash shell
tmux split-window -v python # start a python shell below it
tmux attach-session -d # enter the tmux session
I help maintain a large number of Unix-like servers, and so keep a script called tmux-rebuild that I use to rebuild all the tmux sessions and windows with SSH links to each server.
I have tmux configured to show the window's name in red with an exclamation mark in its status bar when a terminal bell character is printed in that window. This is very handy for programs like irssi alerting me to when I have messages in another window.
I also have my $PS1 set up on every server to print a terminal bell at the end of every prompt. This is useful because if I run a long job in one window and switch to another, I can immediately see when it's finished because when my prompt is written to the screen after the job is done, tmux makes the window name come up in red with an exclamation mark. This is great for my workflow.
However it causes a slight problem with the rebuild script mentioned above, because when I start up tmux after running it, every window in every session is flagged in red, due to the first prompt being printed to the screen. This makes the feature useless until I visit every window, and there are something like 40-50 of them.
Is there something I can add to my script that will clear all alerts from sessions and windows after they are created? I don't mind using a kludge if necessary.
From the tmux man page, specifically the last sentence here:
kill-session [-aC] [-t target-session]
Destroy the given session, closing any windows linked to it
and no other sessions, and detaching all clients attached
to it. If -a is given, all sessions but the specified one is
killed. The -C flag clears alerts (bell, activity, or
silence) in all windows linked to the session.
So, simply:
tmux kill-session -C
Figured out an acceptable workaround; I redefined the next/previous bindings to allow repeats:
# Allow repeats for next/prev window
bind-key -r n next-window
bind-key -r p previous-window
This allows me to quickly sweep up the alerts for all windows in a session by pressing my prefix key and tapping "n" until they're all clear, and I'm back in my original window.
With tmux 1.6 (and later), list-windows can generate customizable output, so it is fairly simple to read the output lines and make a loop that runs select-window for each window.
Add list-session (to loop over all sessions, optionally), and display-message (to parse session specifiers, and to record the current/“last” windows so they can be properly restored), and you might end up with something like this:
#!/bin/sh
# usage: tmux-select-each [session [...]]
#
# Select every window in specified session(s). If no sessions are
# specified, process all windows in all sessions.
#
# This can be handy for clearing the activity flags of windows in
# freshly spawned sessions.
if test $# -gt 0; then
for session; do
tmux display-message -p -t "$session:" '#S'
done
else
tmux list-sessions -F '#{session_name}'
fi |
while read -r session; do
active_window=$(tmux display-message -p -t "$session:" '#S:#I')
last_window=$(tmux display-message -p -t "$session:"\! '#S:#I' 2>/dev/null)
tmux list-windows -t "$session" -F '#{session_name}:#{window_index}' |
while read -r window; do
if test "$window" = "$active_window" ||
test "$window" = "$last_window"; then
continue
fi
tmux select-window -t "$window"
done
if [ -n "$last_window" ]; then
tmux select-window -t "$last_window"
fi
tmux select-window -t "$active_window"
done