I have seen GNU parallel with rsync, unfortunately, I cannot see a clear answer for my use case.
As part of my script I have this:
echo "file01.zip
file02.zip
file03.zip
" | ./gnu-parallel --line-buffer --will-cite \
-j 2 -t --verbose --progress --interactive \
rsync -aPz {} user#example.com:/home/user/
So, I run the script, and as a part of its output, once it gets to the gnu-parallel step, I get this (because I have --interactive, I get prompted to confirm each file:
rsync -aPz file01.zip user#example.com:/home/user/ ?...y
rsync -aPz file02.zip user#example.com:/home/user/ ?...y
Computers / CPU cores / Max jobs to run
1:local / 4 / 2
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
local:2/0/100%/0.0s
... and then, the process just hangs here and does nothing; no numbers change or anything.
At this point, I can do from another terminal this:
$ ps axf | grep rsync
12754 pts/1 S+ 0:00 | | \_ perl ./gnu-parallel --line-buffer --will-cite -j 2 -t --verbose --progress --interactive rsync -aPz {} user#example.com:/home/user/
12763 pts/1 T 0:00 | | \_ rsync -aPz file01.zip user#example.com:/home/user/
12764 pts/1 R 0:11 | | | \_ ssh -l user example.com rsync --server -logDtprze.iLs --log-format=X --partial . /home/user/
12766 pts/1 T 0:00 | | \_ rsync -aPz file02.zip user#example.com:/home/user/
12769 pts/1 R 0:10 | | \_ ssh -l user example.com rsync --server -logDtprze.iLs --log-format=X --partial . /home/user/
... and so I can indeed confirm that processes have been started, but they are apparently not doing anything. As to confirmation that they are not doing anything (as opposed to uploading, which they should be doing in this case), I ran the monitor sudo iptraf, and it reported 0 Kb/s for all traffic on wlan0, which is the only one I have here.
The thing is - the server where I'm logging in to, accepts only SSH authentication with passwords. At first I thought --interactive would allow me to enter the passwords interactively, but instead it prompts the user about whether to run each command line and read a line from the terminal. Only run the command line if the response starts with 'y' or 'Y'.. So ok, above I answered y, but it doesn't prompt me for a password afterwards, and it seems the processes are hanging there waiting for it. My version is "GNU parallel 20160422".
$ ./gnu-parallel --version | head -1
GNU parallel 20160422
So, how can I use GNU parallel, to run multiple rsync tasks with passwords?
Use sshpass:
doit() {
rsync -aPz -e "sshpass -p MyP4$$w0rd ssh" "$1" user#example.com:/home/user
}
export -f doit
parallel --line-buffer -j 2 --progress doit ::: *.zip
The fundamental problem with running interactive programs in parallel is: which program should get the input if two programs are ready for input? Therefore GNU Parallel's --tty implies -j1.
Related
The tini init-process, used in Docker, mentions that process group killing is not activated by default and gives the following example:
docker run krallin/ubuntu-tini sh -c 'sleep 10'
If I run this, and press Ctrl-C immediately after, I indeed have to wait for 10 seconds till the child process exits.
However, if instead of sh I used bash:
docker run krallin/ubuntu-tini bash -c 'sleep 10'
and press Ctrl-C, the process exits immediately.
Why do sh (which is symlinked to dash) and bash behave differently towards this child process?
And how does Bash kill the child process, I thought Bash does not propagate signals by default?
Answered thanks to chepner and Charles Duffy:
bash -c has an implicit optimization where it uses exec to replace itself if possible. sh (dash) does not have this optimization. See also this observation.
To verify:
Process tree using bash:
❯ docker run --name test --rm --detach krallin/ubuntu-tini bash -c 'sleep 60'
03194d48a4dcc8225251fe1e5de2dcbb901c8a9cfd0853ae910bfe4d3735608d
❯ docker exec test ps axfo pid,ppid,args
PID PPID COMMAND
1 0 /usr/bin/tini -- bash -c sleep 60
7 1 sleep 60
Process tree using sh:
❯ docker run --name test --rm --detach krallin/ubuntu-tini sh -c 'sleep 60'
e56f207509df4b0b57f8e6b2b2760835f6784a147b200d798dffad112bb11d6a
❯ docker exec test ps axfo pid,ppid,args
PID PPID COMMAND
1 0 /usr/bin/tini -- sh -c sleep 60
7 1 sh -c sleep 60
8 7 \_ sleep 60
For the following command:
docker ps -a -q | xargs -r docker kill
I get this error:
xargs: illegal option -- r
What would be the MacOS equivalent of the above command?
The equivalent is simply docker ps -a -q | xargs docker kill.
-r (aka. --no-run-if-empty) is only necessary on GNU xargs because it will always run the command at least once by default, even if there is no input; -r disables this. BSD xargs does not have this behavior, so there's no need to disable it.
When I want to run Wireshark locally to display a packet capture running on another machine, this works on bash, using input redirection from the output of a subshell:
wireshark -k -i <(ssh user#machine "sudo dumpcap -P -w - -f '<filter>' -i eth0")
From what I could find, the syntax for similar behavior on the fish shell is the same but when I run that command on fish, I get the Wireshark output on the terminal but can't see the Wireshark window.
Is there something I'm missing?
What you're using there in bash is process substitution (the <() syntax). It is a bash specific syntax (although zsh adopted this same syntax along with its own =()).
fish does have process substitution under a different syntax ((process | psub)). For example:
wireshark -k -i (ssh user#machine "sudo dumpcap -P -w - -f '<filter>' -i eth0" | psub)
bash | equivalent in fish
----------- | ------------------
cat <(ls) | cat (ls|psub)
ls > >(cat) | N/A (need to find a way to use a pipe, e.g. ls|cat)
The fish equivalent of <() isn't well suited to this use case. Is there some reason you can't use this simpler and more portable formulation?
ssh user#machine "sudo dumpcap -P -w - -f '<filter>' -i eth0" | wireshark -k -i -
The original idea was copy out a script to each IP address which would do a yum-install some RPMs and some configuration steps on each machine. Since the yum-install takes about 20 minutes, the hope was to do the install simultaneously on each machine then wait for all the spawned processes to finish before continuing.
#!/bin/bash
PEM=$1
IPS=$2
for IP in IPS; do
scp -i $PEM /tmp/A.sh ec2-user#IP:/tmp
ssh -i $PEM ec2-user#$IP chmod 777 /tmp/A.sh
done
for IP in IPS; do
ssh -t -i $PEM ec2-user#$IP sudo /tmp/A.sh &
done
wait
echo "IPS have been configured."
exit 0
Executing a remote sudo execute command in background on three IP addresses yields three error messages. Obviously, there's flaw in my logic.
Pseudo-terminal will not be allocated because stdin is not a terminal.
Pseudo-terminal will not be allocated because stdin is not a terminal.
Pseudo-terminal will not be allocated because stdin is not a terminal.
sudo: sorry, you must have a tty to run sudo
sudo: sorry, you must have a tty to run sudo
sudo: sorry, you must have a tty to run sudo
All machines are CentOS 6.5
You need to tell ssh not to read from standard input
ssh -n -t root#host "sleep 100" &
Here's an example
drao#darkstar:/tmp$ cat a
date
ssh -n -t me#host1 "sleep 100" &
ssh -n -t me#host2 "sleep 100" &
wait
date
darkstar:/tmp$ . ./a
Mon May 16 15:32:16 CEST 2016
Pseudo-terminal will not be allocated because stdin is not a terminal.
Pseudo-terminal will not be allocated because stdin is not a terminal.
[1]- Done ssh -n -t me#host1 "sleep 100"
[2]+ Done ssh -n -t me#host2 "sleep 100"
Mon May 16 15:33:57 CEST 2016
darkstar:/tmp
That waited in all 101 seconds. Obviously I've the ssh keys so I did not get prompted fro the password.
But looking at your output it looks like sudo on the remote machine is failing ... you might not even need -n.
just to push some devopsy doctrine on you.
Ansible does this amazingly well.
I have running docker ubuntu container with just a bash script inside. I want to start my application inside that container with docker exec like that:
docker exec -it 0b3fc9dd35f2 ./main.sh
Inside main script I want to run another application with nohup as this is a long running application:
#!/bin/bash
nohup ./java.sh &
#with this strange sleep the script is working
#sleep 1
echo `date` finish main >> /status.log
The java.sh script is as follow (for simplicity it is a dummy script):
#!/bin/bash
sleep 10
echo `date` finish java >> /status.log
The problem is that java.sh is killed immediately after docker exec returns. The question is why?
The only solution I found out is to add some dummy sleep 1 into the first script after nohup is started. Than second process is running fine. Do you have any ideas why it is like that?
[EDIT]
Second solution is to add some echo or trap command to java.sh script just before sleep. Than it works fine. Unfortunately I cannot use this workaround as instead of this script I have java process.
This is not an answer, but I still don't have the required reputation to comment.
I don't know why the nohup doesn't work. But I did a workaround that worked, using your ideas:
docker exec -ti running_container bash -c 'nohup ./main.sh &> output & sleep 1'
Okay, let's join two answers above :D
First rcmgleite say exactly right: use
-d
options to run process as 'detached' background.
And second (the most important!) if you run detached process, you don't needed nohup!
deploy_app.sh
#!/bin/bash
cd /opt/git/app
git pull
python3 setup.py install
python3 -u webui.py >> nohup.out
Execute this inside a container
docker exec -itd container_name bash -c "/opt/scripts/deploy_app.sh"
Check it
$ docker attach container_name
$ ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 11768 1940 pts/0 Ss Aug31 0:00 /bin/bash
root 887 0.4 0.0 11632 1396 pts/1 Ss+ 02:47 0:00 /bin/bash /opt/scripts/deploy_app
root 932 31.6 0.4 235288 32332 pts/1 Sl+ 02:47 0:00 python3 -u webui.py
I know this is a late response but I will add it here for documentation reasons.
When using nohup on bash and running it with 'exec' on a docker container, you should use
$ docker exec -d 0b3fc9dd35f2 /bin/bash -c "./main.sh"
The -d option means:
-d, --detach Detached mode: run command in the
background
for more information about docker exec, see:
https://docs.docker.com/engine/reference/commandline/exec/
This should do the trick.