I have a python file that I run using a .sh file providing some args. When I run this file on the terminal using bash myscript.sh it runs fine, prints progress on the console. When I use nohup like nohup bash myscript.sh > log.log nothing gets saved to log.log file but the processes are running (it's a 4 GPU process and I can see the GPU usage in top and nvidia-smi). As soon as I kill the process using either ctrl+c or kill command, all the output gets printed to the file at once along with keyboard interrupt or process killed message.
I have tried nohup myscript.sh &> log.log nohup myscript.sh &>> log.log but the issue remains the same. What's the reason for such a behaviour?
myscript.sh runs a python file somewhat like
python main.py --arg1 val1 --arg2 val2
I tried
python -u main.py
but it doesn't help. I know the script is working fine as it occupies exactly same amount of memory as it should.
Use stdbuf:
nohup stdbuf -oL bash myscript.sh > log.log
Your problem is related to output buffering. In general, non-interactive output to files tend to be block buffered. The -oL changes output buffering to line mode. There is also the less efficient -o0 to change it to unbuffered.
Related
I've been using below command to make tail to write nohup.out and also print the output on the terminal.
nohup train.py & tail -f nohup.out
However, I need nohup to use different file names.
When I try
nohup python train.py & tail -F vanila_v1.out
I'm getting following error message.
tail: cannot open 'vanila_v1.out' for readingnohup: ignoring input and appending output to 'nohup.out': No such file or directory
I also tried
nohup python train.py & tail -F nohup.out > vanila_v1.txt
Then it doesn't write an output on stdout.
How do I make nohup to write other than nohup.out? I don't mind simultaneously writing two different files. But to keep track of different processes, I need the name to be different.
Thanks.
You need to pipe the STDOUT and STDERR for the nohup command like:
$ nohup python train.py > vanila_v1.out 2>&1 & tail -F vanila_v1.out
At this point, the process will go into the background and you can use tail -f vanila_v1.out. That's one way to do it.
A little more information is available here for the STDOUT and STDERR link. Here is another question that uses the tee command rather that > to achieve the same in one go.
When using nohup the output of a script is buffered and only gets dumped to the log file (nohup.out) after the script has finished executing. It would be really useful to see the script output in something close to real-time, to know how it is progressing.
Is there a way to make nohup write the output whenever it is produced by the script? Or, since such frequent file access operations are slow, to dump the output periodically during execution?
There's a special program for this: unbuffer! See http://linux.die.net/man/1/unbuffer
The idea is that your program's output routines recognize that stdout is not a terminal (isatty(stdout) == false), so they buffer output up to some maximum size. Using the unbuffer program as a wrapper for your program will "trick" it into writing the output one line at a time, as it would do if you ran the program in an interactive terminal directly.
What's the command you are executing? You could create a .bash file which inside is redirecting the output to files after each command (echo "blabh" >> your-output.txt) so you can check that file during the nohup execution (you should run: nohup script.bash &)
Cheers
Please use stdbuf. It's added in GNU coreutils from 7.5.
stdbuf -i0 -o0 -e0 cmd
Command with nohup like:
nohup stdbuf -i0 -o0 -e0 ./server >> log_server.log 2>&1
Reference
I run my_program via a bash wrapper script, and use exec to prevent forking a separate process:
#! /bin/bash
exec my_program >> /tmp/out.log 2>&1
Now I would like to duplicate all output into two different files, but still prevent forking, so I do not want to use a pipe and tee like this:
#! /bin/bash
exec my_program 2>&1 | tee -a /tmp/out.log >> /tmp/out2.log
How to do that with bash?
The reasons for avoid forking is to make sure that:
all signals sent to the bash script also reaches my_program (including non-trappable signals).
waitpid(3) on the bash-script can never return before my_program has also terminated.
I think the best you can do is to redirect standard output and error to tee via a process substitution:
exec > >( tee -a /tmp/out.log >> /tmp/out2.log) 2>&1
then exec to replace the bash script with your program (which will keep the same open file handles to standard output).
exec my_program
I'm trying to write a Linux shell script (preferably bash),
supposedly named detach.sh, to safely detach programs from a terminal,
such that:
Invocation: ./detach.sh prog [arg1 arg2 ...].
Is exec-able, eg. by running this in your shell:
exec ./detach.sh prog [arg1 arg2 ...]
With proper quoting (mainly handling of arguments containing whitespaces).
Discards the outputs (since they are unneeded).
Does not use screen, tmux, etc.
(same reason with 4, plus no need for an extra babysitting process).
Uses (reasonably) portable commands and programs,
and no things like start-stop-daemon which is quite distro-specific.
I have thought of several ways (shebang lines #!/bin/bash neglected
for the sake of briefness):
nohup:
nohup "$#" >& /dev/null &
disown:
"$#" >& /dev/null &
disown
setsid:
setsid "$#" >& /dev/null &
Using a subshell:
("$#" >& /dev/null &)
nohup/setsid combined with subshell:
# Or alternatively:
# (nohup "$#" >& /dev/null &)
(setsid "$#" >& /dev/null &)
When using gedit as the test program (substituting the "$#" part),
condition 1 can be satisfied with all the above methods,
but condition 2 can be satisfied with none.
However, if an arbitrary program (but not a shell builtin) is appended to script 5,
all the conditions seem to be satisfied (at least for me in the gedit case).
For example:
(setsid "$#" >& /dev/null &)
# Not just `true' because it is also a shell builtin.
/bin/true
Anyone with an idea about an explanation of the above phenomenons
and how to correctly implement the requirements?
EDIT:
With condition 2, I mean the program should be detached from the terminal but runs as usual otherwise. For example, with the gedit case, the condition fails if gedit just exits immediately right after the process of the script has ended.
Upon closer investigation, these previously unnoticed facts were revealed:
Both scripts 3 and 5 (the setsid variant only) will
satisfy all the conditions if a /bin/true is appended to the script.
These scripts, as modified in fact 1, will work as well if
/bin/true is replaced with for i in {0..9999}; do :; done.
Therefore we can conclude that:
(From fact 1)
Multiple levels of detaching (as in script 5) is unnecessary,
and the key is to use the right utility (setsid).
(From fact 2)
A suitable delay before bash exit is necessary for the success of the script.
(Calling external program /bin/true consumes some time,
just like the pure-bash time consumer for i in {0..9999}; do :; done.)
I have not looked at the source code, but I guess a possible explanation
is that bash may exit before setsid finishes configuring the execution
environment of the program to run, if an appropriate delay is not applied.
And finally, an optimal solution should be
#!/bin/bash
setsid "$#" >& /dev/null &
sleep 0.01
EDIT 1:
The necessity of a delay has been explained here. Many thanks to #wilx!
EDIT 2:
(Thanks to #MateiDavid) we seem to have forgotten to redirect the standard input, and a better way would be:
#!/bin/bash
setsid "$#" >& /dev/null < /dev/null &
I think you need to do setsid "$#" >& /dev/null & wait so that the controlling terminal does not disappear before setsid manages to fork the child.
UPDATE
It seems to me that this works both on command line and as argument of -c:
(setsid tail -F /var/log/messages >& /dev/null) & disown
You are trying to create a UNIX daemon process (i.e., a process that has no controlling terminal and that is its own session leader). The setsid command should do this for you, but you are responsible for closing all file descriptors that are open on the terminal you are abandoning. This can be done by redirecting them to /dev/null or using the shell's syntax for closing file descriptors (e.g., 2>&- and 0<&- in Bash).
I'm using a bash script to automate a set of tests for some other code. The script looks like this:
for i in $(seq 1 10)
do
cd ~/BuildBot
rm -rf program # Remove the directory each time to test installation
git clone /localrepo/
cd ~/into/program
python2.6 setup.py build_ext -i
cd tests
python runtest.py >& ~/into/reptest/runtest-all.out.$i
echo $? > ~/into/reptest/runtest-all.exit.$i
done
When run like this, the script does what I want - shows me a Wall of Text, and saves it to a file in the directory reptest. Now that I've tested installation, it's getting annoying to have to wait for the whole program to reinstall. However, when I cut the script down to
for i in $(seq 1 10)
do
cd ~/into/program/tests
python runtest.py >& ~/into/reptest/runtest-all.out.$i
echo $? > ~/into/reptest/runtest-all.exit.$i
done
The script hangs, nothing happens, the shell waits on a blank line until I Ctrl-C it. What happens to the output? How do I get back my Wall of Text?
python runtest.py >& ~/into/reptest/runtest-all.out.$i
Redirects both stdout and stderr from runtest.py into the file ~/into/reptest/runtest-all.out.$i. Your wall of text comes from the statements you've pruned away.
What you're interested in are probably something like:
( python runtest.py 2>&1 ) | tee ~/into/reptest/runtest-all.out.$i
Which runs python runtest.py in a subshell and redirect it's stderr to stdout, then pipe the output from that subshell into "tee ~/into/reptest/runtest-all.out.$i". tee saves it's stdin to the file given as an argument in addition to copying it to stdout.
As to why your programs waits until you send it SIGINT, I'm not sure, I don't see anything bash-related which should make your script hang until interrupted.