I found that some people run a program in shell like this
exe > the.log 2>&1 &!
I understand the first part, it redirects stderr to stdout also "&" means runs the program in background, but I don't know what does "&!" mean, what does the exclamation mark mean?
Within zsh the command &! is a shortcut for disown, i.e. the program won't get killed upon exiting the invoking shell.
See man zshbuiltins
disown [ job ... ]
job ... &|
job ... &!
Remove the specified jobs from the job table; the shell will no longer report their status, and will not complain if you try to exit an interactive shell with them running or stopped. If no job is specified, disown the current job. If the jobs are currently stopped and the AUTO_CONTINUE option is not set, a warning is printed containing information about how to make them running after they have been disowned. If one of the latter two forms is used, the jobs will automatically be made running, independent of the setting of the AUTO_CONTINUE option.
Related
When I start some background process in the shell, for example:
geth --maxpeers 0 --rpc &
It returns something like:
[1] 1859
...without any out stream of this process. I do not understand what is it? And how can I get stdout of geth? There is information in documentation that stdout of background process is displayed in shell by default.
My shell is running in remote Ubuntu system.
The "&" directs the shell to run the command in the background. It uses the fork system call to create a sub-shell and run the job asynchronously.
The stdout and stderr should still be printed to the screen.
If you do not want to see any output on the screen, redirect both stdout and stderr to a file by:
geth --maxpeers 0 --rpc > logfile 2>&1 &
Regarding the first part of your question:
...without any out stream of this process. I do not understand what is
it?
This is part of the command execution environment (part of the shell itself) and is not the result of your script. (it is how the shell handles backgrounding your script, and keeping track of the process to allow pause and resume of jobs).
If you look at man bash under JOB CONTROL, it explains what you are seeing in detail, e.g.
The shell associates a job with each pipeline. It keeps a table
of currently executing jobs, which may be listed with the jobs
command. When bash starts a job asynchronously (in the background),
it prints a line that looks like:
[1] 25647
I do not understand what is it? [1] 1859
It is the output from Bash's job feature, which enables managing background processes (jobs), and it contains information about the job just started, printed to stderr:
1 is the job ID (which, prefixed with %, can be used with builtins such as kill and wait)
25647 is the PID (process ID) of the background process.
Read more in the JOB CONTROL section of man bash.
how can I get stdout of geth? There is information in documentation that stdout of background process is displayed in shell by default.
Indeed, background jobs by default print their output to the current shell's stdout and stderr streams, but note that they do so asynchronously - that is, output from background jobs will appear as it is produced (potentially buffered), interleaved with output sent directly to the current shell, which can be disruptive.
You can apply redirections as usual to a background command in order to capture its output in file(s), as demonstrated in user3589054's helpful answer, but note that doing so will not silence the job-control message ([1] 1859 in the example above).
If you want to silence the job-control message on creation, use:
{ geth --maxpeers 0 --rpc & } 2>/dev/null
To silence the entire life cycle of a job, see this answer of mine.
I'm seeking for an advice regarding the best practice of starting (java) programs from shell scripts.
Currently the practice within our firm seems to be having a shell script which sets all the environment variables and launches the java (which is not important in this case) process on background similar to:
nohup $JAVA_CMD > $LOG_DIR/$LOG_FILE 2>&1 &
which is the last line of the script. We are launching single process.
One doubt I have is that return code of such shell process is always 0 even for the case when the program fails to start due to some Exception/Error. This makes it hard for monitoring tools - they can't rely on the shell exit code for example.
I found out this can be fixed by waiting for the process to end like:
nohup $JAVA_CMD > $LOG_DIR/$LOG_FILE 2>&1 &
wait $!
But my understanding is that this makes the last & completely useless since running:
nohup $JAVA_CMD > $LOG_DIR/$LOG_FILE 2>&1
will have the same effect.
So my question is: what is the best practice of launching programs from shell? Does the running on background have some benefits I'm overlooking?
You should look into at and batch, and possibly cron. These are all tools to run commands scripts, job streams non-interactively. at runs a job then emails stderr output back to the user - default behavior.
at -k now <<!
$JAVA_CMD > $LOG_DIR/$LOG_FILE 2>&1
!
The batch command will let you write a series of commands to a file, then execute the file as if it were stdin, you can also do this interactively.
cron jobs (crontab) run at specified times and dates, like every Monday at 0200. This does not seem to fit your question.
Try this:
http://www.thegeekstuff.com/2010/06/at-atq-atrm-batch-command-examples/
I'm trying to use a shell script to start a command. I don't care if/when/how/why it finishes. I want the process to start and run, but I want to be able to get back to my shell immediately...
You can just run the script in the background:
$ myscript &
Note that this is different from putting the & inside your script, which probably won't do what you want.
Everyone just forgot disown. So here is a summary:
& puts the job in the background.
Makes it block on attempting to read input, and
Makes the shell not wait for its completion.
disown removes the process from the shell's job control, but it still leaves it connected to the terminal.
One of the results is that the shell won't send it a SIGHUP(If the shell receives a SIGHUP, it also sends a SIGHUP to the process, which normally causes the process to terminate).
And obviously, it can only be applied to background jobs(because you cannot enter it when a foreground job is running).
nohup disconnects the process from the terminal, redirects its output to nohup.out and shields it from SIGHUP.
The process won't receive any sent SIGHUP.
Its completely independent from job control and could in principle be used also for foreground jobs(although that's not very useful).
Usually used with &(as a background job).
nohup cmd
doesn't hangup when you close the terminal. output by default goes to nohup.out
You can combine this with backgrounding,
nohup cmd &
and get rid of the output,
nohup cmd > /dev/null 2>&1 &
you can also disown a command. type cmd, Ctrl-Z, bg, disown
Alternatively, after you got the program running, you can hit Ctrl-Z which stops your program and then type
bg
which puts your last stopped program in the background. (Useful if your started something without '&' and still want it in the backgroung without restarting it)
screen -m -d $command$ starts the command in a detached session. You can use screen -r to attach to the started session. It is a wonderful tool, extremely useful also for remote sessions. Read more at man screen.
Bash is quite verbose when running jobs in the background:
$ echo toto&
toto
[1] 15922
[1]+ Done echo toto
Since I'm trying to run jobs in parallel and use the output, I'd like to find a way to silence bash. Is there a way to remove this superfluous output?
You can use parentheses to run a background command in a subshell, and that will silence the job control messages. For example:
(sleep 10 & )
Note: The following applies to interactive Bash sessions. In scripts, job-control messages are never printed.
There are 2 basic scenarios for silencing Bash's job-control messages:
Launch-and-forget:
CodeGnome's helpful answer answer suggests enclosing the background command in a simple subshell - e.g, (sleep 10 &) - which effectively silences job-control messages - both on job creation and on job termination.
This has an important side effect:
By using control operator & inside the subshell, you lose control of the background job - jobs won't list it, and neither %% (the spec. (ID) of the most recently launched job) nor $! (the PID of the (last) process launched (as part of) the most recent job) will reflect it.[1]
For launch-and-forget scenarios, this is not a problem:
You just fire off the background job,
and you let it finish on its own (and you trust that it runs correctly).
[1] Conceivably, you could go looking for the process yourself, by searching running processes for ones matching its command line, but that is cumbersome and not easy to make robust.
Launch-and-control-later:
If you want to remain in control of the job, so that you can later:
kill it, if need be.
synchronously wait (at some later point) for its completion,
a different approach is needed:
Silencing the creation job-control messages is handled below, but in order to silence the termination job-control messages categorically, you must turn the job-control shell option OFF:
set +m (set -m turns it back on)
Caveat: This is a global setting that has a number of important side effects, notably:
Stdin for background commands is then /dev/null rather than the current shell's.
The keyboard shortcuts for suspending (Ctrl-Z) and delay-suspending (Ctrl-Y) a foreground command are disabled.
For the full story, see man bash and (case-insensitively) search for occurrences of "job control".
To silence the creation job-control messages, enclose the background command in a group command and redirect the latter's stderr output to /dev/null
{ sleep 5 & } 2>/dev/null
The following example shows how to quietly launch a background job while retaining control of the job in principle.
$ set +m; { sleep 5 & } 2>/dev/null # turn job-control option off and launch quietly
$ jobs # shows the job just launched; it will complete quietly due to set +m
If you do not want to turn off the job-control option (set +m), the only way to silence the termination job-control message is to either kill the job or wait for it:
Caveat: There are two edge cases where this technique still produces output:
If the background command tries to read from stdin right away.
If the background command terminates right away.
To launch the job quietly (as above, but without set +m):
$ { sleep 5 & } 2>/dev/null
To wait for it quietly:
$ wait %% 2>/dev/null # use of %% is optional here
To kill it quietly:
{ kill %% && wait; } 2>/dev/null
The additional wait is necessary to make the termination job-control message that is normally displayed asynchronously by Bash (at the time of actual process termination, shortly after the kill) a synchronous output from wait, which then allows silencing.
But, as stated, if the job completes by itself, a job-control message will still be displayed.
Wrap it in a dummy script:
quiet.sh:
#!/bin/bash
$# &
then call it, passing your command to it as an argument:
./quiet.sh echo toto
You may need to play with quotes depending on your input.
Interactively, no. It will always display job status. You can influence when the status is shown using set -b.
There's nothing preventing you from using the output of your commands (via pipes, or storing it variables, etc). The job status is sent to the controlling terminal by the shell and doesn't mix with other I/O. If you're doing something complex with jobs, the solution is to write a separate script.
The job messages are only really a problem if you have, say, functions in your bashrc which make use of job control which you want to have direct access to your interactive environment. Unfortunately there's nothing you can do about it.
One solution (in bash anyway) is to route all the output to /dev/null
echo 'hello world' > /dev/null &
The above will not give you any output other than the id for the bg process.
I am using bash on Ubuntu. I would like to have a shell script open a program and continue on to the next line of the shell script, even though the program has not terminated.
Adding an & to a command places it in background.
example:
/path/to/foo
/path/to/bar # not executed untill foo is done
/path/to/foo & # in background
/path/to/bar & # executes as soon as foo is started
Read more about job-control here and here
Use something like this (my-long-running-process &) . This will launch your script as a separate process in the background.
You must run the process in the background, but you must enable job-control first. Otherwise, you cannot kill or bring the process to foreground if desired.
To enable job-control, execute:
set -m
To run some task in the background, execute:
task &
To manipulate the background task, use the jobspec syntax (%[n]). For example, to kill the last launched process, execute:
kill %
Note that enabling job-control is required only if you're actually running a script (as stated in the question). If running interactively, job-control is already enabled by default.
The manpage for bash has much more information in the JOB CONTROL section.
http://ubuntuforums.org/showthread.php?t=1657602
It looks like all you have to do is add a & at the end of the line.