I have a python script that starts a bunch of processes, but doesn't use the expect features to interact with processes. So I encounter issues where it seems the stdout buffer gets full and causes processes stdout to block. Is there a way to just have stdout/stderr redirected to /dev/null or just flushed? I have tried using logfile feature and opening /dev/null, but this still requires reading from process. Is there a way to do this?
Thanks.
You can direct all output to /dev/null so there will be no data waiting to be read. For example:
pexpect.spawn('/bin/bash', ['-c', '/your/command ... > /dev/null 2>&1'])
Related
According to this docker tutorial
What's the difference between
./my_first_process -D
./my_main_process &
They both seem unblocking to bash script and run in background
& tells the shell to put the command that precedes it into the background. -D is simply a flag that is passed to my_first_process and is interpreted by it; it has absolutely nothing whatsoever to do with the shell.
You will have to look into the documentation of my_first_process to see what -D does … it could mean anything. E.g. in npm, -D means "development", whereas in some other tools, it may mean "directory". In diff, it means "Output merged file to show `#ifdef NAME' diffs."
Some programs, by convention, take -D as an instruction to self-daemonize. Doing this looks something like the following:
Call fork(), and exit if it returns 0 (so only the child survives).
Close stdin, stdout and stderr if they are attached to the console (ideally, replacing their file descriptors with handles on /dev/null, so writes don't trigger an error).
Call setsid() to create a new session.
Call fork() again, and exit if it returns 0 again.
That's a lot more work than what just someprogram & does! A program that has self-daemonized can no longer log to the terminal, and will no longer be impacted if the terminal itself closes. That's not true of a program that's just started in the background.
To get something similar to the same behavior from bash, correct code would be something like:
someprogram </dev/null >/dev/null 2>&1 & disown -h
...wherein disown -h tells the shell not to pass along a SIGHUP to that process. It's also not uncommon to see the external tool nohup used for this purpose (though by default, it redirects stdout and stderr to a file called nohup.out if they're pointed at the TTY, the end purpose -- of making sure they're not pointed at the terminal, and thus that writes to them don't start failing if the terminal goes away -- is achieved):
nohup someprogram >/dev/null &
When I start some background process in the shell, for example:
geth --maxpeers 0 --rpc &
It returns something like:
[1] 1859
...without any out stream of this process. I do not understand what is it? And how can I get stdout of geth? There is information in documentation that stdout of background process is displayed in shell by default.
My shell is running in remote Ubuntu system.
The "&" directs the shell to run the command in the background. It uses the fork system call to create a sub-shell and run the job asynchronously.
The stdout and stderr should still be printed to the screen.
If you do not want to see any output on the screen, redirect both stdout and stderr to a file by:
geth --maxpeers 0 --rpc > logfile 2>&1 &
Regarding the first part of your question:
...without any out stream of this process. I do not understand what is
it?
This is part of the command execution environment (part of the shell itself) and is not the result of your script. (it is how the shell handles backgrounding your script, and keeping track of the process to allow pause and resume of jobs).
If you look at man bash under JOB CONTROL, it explains what you are seeing in detail, e.g.
The shell associates a job with each pipeline. It keeps a table
of currently executing jobs, which may be listed with the jobs
command. When bash starts a job asynchronously (in the background),
it prints a line that looks like:
[1] 25647
I do not understand what is it? [1] 1859
It is the output from Bash's job feature, which enables managing background processes (jobs), and it contains information about the job just started, printed to stderr:
1 is the job ID (which, prefixed with %, can be used with builtins such as kill and wait)
25647 is the PID (process ID) of the background process.
Read more in the JOB CONTROL section of man bash.
how can I get stdout of geth? There is information in documentation that stdout of background process is displayed in shell by default.
Indeed, background jobs by default print their output to the current shell's stdout and stderr streams, but note that they do so asynchronously - that is, output from background jobs will appear as it is produced (potentially buffered), interleaved with output sent directly to the current shell, which can be disruptive.
You can apply redirections as usual to a background command in order to capture its output in file(s), as demonstrated in user3589054's helpful answer, but note that doing so will not silence the job-control message ([1] 1859 in the example above).
If you want to silence the job-control message on creation, use:
{ geth --maxpeers 0 --rpc & } 2>/dev/null
To silence the entire life cycle of a job, see this answer of mine.
I tried to redirect standard error to a file having restricted permissions. This is what I did:
exec 2> >(umask 077; exec > stderr.log)
The idea was to redirect standard error to a process, change the umask and redirect once more to the log file.
But it does not work. The command stalls and terminates with 141 after pressing return.
The Bash manual does not define "process list" in the manual.
Can anybody explain the failure?
You should use cat inside sub-process to write the data coming in stdin of the process inside (...) which is actually stderr of parent process:
exec 2> >(umask 077; cat > stderr.log)
Process substitution feeds the output of a process (or processes) into the stdin of another process. Just by doing exec > stderr.log you're merely redirecting stdout of sub-process to a file however you're not actually writing anything to stdout inside >(...)
To redirect stderr for a command to syslog I use a helper like
with_logger my-tag command arg1 ...
where with_logger is
#!/bin/bash
syslog_tag="$1"
shift
exec "$#" 2> >(exec /usr/bin/logger -t "$syslog_tag")
Here 2 exec calls and the process substitution is used to avoid having a bash process waiting for the command or logger command to finish. However this creates a zombie. That is, when the logger process exits after the command exits closing its stderr, nobody waited for the process. This results in the parent process receiving an unexpected signal about unknown child processes.
To solve this I suppose I have to somehow disown the >() process. Is there a way to do it?
Update to clarify the question
I need to invoke my wrapper script from another program, not from a bash script.
Update 2 - this was a wrong question
See the answer below.
I would just define a short shell function
to_logger () {
exec /usr/bin/logger -t "$1"
}
and call your code with the minimally longer
2> >(to_logger my-tag) command arg1 ...
This has several benefits:
The command can be any shell construct; you aren't passing the command as arguments to another command; you are just redirecting standard error of an arbitrary command.
You are spawning one fewer process to handle the logging.
My question was wrong.
In my setup I use supervisord to control few processes. As it has limited syslog support and does not allow to use different tags when redirecting processes' stderr to syslog, I use the above shell script for that. While testing the script I noticed CRIT reaped unknown pid <number> messages in the log for supervisord itself. I assumed that this was bad and tried to fix this.
But it turned out the messages are not critical at all. In fact supervisord was doing the proper job and in its latest source the message was changed from CRIT to INFO. So there is nothing to answer here as there are no issues with the script in question :)
I logged in to a remote server via ssh and started a php script. Appereantly, it will take 17 hours to complete, is there a way to break the connection but the keep the script executing? I didn't make any output redirection, so I am seeing all the output.
Can you stop the process right now? If so, launch screen, start the process and detach screen using ctrl-a then ctrl-d. Use screen -r to retrieve the session later.
This should be available in most distros, failing that, a package will definitely be available for you.
ctrl + z
will pause it. Than type
bg
to send it to background. Write down the PID of the process for later usage ;)
EDIT: I forgot, you have to execute
disown -$PID
where $PID is the pid of your process
after that, and the process will not be killed after you close the terminal.
you described it's important to protect script continuation. Unfortunately I don't know, you make any interaction with script and script is made by you.
continuation protects 'screen' command. your connection will break, but screen protect pseudo terminal, you can reconnect to this later, see man.
if you don't need operators interaction with script, you simply can put script to background at the start, and log complete output into log file. Simply use command:
nohup /where/is/your.script.php >output.log 2&>1 &
>output.log will redirect output into log file, 2&>1 will append error stream into output, effectively into log file. last & will put command into background. Notice, nohup command will detach process from terminal group.
At now you can safely exit from ssh shell. Because your script is out of terminal group, then it won't be killed. It will be rejoined from your shell process, into system INIT process. It is unix like system behavior. Complete output you can monitor using command
tail -f output.log #allways breakable by ^C, it is only watching
Using this method you do not need use ^Z , bg etc shell tricks for putting command to the background.
Notice, using redirection to nohup command is preferred. Otherwise nohup will auto redirect all outputs for you to nohup.out file in the current directory.
You can use screen.