I have a really huge archive, which must be extracted file by file and further processed. I don't have enough memory to extract the whole archive (neither in RAM nor on the flash) - thus I wrote a small application which stops (raise(SIGSTOP)) after each extracted file. An equivalent could be this code:
#include <stdio.h>
#include <signal.h>
int main() {
printf("started.\n"); fflush(stdout);
sleep(2); // extracting archive
printf("stopping\n"); fflush(stdout);
raise(SIGSTOP); // stopping
printf("resume + done\n"); fflush(stdout);
return 0;
}
This works fine while I execute this in the terminal:
$ gcc -o dosleep main.c
$ ./dosleep
started.
stopping
[1]+ Stopped ./dosleep
$ fg
./dosleep
resume + done
$
But when calling the method from a script, the command never returns:
$ cat doit.sh
#!/bin/sh
echo "STARTING"
./dosleep
echo "BACK"
$ ./doit.sh
STARTING
started.
stopping
^C^C^C^C^C^C
Why do terminal and script behave so differently? Is there a way to change this behavior?
Thanks, Karl
Job control is off-by-default in noninteractive shells, but you can enable it explicitly:
set -m
Thus, if we modify your script to add either the set -m line or a -m on the shebang as follows:
#!/bin/bash -m
./start-delay
echo BACK
...then BACK is emitted after stopping.
Quoting from the bash man page section on set, emphasis added:
-m Monitor mode. Job control is enabled. This option is on by default for interactive shells on systems that support it (see JOB CONTROL above). All processes run in a separate process group. When a background job completes, the shell prints a line containing its exit status.
Related
I want to monitor the realtime output of a program that I will start. I am trying to do this by redirecting the output of the program to a pipe and then reading the pipe from a monitoring script.
./program >> apipe
then from the monitoring script
cat apipe
However due to the buffer in >> there is no output. Anyway I can disable this buffer? I am running on a barebones embedded system (petalinux) so I don't have access to unbuffer, script, or stdbuf to help me out.
I have tried the scripts on another platform where unbuffer is available it works as I expect.
Any way I can configure this buffer, or use another binary to redirect?
Edit: I do not have access to the source code of the command I am trying to run. It is a legacy binary.
If you don't have access to stdbuf, you might as well simulate it and unbuffer the stdout manually with gdb (assuming obviously you have access to gdb).
Let's take a look at how stdbuf actually operates. The stdbuf GNU coreutils command basically only injects libstdbuf in the user program by setting LD_PRELOAD environment variable. (Irrelevant, but for the record, options are passed via _STDBUF_E/_STDBUF_I/_STDBUF_O env vars.)
Then, when the libstdbuf is run, it calls setvbuf libc function (which in turn executes the underlaying syscall) on appropriate file descriptors (stdin/stdout/stderr), with the appropriate mode (fully buffered, line buffered, or unbuffered).
Declaration for setvbuf is in stdio.h, available with man 3 setvbuf:
#include <stdio.h>
int setvbuf(FILE *stream, char *buf, int mode, size_t size);
Values for mode are: _IONBF, _IOLBF, _IOFBF, as defined in stdio.h. We are here only interested in the unbuffered mode: _IONBF. It has a value of 2 (you can check your /usr/include/stdio.h).
Unbuffer script
So, to unbuffer a stdout for some process, we just need to call:
setvbuf(stdout, NULL, _IONBF, 0)
We can easily do that with gdb. Let's make a script we can call, unbuffer-stdout.sh:
#!/bin/bash
# usage: unbuffer-stdout.sh PID
gdb --pid "$1" -ex "call setvbuf(stdout, 0, 2, 0)" --batch
Then, we can call it like:
$ ./unbuffer-stdout.sh "$(pgrep -f my-program-name)"
(You'll probably need sudo to run it as root.)
Testing
We can use this simple Python program with buffered standard output (if not called with -u, and with unset PYTHONUNBUFFERED), writer.py:
#!/usr/bin/python
import sys, time
while True:
sys.stdout.write("output")
time.sleep(0.5)
Run it with:
$ ./writer.py >/tmp/output &
$ tailf /tmp/output
and observe no output appears until we run:
$ sudo ./unbuffer-stdout.sh "$(pgrep -f writer.py)"
I want to monitor the realtime output of a program that I will start. I am trying to do this by redirecting the output of the program to a pipe and then reading the pipe from a monitoring script.
./program >> apipe
then from the monitoring script
cat apipe
However due to the buffer in >> there is no output. Anyway I can disable this buffer? I am running on a barebones embedded system (petalinux) so I don't have access to unbuffer, script, or stdbuf to help me out.
I have tried the scripts on another platform where unbuffer is available it works as I expect.
Any way I can configure this buffer, or use another binary to redirect?
Edit: I do not have access to the source code of the command I am trying to run. It is a legacy binary.
If you don't have access to stdbuf, you might as well simulate it and unbuffer the stdout manually with gdb (assuming obviously you have access to gdb).
Let's take a look at how stdbuf actually operates. The stdbuf GNU coreutils command basically only injects libstdbuf in the user program by setting LD_PRELOAD environment variable. (Irrelevant, but for the record, options are passed via _STDBUF_E/_STDBUF_I/_STDBUF_O env vars.)
Then, when the libstdbuf is run, it calls setvbuf libc function (which in turn executes the underlaying syscall) on appropriate file descriptors (stdin/stdout/stderr), with the appropriate mode (fully buffered, line buffered, or unbuffered).
Declaration for setvbuf is in stdio.h, available with man 3 setvbuf:
#include <stdio.h>
int setvbuf(FILE *stream, char *buf, int mode, size_t size);
Values for mode are: _IONBF, _IOLBF, _IOFBF, as defined in stdio.h. We are here only interested in the unbuffered mode: _IONBF. It has a value of 2 (you can check your /usr/include/stdio.h).
Unbuffer script
So, to unbuffer a stdout for some process, we just need to call:
setvbuf(stdout, NULL, _IONBF, 0)
We can easily do that with gdb. Let's make a script we can call, unbuffer-stdout.sh:
#!/bin/bash
# usage: unbuffer-stdout.sh PID
gdb --pid "$1" -ex "call setvbuf(stdout, 0, 2, 0)" --batch
Then, we can call it like:
$ ./unbuffer-stdout.sh "$(pgrep -f my-program-name)"
(You'll probably need sudo to run it as root.)
Testing
We can use this simple Python program with buffered standard output (if not called with -u, and with unset PYTHONUNBUFFERED), writer.py:
#!/usr/bin/python
import sys, time
while True:
sys.stdout.write("output")
time.sleep(0.5)
Run it with:
$ ./writer.py >/tmp/output &
$ tailf /tmp/output
and observe no output appears until we run:
$ sudo ./unbuffer-stdout.sh "$(pgrep -f writer.py)"
Editor's note: The OP is ultimately looking to package the code from this answer
as a script. Said code creates a stay-open FIFO from which a background command reads data to process as it arrives.
It works if I type it in the terminal, but it won't work if I enter those commands in a script file and run it.
#!/bin/bash
cat >a&
pid=$!
it seems that the program is stuck at cat>a&
$pid has no value after running the script, but the cat process seems to exist.
cdarke's answer contains the crucial pointer: your script mustn't run in a child process, so you have to source it.
Based on the question you linked to, it sounds like you're trying to do the following:
Open a FIFO (named pipe).
Keep that FIFO open indefinitely.
Make a background command read from that FIFO whenever new data is sent to it.
See bottom for a working solution.
As for an explanation of your symptoms:
Running your script NOT sourced (NOT with .) means that the script runs in a child process, which has the following implications:
Variables defined in the script are only visible inside that script, and the variables cease to exist altogether when the script finishes running.
That's why you didn't see the script's $myPid variable after running the script.
When the script finishes running, its background tasks (cat >a&) are killed (as cdarke explains, the SIGHUP signal is sent to them; any process that doesn't explicitly trap that signal is terminated).
This contradicts your claim that the cat process continues to exist, but my guess is that you mistook an interactively started cat process for one started by a script.
By contrast, any FIFO created by your script (with mkfifo) does persist after the script exits (a FIFO behaves like a file - it persists until you explicitly delete it).
However, when you write to that FIFO without another process reading from it, the writing command will block and thus appear to hang (the writing process blocks until another process reads the data from the FIFO).
That's probably what happened in your case: because the script's background processes were killed, no one was reading from the FIFO, causing an attempt to write to it to block. You incorrectly surmised that it was the cat >a& command that was getting "stuck".
The following script, when sourced, adds functions to the current shell for setting up and cleaning up a stay-open FIFO with a background command that processes data as it arrives. Save it as file bgfifo_funcs:
#!/usr/bin/env bash
[[ $0 != "$BASH_SOURCE" ]] || { echo "ERROR: This script must be SOURCED." >&2; exit 2; }
# Set up a background FIFO with a command listening for input.
# E.g.:
# bgfifo_setup bgfifo "sed 's/^/# /'"
# echo 'hi' > bgfifo # -> '# hi'
# bgfifo_cleanup
bgfifo_setup() {
(( $# == 2 )) || { echo "ERROR: usage: bgfifo_setup <fifo-file> <command>" >&2; return 2; }
local fifoFile=$1 cmd=$2
# Create the FIFO file.
mkfifo "$fifoFile" || return
# Use a dummy background command that keeps the FIFO *open*.
# Without this, it would be closed after the first time you write to it.
# NOTE: This call inevitably outputs a job control message that looks
# something like this:
# [1]+ Stopped cat > ...
{ cat > "$fifoFile" & } 2>/dev/null
# Note: The keep-the-FIFO-open `cat` PID is the only one we need to save for
# later cleanup.
# The background processing command launched below will terminate
# automatically then FIFO is closed when the `cat` process is killed.
__bgfifo_pid=$!
# Now launch the actual background command that should read from the FIFO
# whenever data is sent.
{ eval "$cmd" < "$fifoFile" & } 2>/dev/null || return
# Save the *full* path of the FIFO file in a global variable for reliable
# cleanup later.
__bgfifo_file=$fifoFile
[[ $__bgfifo_file == /* ]] || __bgfifo_file="$PWD/$__bgfifo_file"
echo "FIFO '$fifoFile' set up, awaiting input for: $cmd"
echo "(Ignore the '[1]+ Stopped ...' message below.)"
}
# Cleanup function that you must call when done, to remove
# the FIFO file and kill the background commands.
bgfifo_cleanup() {
[[ -n $__bgfifo_file ]] || { echo "(Nothing to clean up.)"; return 0; }
echo "Removing FIFO '$__bgfifo_file' and terminating associated background processes..."
rm "$__bgfifo_file"
kill $__bgfifo_pid # Note: We let the job control messages display.
unset __bgfifo_file __bgfifo_pid
return 0
}
Then, source script bgfifo_funcs, using the . shell builtin:
. bgfifo_funcs
Sourcing executes the script in the current shell (rather than in a child process that terminates after the script has run), and thus makes the script's functions and variables available to the current shell. Functions by definition run in the current shell, so any background commands started from functions stay alive.
Now you can set up a stay-open FIFO with a background process that processes input as it arrives as follows:
# Set up FIFO 'bgfifo in the current dir. and process lines sent to it
# with a sample Sed command that simply prepends '# ' to every line.
$ bgfifo_setup bgfifo "sed 's/^/# /'"
# Send sample data to the FIFO.
$ echo 'Hi.' > bgfifo
# Hi.
# ...
$ echo 'Hi again.' > bgfifo
# Hi again.
# ...
# Clean up when done.
$ bgfifo_cleanup
The reason that cat >a "hangs" is because it is reading from the standard input stream (stdin, file descriptor zero), which defaults to the keyboard.
Adding the & causes it to run in background, which disconnects from the keyboard. Normally that would leave a suspended job in background, but, since you exit your script, its background tasks are killed (sends a SIGHUP signal).
EDIT: although I followed the link in the question, it was not stated originally that the OP was actually using a FIFO at that stage. So thanks to #mklement0.
I don't understand what you are trying to do here, but I suspect you need to run it as a "sourced" file, as follows:
. gash.sh
Where gash.sh is the name of your script. Note the preceding .
You need to specify a file with "cat":
#!/bin/bash
cat SOMEFILE >a &
pid=$!
echo PID $pid
Although that seems a bit silly - why not just "cp" the file (cp SOMEFILE a)?
Q: What exactly are you trying to accomplish?
I'd like to check whether a program is installed on an UNIX system.
I could use commands like:
command -v,
hash,
type,
which...
...and all of them are already mentioned in this answer.
However, none of them work if I want to test, as a normal user, whether I or any superuser can run given command.
Here's an example of what I mean:
dummy:~$ command -v poweroff; echo $?
1
dummy:~$ su
root:~# command -v poweroff; echo $?
/sbin/poweroff
0
As you see, normal user didn't find out about the existence of the poweroff command. Note that dummy users can freely see what's in /sbin anyway.
Source of the problem
The reason the commands you tried do not work is that they only look for executables in $PATH variable. First, let's test our hypothesis.
dummy:~$ mkdir test
dummy:~$ cd test
dummy:~/test$ echo '#!/bin/sh' >test.sh
dummy:~/test$ chmod +x test.sh
dummy:~/test$ cd
dummy:~$ command -v test.sh
dummy:~$ PATH+=:/home/dummy/test/
dummy:~$ command -v test.sh
/home/dummy/test/test.sh
This confirms my statement above.
Now, let's have a look what $PATH looks like for different users:
dummy:~$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
dummy:~$ su
root:~# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
So in order to check whether given command is available to given user (in your question, namely: root), you need to know his $PATH environment variable.
The solution
Values of such environment variables on Debian can be usually found in /etc/profile and in /etc/environment/ files. There is no easy way to get these values by fishing them out of files.
The most basic solution is to temporarily add known directories to your $PATH variable and then use command -v:
dummy~$ OLDPATH=$PATH
dummy~$ PATH=$OLDPATH:/sbin:/usr/sbin/:/usr/local/sbin/
dummy~$ command -v poweroff
/sbin/poweroff
dummy~$ PATH=$OLDPATH
There is one problem with this solution: if you want to be portable, you don't really know what are the folders that you should concatenate. In most cases this approach should be sufficient, though.
Alternative solution
What you can do instead is to write a script program that makes use of setuid bit. Setuid bit is a somewhat hidden feature of Linux operating systems that allows programs to be executed on their owner privileges. So you write a program that executes some commands like superuser would, except that it can be run by normal users. That way you can see output of command -v poweroff like a root would do.
Unfortunately, stuff that uses shebang can't have setuid bit, so you cannot create a shell script for this and you need a program in C. Here's an example program that would do the job:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char** argv)
{
if (argc <= 1)
{
fprintf(stderr, "No arguments.\n");
return 1;
}
//validate the argv
char* prog = argv[1];
int i;
for (i = 0; i < strlen(prog); i ++)
{
if (prog[i] < 'a' || prog[i] > 'z')
{
fprintf(stderr, "%s contains invalid characters (%c), exiting.", prog, prog[i]);
return 1;
}
}
//here's the `which` command. We start it in new interactive shell,
//since this program inherits environment variables from its
//parent shell. We need to start *new* shell that will initialize
//and overwrite existing PATH environment variable.
char* command = (char*) malloc(strlen(prog) + 30);
if (!command)
{
fprintf(stderr, "No memory!\n");
return 1;
}
sprintf(command, "bash -cli 'command -v %s'", prog);
int exists = 0;
//first we try to execute the command as a dummy user.
exists |= system(command) == 0;
if (!exists)
{
//then we try to execute the command as a root user.
setuid(0);
exists |= system(command) == 0;
}
return exists ? 0 : 1;
}
Security note: the version above has very simple argument validation (it lets through only strings matching ^[a-z]*$). Real program should probably include better validation.
Testing
Suppose we saved the file in test.c. We compile it and add setuid bit:
root:~# gcc ./test.c -o ./test
root:~# chown root:root ./test
root:~# chmod 4755 ./test
Note that chown goes before chmod. The 4 before usual 755 pattern is the setuid bit.
Now we can test the program as a normal user.
dummy:~$ ./test ls; echo $?
alias ls='ls -vhF1 --color=auto --group-directories-first'
0
dummy:~$ ./test blah; echo $?
1
dummy:~$ ./test poweroff; echo $?
/sbin/poweroff
0
And best of all - it's portable enough to work on cygwin with no problems. :)
The real answer is that you can't satisfy the "any superuser" aspect IF you mean: "Does this command appear in the search path of any user with sudo access?". The reason is that you'd have to run each user's startup scripts to find out what his search path ends up being - many users will include their own ~/bin or ~/pod/abi/x86_64-ubu-1204/bin or whatever and god knows what else (/afs//bin, anyone?) - and many startup scripts have side effects that could turn the whole thing into a real mess, including generating logs, starting up daemons of various kinds, and so on. You'll really be in trouble if one of those users' startup scripts tries to run your new command itself, since they'll then recurse and inflict a denial-of-service attack on your own system.
What you can test more safely is:
can anyone can run a command given by full pathname? (skips the startup script insanity)
can the current user run the simple (not fully pathed) command?
can the current user run the simple command with sudo? (this makes some assumptions about root's startup scripts)
can any user running a default environment run the command? (use a dummy user with a known setup).
On a larger system with thousands of users, the "anyone" option isn't practical. Mature sites will NOT want a root-capable command running arbitrary users' scripts, either, even those of sudo-enabled, generally trustworthy admins.
I've searched for a while but i can't either find an answer or come up with a solution of my own, so I turn to you guys. First question I actually ask here :)
I would like to run several instances of the same program, and redirect each of these programs' standard output to a file that contains that same process' pid, something like:
my_program > <pid of the instance of my_program that is called in this command>.log
I'm aware that this is not even close of the way to go :P I have tinkered around with exec and $PPID but to no avail. My bash-fu is weak :| please help me, point me somewhere! Thanks!
The problem here is that each new process started by bash gets new PID and you have to redirect the output of that process before starting it. But you cannot tell what PID will be assigned to that process by OS.
The solution to this problem is not to start a new process, but replace the existing bash process with a new one using exec.
Here is an example. First, we write a basic C program that prints its PID:
// printpid.c
#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
int main()
{
printf ("C process pid is %d\n", getpid());
return 0;
}
Then we write a simple bash script that will print its PID and replace itself with this program using exec:
#!/bin/bash
# printpid.sh
echo Bash process PID is $$
exec ./printpid > $$.log
Now, let's write a script that will call this printpid.sh script multiple times:
#!/bin/bash
# example.sh
./printpid.sh
./printpid.sh
./printpid.sh
Now, let's make sure it works:
$ ls
example.sh printpid printpid.c printpid.sh
$ ./example.sh
Bash process PID is 6397
Bash process PID is 6398
Bash process PID is 6399
$ ls
6397.log 6398.log 6399.log example.sh printpid printpid.c printpid.sh
$ cat 6397.log
C process pid is 6397
$ cat 6398.log
C process pid is 6398
$ cat 6399.log
C process pid is 6399
$
Be aware that when you are using exec you cannot put anything else after that in the script as bash shell replaces itself with a new process specified as command line arguments for exec.
Good luck hacking!
If you have bash 3 or newer, you could combine subshells, exec and ${BASHPID}
That is, create a subshell with ( ), then set up redirection to <pid of subshell>.log and exec my_program, which should replace the process-image of the subshell, inheriting it's pid (among other things).
( exec my_program >${BASHPID}.log )