I have an interesting situation. All code here is a functional pseudo-code example of the exact issue I am facing so no jokes about assign the output of date. I actually want to capture the output of a slower more resource dependent function, but date works well to show the functional obstacle I have run into.
I am writing bash script where I want to assign the output of a process to a variable like so:
RESPONSE=$(nice -n 19 date);
Now that gives me the RESPONSE in a nice variable, right? Okay, what if I want to get the process ID of the function called within $()? How would I do that? I assumed this would work:
RESPONSE=$(nice -n 19 date & PID=(`jobs -l | awk '{print $2}'`));
Which does give me the process ID in the variable PID, but then I no longer get the output sent to RESPONSE.
The code I am using as a functional example is this. This example works, but no PID; yes I am not assigning a PID but again this is an example:
RESPONSE=$(nice -n 19 date);
wait ${PID};
echo "${RESPONSE}";
echo "${PID}";
This example gives me a PID but no RESPONSE:
RESPONSE=$(nice -n 19 date & PID=(`jobs -l | awk '{print $2}'`));
wait ${PID};
echo "${RESPONSE}";
echo "${PID}";
Anyone know how I can get the value of RESPONSE with the PID as well?
Depending on exactly how you set it up, using RESPONSE=$(backgroundcommand) will either wait for the command to complete (in which case it's too late to get its PID), or won't wait for the command to complete (in which case its output won't exist yet, and this can't be assigned to RESPONSE). You're going to have to store the command's output someplace else (like a temporary file), and then collect it when the process finishes:
responsefile=$(mktemp -t response)
nice -n 19 date >$responsefile &
pid=$!
wait $pid
response=$(<$responsefile)
rm $responsefile
(Note that the $(<$responsefile) construct is only available in bash, not in plain posix shells. If you don't start the script with #!/bin/bash, use $(cat $responsefile) instead.)
This still may not do quite what you're looking for, because (at least as far as I can see) what you're looking for doesn't really make sense. The command's PID and output never exist at the same time, so getting both isn't really meaningful. The script I gave above technically gives you both, but the pid is irrelevant (the process has existed) by the time response gets assigned, so while you have the pid at the end, it's meaningless.
Related
I have this need to collect\log all the command lines that were used to start a process on my machine during the execution of a Perl script which happens to be a test automation script. This Perl script starts the executable in question (MySQL) multiple times with various command lines and I would like to inspect all of the command lines of those invocations. What would be the right way to do this? One possibility i see is run something like "ps -aux | grep mysqld | grep -v grep" in a loop in a shell script and capture the results in a file but then I would have to do some post processing on this and remove duplicates etc and I could possibly miss some process command lines because of timing issues. Is there a better way to achieve this.
Processing the ps output can always miss some processes. It will only capture the ones currently existing. The best way would be to modify the Perl script to log each command before or after it executes it.
If that's not an option, you can get the child pids of the perl script by running:
pgrep -P $pid -a
-a gives the full process command. $pid is the pid of the perl script. Then process just those.
You could use strace to log calls to execve.
$ strace -f -o strace.out -e execve perl -e 'system("echo hello")'
hello
$ egrep ' = 0$' strace.out
11232 execve("/usr/bin/perl", ["perl", "-e", "system(\"echo hello\")"], 0x7ffc6d8e3478 /* 55 vars */) = 0
11233 execve("/bin/echo", ["echo", "hello"], 0x55f388200cf0 /* 55 vars */) = 0
Note that strace.out will also show the failed execs (where execve returned -1), hence the egrep command to find the successful ones. A successful execve call does not return, but strace records it as if it returned 0.
Be aware that this is a relatively expensive solution because it is necessary to include the -f option (follow forks), as perl will be doing the exec call from forked subprocesses. This is applied recursively, so it means that your MySQL executable will itself run through strace. But for a one-off diagnostic test it might be acceptable.
Because of the need to use recursion, any exec calls done from your MySQL executable will also appear in the strace.out, and you will have to filter those out. But the PID is shown for all calls, and if you were to log also any fork or clone calls (i.e. strace -e execve,fork,clone), you would see both the parent and child PIDs, in the form <parent_pid> clone(......) = <child_pid> so then you should hopefully then have enough information to reconstruct the process tree and decide which processes you are interested in.
I have read several posts here about cases where pgrep 'seems' to return itself even though it never should. The key seems to be the difference between how bash and sh function. Except that in my case, I have confirmed that sh really is a link to bash.
I'm running on SuSE 12 x86_64
/bin/sh is a link to bash
/bin/bash is the real binary
I have a Ruby script which calls pgrep like this:
cmd="/usr/bin/pgrep -lf \"#{target}\""
pidList=`#{cmd}`
I need to use the full command line because I'm actually using an argument to uniquely identify a specific 'java' process.
Now, due to some unrelated foolishness, I almost immediately do a ps -p on each of the pids returned. For a while, this was causing me great grief because the ps would sometimes return nothing. Eventually I was able to catch a case where the ps on the pid returned the pgrep command. But it was the pgrep command itself, not something like sh -c "pgrep -f blah"
To recap:
pgrep never returns itself. But differences in sh vs bash can cause it to show a subshell. But I verified that sh is a link to bash, so there should be no difference in behavior.
What I suspect (and am looking for confirmation for) is that an extra subcommand is being created because of the Ruby backticks and that is what is (only sometimes.. timing issues?) being picked up by the pgrep command.
This has been a real pain and I want to make sure the fix I implement will truly make the problem go away. Given the code I'm working with, I'm either going to
append a | grep -v grep to the end of my command
throw out any results containing 'grep' while looping through the returned results within the Ruby script
I figure #2 is faster, but it still irks me that I have to filter out pgrep itself.
Am I on the right track or do you think something else is at play?
Thanks for your time!
The problem is not in shell flavor: the shell process which calls pgrep also shows among processes (and has the searched string in its full command), so we need to filter it out like this:
pgrep -f target | grep -v $$
The answer is already in the comments to my question, but I figure I'll close this out with an official answer.
The piece of information I was missing is that
When bash is invoked as sh it behaves as a POSIX sh, not bash. – Jörg W Mittag Jan 23 at 23:31
So yes, pgrep was behaving normally. But when you call it from a Ruby script via backticks, you still need to filter out 'pgrep'
I tried to use herestrings to automate a script input and hence wanted to quit "more" of a file in it.
I've used:
$ echo q | more big_file.txt
$ more big_file.txt <<< q
$ yes q | more big_file.txt
$ echo -e "q\n" | more build.txt
but "more" command fails to quit.
Generally, the above mentioned input methods work for other commands in bash, but more seems to be exceptional,
Any idea what makes this a foul attempt?
NOTE: I don't want the data out of "more", but to quit "more" through an automated sequence is the target
When it detects it's running on a terminal, more only takes its input from it. This is so you can run things like:
$ cat multiple_files*.txt | more
(When it's not on a terminal, it doesn't even page, it degrades to behaving like cat.)
Seriously though, whatever your misguided reason for wanting to keep more in the loop is, you're not going to have it quit voluntarily with anything else than terminal input. So:
either you fake it. E.g. with expect
or you have it quit nonvoluntary. With a signal. Try SIGQUIT, move to SIGTERM, fall down to SIGKILL as a last resort.
I am trying to write a task-runner for command line. No rationale. Just wanted to do it. Basically it just runs a command, stores the output in a file (instead of stdout) and meanwhile prints a progress indicator of sorts on stdout and when its all done, prints Completed ($TIME_HERE).
Here's the code:
#!/bin/bash
task() {
TIMEFORMAT="%E"
COMMAND=$1
printf "\033[0;33m${2:-$COMMAND}\033[0m\n"
while true
do
for i in 1 2 3 4 5
do
printf '.'
sleep 0.5
done
printf "\b\b\b\b\b \b\b\b\b\b"
sleep 0.5
done &
WHILE=$!
EXECTIME=$({ TIMEFORMAT='%E';time $COMMAND >log; } 2>&1)
kill -9 $WHILE
echo $EXECTIME
#printf "\rCompleted (${EXECTIME}s)\n"
}
There are some unnecessarily fancy bits in there I admit. But I went through tons of StackOverflow questions to do different kinds of fancy stuff just to try it out. If it were to be applied anywhere, a lot of fat could be cut off. But it's not.
It is to be called like:
task "ping google.com -c 4" "Pinging google.com 4 times"
What it'll do is print Pinging google.com 4 times in yellow color, then on the next line, print a period. Then print another period every .5 seconds. After five periods, start from the beginning of the same line and repeat this until the command is complete. Then it's supposed to print Complete ($TIME_HERE) with (obviously) the time it took to execute the command in place of $TIME_HERE. (I've commented that part out, the current version would just print the time).
The Issue
The issue is that that instead of the execution time, something very weird gets printed. It's probably something stupid I'm doing. But I don't know where that problem originates from. Here's the output.
$ sh taskrunner.sh
Pinging google.com 4 times
..0.00user 0.00system 0:03.51elapsed 0%CPU (0avgtext+0avgdata 996maxresident)k 0inputs+16outputs (0major+338minor)pagefaults 0swaps
Running COMMAND='ping google.com -c 4';EXECTIME=$({ TIMEFORMAT='%E';time $COMMAND >log; } 2>&1);echo $EXECTIME in a terminal works as expected, i.e. prints out the time (3.559s in my case.)
I have checked and /bin/sh is a symlink to dash. (However that shouldn't be a problem because my script runs in /bin/bash as per the shebang on the top.)
I'm looking to learn while solving this issue so a solution with explanation will be cool. T. Hanks. :)
When you invoke a script with:
sh scriptname
the script is passed to sh (dash in your case), which will ignore the shebang line. (In a shell script, a shebang is a comment, since it starts with a #. That's not a coincidence.)
Shebang lines are only interpreted for commands started as commands, since they are interpreted by the system's command launcher, not by the shell.
By the way, your invocation of time does not correctly separate the output of the time builtin from any output the timed command might sent to stderr. I think you'd be better with:
EXECTIME=$({ TIMEFORMAT=%E; time $COMMAND >log.out 2>log.err; } 2>&1)
but that isn't sufficient. You will continue to run into the standard problems with trying to put commands into string variables, which is that it only works with very simple commands. See the Bash FAQ. Or look at some of these answers:
How to escape a variable in bash when passing to command line argument
bash quotes in variable treated different when expanded to command
Preserve argument splitting when storing command with whitespaces in variable
find command fusses on -exec arg
Using an environment variable to pass arguments to a command
(Or probably hundreds of other similar answers.)
I want to execute Program_A, and have it output examined by Program_B. e.g.
$ Program_A | Program_B
Within Program_B, I would like to be able to terminate Program_A if a certain condition matched.
I am looking for a way to do this in BASH (have a solution in python with popen).
Also, pidof Program_A and alike are not a good solution since there can be many instances of it and only a particular one shall be terminated.
You can get the process id of Program_A in the Program_A itself and print that as the first line and other outputs to Program_B. If you do this then you can kill the Program_A with kill call from Program_B
In Program_B you could close() stdin when you detect the closing condition. It will cause Program_A to receive error upon write() and possibly terminate.
Now, the Program_A may choose to ignore those errors (I don't know if it's your own application or a pre-compiled tool) and here comes the funny part. In Program_B you can examine where does your stdin come from:
my_input=`readlink /proc/$$/fd/0`
then, if it's a something like "pipe:[134414]", find any other process whose stdout equals
case "$my_input" in
pipe:*)
for p in /proc/[0-9]*
do
if [ "$my_input" = "`readlink $p/fd/1`" ]
then
bad_guy=`sed -e 's:.*/::' <<< "$p"`
echo "Would you please finish, Mr. $bad_guy ?!"
kill $bad_guy
break
fi
done
;;
*)
echo "We're good";;
esac