I am running a for loop in which a command is run in background using &. In the end i want all commands to return value..
Here is the code i tried
for((i=0 ;i<3;i++)) {
// curl command which returns a value &
}
wait
// next piece of code
I want to get all three returned value and then proceed.. But the wait command does not wait for background processes to complete and runs the next part of code. I need the returned values to proceed..
Shell builtins have documentation accessible with help BUILTIN_NAME.
help wait yields:
wait: wait [-n] [id ...]
Wait for job completion and return exit status.
Waits for each process identified by an ID, which may be a process ID or a
job specification, and reports its termination status. If ID is not
given, waits for all currently active child processes, and the return
status is zero. If ID is a a job specification, waits for all processes
in that job's pipeline.
If the -n option is supplied, waits for the next job to terminate and
returns its exit status.
Exit Status:
Returns the status of the last ID; fails if ID is invalid or an invalid
option is given.
which implies that to get the return statuses, you need to save the pid and then wait on each pid, using wait $THE_PID.
Example:
sl() { sleep $1; echo $1; return $(($1+42)); }
pids=(); for((i=0;i<3;i++)); do sl $i & pids+=($!); done;
for pid in ${pids[#]}; do wait $pid; echo ret=$?; done
Example output:
0
ret=42
1
ret=43
2
ret=44
Edit:
With curl, don't forget to pass -f (--fail) to make sure the process will fail if the HTTP request did:
CURL Example:
#!/bin/bash
URIs=(
https://pastebin.com/raw/w36QWU3D
https://pastebin.com/raw/NONEXISTENT
https://pastebin.com/raw/M9znaBB2
)
pids=(); for((i=0;i<3;i++)); do
curl -fL "${URIs[$i]}" &>/dev/null &
pids+=($!)
done
for pid in "${pids[#]}"; do
wait $pid
echo ret=$?
done
CURL Example output:
ret=0
ret=22
ret=0
GNU Parallel is a great way to do high-latency things like curl in parallel.
parallel curl --head {} ::: www.google.com www.hp.com www.ibm.com
Or, filtering results:
parallel curl --head -s {} ::: www.google.com www.hp.com www.ibm.com | grep '^HTTP'
HTTP/1.1 302 Found
HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
Here is another example:
parallel -k 'echo -n Starting {} ...; sleep 5; echo done.' ::: 1 2 3 4
Starting 1 ...done.
Starting 2 ...done.
Starting 3 ...done.
Starting 4 ...done.
I have troubles with the output of this simple query:
select
pid,
state
from pg_stat_activity
where datname = 'My_DB_name'
while running it different ways:
In IDE
Via running psql in terminal
In bash script:
QUERY="copy (select pid, state from pg_stat_activity where datname = 'My_DB_name') to stdout with csv"
psql -h host -U user -d database -t -c "$QUERY" >> result
1 and 2 return results as I need them:
1:
pid state
------ -----------------------------
23126 idle
25573 active
2642 active
20420 idle
23391 idle
5339 idle
7710 idle
1558 idle
12506 idle
2862 active
716 active
9834 idle in transaction (aborted)
2:
pid | state
-------+-------------------------------
23126 | idle
25573 | idle
2642 | active
20420 | idle
23391 | idle
5339 | active
7710 | idle
1558 | idle
12506 | idle
2211 | active
716 | active
9834 | idle in transaction (aborted)
3 is weird - it doesnt give me any state name except 'active'
23126,
25573,
2642,
20420,
23391,
5339,
7710,
1558,
12506,
1660,active
716,active
1927,active
9834,
What am I missing? How to get all the state names via bash script?
pg_stat_activity is a catalog view that will show different content depending on whether you're logged in as a superuser, or as a non-privileged user.
From your output, it looks like you're logged in as superuser in #1 and #2, but as a normal user in #3.
I'd like to automatically kill a command after a certain amount of time. I have in mind an interface like this:
% constrain 300 ./foo args
Which would run "./foo" with "args" but automatically kill it if it's still running after 5 minutes.
It might be useful to generalize the idea to other constraints, such as autokilling a process if it uses too much memory.
Are there any existing tools that do that, or has anyone written such a thing?
ADDED: Jonathan's solution is precisely what I had in mind and it works like a charm on linux, but I can't get it to work on Mac OSX. I got rid of the SIGRTMIN which lets it compile fine, but the signal just doesn't get sent to the child process. Anyone know how to make this work on Mac?
[Added: Note that an update is available from Jonathan that works on Mac and elsewhere.]
GNU Coreutils includes the timeout command, installed by default on many systems.
https://www.gnu.org/software/coreutils/manual/html_node/timeout-invocation.html
To watch free -m for one minute, then kill it by sending a TERM signal:
timeout 1m watch free -m
Maybe I'm not understanding the question, but this sounds doable directly, at least in bash:
( /path/to/slow command with options ) & sleep 5 ; kill $!
This runs the first command, inside the parenthesis, for five seconds, and then kills it. The entire operation runs synchronously, i.e. you won't be able to use your shell while it is busy waiting for the slow command. If that is not what you wanted, it should be possible to add another &.
The $! variable is a Bash builtin that contains the process ID of the most recently started subshell. It is important to not have the & inside the parenthesis, doing it that way loses the process ID.
I've arrived rather late to this party, but I don't see my favorite trick listed in the answers.
Under *NIX, an alarm(2) is inherited across an execve(2) and SIGALRM is fatal by default. So, you can often simply:
$ doalarm () { perl -e 'alarm shift; exec #ARGV' "$#"; } # define a helper function
$ doalarm 300 ./foo.sh args
or install a trivial C wrapper to do that for you.
Advantages Only one PID is involved, and the mechanism is simple. You won't kill the wrong process if, for example, ./foo.sh exited "too quickly" and its PID was re-used. You don't need several shell subprocesses working in concert, which can be done correctly but is rather race-prone.
Disadvantages The time-constrained process cannot manipulate its alarm clock (e.g., alarm(2), ualarm(2), setitimer(2)), since this would likely clear the inherited alarm. Obviously, neither can it block or ignore SIGALRM, though the same can be said of SIGINT, SIGTERM, etc. for some other approaches.
Some (very old, I think) systems implement sleep(2) in terms of alarm(2), and, even today, some programmers use alarm(2) as a crude internal timeout mechanism for I/O and other operations. In my experience, however, this technique is applicable to the vast majority of processes you want to time limit.
There is also ulimit, which can be used to limit the execution time available to sub-processes.
ulimit -t 10
Limits the process to 10 seconds of CPU time.
To actually use it to limit a new process, rather than the current process, you may wish to use a wrapper script:
#! /usr/bin/env python
import os
os.system("ulimit -t 10; other-command-here")
other-command can be any tool. I was running a Java, Python, C and Scheme versions of different sorting algorithms, and logging how long they took, whilst limiting execution time to 30 seconds. A Cocoa-Python application generated the various command lines - including the arguments - and collated the times into a CSV file, but it was really just fluff on top of the command provided above.
I have a program called timeout that does that - written in C, originally in 1989 but updated periodically since then.
Update: this code fails to compile on MacOS X because SIGRTMIN is not defined, and fails to timeout when run on MacOS X because the `signal()` function there resumes the `wait()` after the alarm times out - which is not the required behaviour. I have a new version of `timeout.c` which deals with both these problems (using `sigaction()` instead of `signal()`). As before, contact me for a 10K gzipped tar file with the source code and a manual page (see my profile).
/*
#(#)File: $RCSfile: timeout.c,v $
#(#)Version: $Revision: 4.6 $
#(#)Last changed: $Date: 2007/03/01 22:23:02 $
#(#)Purpose: Run command with timeout monitor
#(#)Author: J Leffler
#(#)Copyright: (C) JLSS 1989,1997,2003,2005-07
*/
#define _POSIX_SOURCE /* Enable kill() in <unistd.h> on Solaris 7 */
#define _XOPEN_SOURCE 500
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <errno.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>
#include "stderr.h"
#define CHILD 0
#define FORKFAIL -1
static const char usestr[] = "[-vV] -t time [-s signal] cmd [arg ...]";
#ifndef lint
/* Prevent over-aggressive optimizers from eliminating ID string */
const char jlss_id_timeout_c[] = "#(#)$Id: timeout.c,v 4.6 2007/03/01 22:23:02 jleffler Exp $";
#endif /* lint */
static void catcher(int signum)
{
return;
}
int main(int argc, char **argv)
{
pid_t pid;
int tm_out;
int kill_signal;
pid_t corpse;
int status;
int opt;
int vflag = 0;
err_setarg0(argv[0]);
opterr = 0;
tm_out = 0;
kill_signal = SIGTERM;
while ((opt = getopt(argc, argv, "vVt:s:")) != -1)
{
switch(opt)
{
case 'V':
err_version("TIMEOUT", &"#(#)$Revision: 4.6 $ ($Date: 2007/03/01 22:23:02 $)"[4]);
break;
case 's':
kill_signal = atoi(optarg);
if (kill_signal <= 0 || kill_signal >= SIGRTMIN)
err_error("signal number must be between 1 and %d\n", SIGRTMIN - 1);
break;
case 't':
tm_out = atoi(optarg);
if (tm_out <= 0)
err_error("time must be greater than zero (%s)\n", optarg);
break;
case 'v':
vflag = 1;
break;
default:
err_usage(usestr);
break;
}
}
if (optind >= argc || tm_out == 0)
err_usage(usestr);
if ((pid = fork()) == FORKFAIL)
err_syserr("failed to fork\n");
else if (pid == CHILD)
{
execvp(argv[optind], &argv[optind]);
err_syserr("failed to exec command %s\n", argv[optind]);
}
/* Must be parent -- wait for child to die */
if (vflag)
err_remark("time %d, signal %d, child PID %u\n", tm_out, kill_signal, (unsigned)pid);
signal(SIGALRM, catcher);
alarm((unsigned int)tm_out);
while ((corpse = wait(&status)) != pid && errno != ECHILD)
{
if (errno == EINTR)
{
/* Timed out -- kill child */
if (vflag)
err_remark("timed out - send signal %d to process %d\n", (int)kill_signal, (int)pid);
if (kill(pid, kill_signal) != 0)
err_syserr("sending signal %d to PID %d - ", kill_signal, pid);
corpse = wait(&status);
break;
}
}
alarm(0);
if (vflag)
{
if (corpse == (pid_t) -1)
err_syserr("no valid PID from waiting - ");
else
err_remark("child PID %u status 0x%04X\n", (unsigned)corpse, (unsigned)status);
}
if (corpse != pid)
status = 2; /* I don't know what happened! */
else if (WIFEXITED(status))
status = WEXITSTATUS(status);
else if (WIFSIGNALED(status))
status = WTERMSIG(status);
else
status = 2; /* I don't know what happened! */
return(status);
}
If you want the 'official' code for 'stderr.h' and 'stderr.c', contact me (see my profile).
Perl one liner, just for kicks:
perl -e '$s = shift; $SIG{ALRM} = sub { print STDERR "Timeout!\n"; kill INT => $p }; exec(#ARGV) unless $p = fork; alarm $s; waitpid $p, 0' 10 yes foo
This prints 'foo' for ten seconds, then times out. Replace '10' with any number of seconds, and 'yes foo' with any command.
The timeout command from Ubuntu/Debian when compiled from source to work on the Mac. Darwin
10.4.*
http://packages.ubuntu.com/lucid/timeout
My variation on the perl one-liner gives you the exit status without mucking with fork() and wait() and without the risk of killing the wrong process:
#!/bin/sh
# Usage: timelimit.sh secs cmd [ arg ... ]
exec perl -MPOSIX -e '$SIG{ALRM} = sub { print "timeout: #ARGV\n"; kill(SIGTERM, -$$); }; alarm shift; $exit = system #ARGV; exit(WIFEXITED($exit) ? WEXITSTATUS($exit) : WTERMSIG($exit));' "$#"
Basically the fork() and wait() are hidden inside system(). The SIGALRM is delivered to the parent process which then kills itself and its child by sending SIGTERM to the whole process group (-$$). In the unlikely event that the child exits and the child's pid gets reused before the kill() occurs, this will NOT kill the wrong process because the new process with the old child's pid will not be in the same process group of the parent perl process.
As an added benefit, the script also exits with what is probably the correct exit status.
#!/bin/sh
( some_slow_task ) & pid=$!
( sleep $TIMEOUT && kill -HUP $pid ) 2>/dev/null & watcher=$!
wait $pid 2>/dev/null && pkill -HUP -P $watcher
The watcher kills the slow task after given timeout; the script waits for the slow task and terminates the watcher.
Examples:
The slow task run more than 2 sec and was terminated
Slow task interrupted
( sleep 20 ) & pid=$!
( sleep 2 && kill -HUP $pid ) 2>/dev/null & watcher=$!
if wait $pid 2>/dev/null; then
echo "Slow task finished"
pkill -HUP -P $watcher
wait $watcher
else
echo "Slow task interrupted"
fi
This slow task finished before the given timeout
Slow task finished
( sleep 2 ) & pid=$!
( sleep 20 && kill -HUP $pid ) 2>/dev/null & watcher=$!
if wait $pid 2>/dev/null; then
echo "Slow task finished"
pkill -HUP -P $watcher
wait $watcher
else
echo "Slow task interrupted"
fi
Try something like:
# This function is called with a timeout (in seconds) and a pid.
# After the timeout expires, if the process still exists, it attempts
# to kill it.
function timeout() {
sleep $1
# kill -0 tests whether the process exists
if kill -0 $2 > /dev/null 2>&1 ; then
echo "killing process $2"
kill $2 > /dev/null 2>&1
else
echo "process $2 already completed"
fi
}
<your command> &
cpid=$!
timeout 3 $cpid
wait $cpid > /dev/null 2>&
exit $?
It has the downside that if your process' pid is reused within the timeout, it may kill the wrong process. This is highly unlikely, but you may be starting 20000+ processes per second. This could be fixed.
How about using the expect tool?
## run a command, aborting if timeout exceeded, e.g. timed-run 20 CMD ARGS ...
timed-run() {
# timeout in seconds
local tmout="$1"
shift
env CMD_TIMEOUT="$tmout" expect -f - "$#" <<"EOF"
# expect script follows
eval spawn -noecho $argv
set timeout $env(CMD_TIMEOUT)
expect {
timeout {
send_error "error: operation timed out\n"
exit 1
}
eof
}
EOF
}
pure bash:
#!/bin/bash
if [[ $# < 2 ]]; then
echo "Usage: $0 timeout cmd [options]"
exit 1
fi
TIMEOUT="$1"
shift
BOSSPID=$$
(
sleep $TIMEOUT
kill -9 -$BOSSPID
)&
TIMERPID=$!
trap "kill -9 $TIMERPID" EXIT
eval "$#"
I use "timelimit", which is a package available in the debian repository.
http://devel.ringlet.net/sysutils/timelimit/
A slight modification of the perl one-liner will get the exit status right.
perl -e '$s = shift; $SIG{ALRM} = sub { print STDERR "Timeout!\n"; kill INT => $p; exit 77 }; exec(#ARGV) unless $p = fork; alarm $s; waitpid $p, 0; exit ($? >> 8)' 10 yes foo
Basically, exit ($? >> 8) will forward the exit status of the subprocess. I just chose 77 at the exit status for timeout.
Isn't there a way to set a specific time with "at" to do this?
$ at 05:00 PM kill -9 $pid
Seems a lot simpler.
If you don't know what the pid number is going to be, I assume there's a way to script reading it with ps aux and grep, but not sure how to implement that.
$ | grep someprogram
tony 11585 0.0 0.0 3116 720 pts/1 S+ 11:39 0:00 grep someprogram
tony 22532 0.0 0.9 27344 14136 ? S Aug25 1:23 someprogram
Your script would have to read the pid and assign it a variable.
I'm not overly skilled, but assume this is doable.
I've been looking for a way to run a one time script that loads data into our database. Currently we're using dokku-alt for our development environment and we have a python script that runs to update our schema, data and functions we need available for our application. The problem that I'm facing is trying to find a way to run our script on application deployment through dokku-alt.
I've ventured into using a worker, but the workers themselves don't perform how I would expect them to. From what I've noticed is that a worker will terminate every process once it's complete. This is NOT what we need. We need to run the script once to load up our data and schema and close gracefully. We still want our web process to continue working, so the child process sending a kill signal to the other process.
So my question is, is there a way to run a script just one time on deployment without having to write a custom plugin?
05:23:07 schema.1 | started with pid 15
05:23:07 function.1 | started with pid 17
05:23:07 data.1 | started with pid 19
05:23:07 web.1 | started with pid 21
05:23:07 web.1 | Picked up JAVA_TOOL_OPTIONS: -Xmx384m -Xss512k -Dfile.encoding=UTF-8 -Djava.rmi.server.useCodebaseOnly=true
05:23:12 function.1 | Begin dbupdater
05:23:12 function.1 | pq://user:UcW3P587Eki8Fqrr#postgresql:5432/db
05:23:13 schema.1 | Begin dbupdater
05:23:13 data.1 | Begin dbupdater
05:23:13 data.1 | pq://user:UcW3P587Eki8Fqrr#postgresql:5432/db
05:23:13 schema.1 | pq://user:UcW3P587Eki8Fqrr#postgresql:5432/db
05:23:13 schema.1 | do (AccountCrosstabKey_create.sql)
05:23:13 schema.1 | Done
05:23:13 data.1 | do (Accountinfo_data.sql)
05:23:13 function.1 | do (Connectby_create.sql)
05:23:13 function.1 | Done
05:23:13 data.1 | Done
05:23:13 schema.1 | exited with code 0
05:23:13 system | sending SIGTERM to all processes
05:23:13 function.1 | terminated by SIGTERM
05:23:13 data.1 | terminated by SIGTERM
05:23:13 web.1 | terminated by SIGTERM
Python script:
#!/usr/bin/python
import os
import sys
import glob
import shlex
import subprocess
import postgresql
import postgresql.driver as pg_driver
try:
print('Begin dbupdater')
dbhost = os.environ.get('DATABASE_URL','localhost').replace('postgres://', 'pq://')
print(dbhost)
targetDir = sys.argv[1]
db = postgresql.open(dbhost)
os.chdir(targetDir)
currDir = os.getcwd()
for file in glob.glob("*.sql"):
sqlCmd = ''
with open(file,'r') as myfile:
sqlCmd = myfile.read().replace('DO', '').replace('$do$', '')
db.do('plpgsql',sqlCmd)
print('do (' + file + ')')
db.close()
print('Done')
except (ValueError, KeyError, TypeError) as error:
print (error)
You can execute this, to rum custom python script in dokku instance.
dokku --rm-container run [APP_NAME] python [your_script_name.py]
--rm-container flag delete the container after script finished.
Why I keeping getting error "Can’t make \"0.0\" into type number." number -1700 from "0.0" to number? If I remove as number, the display dialog always showing.
tell application "System Events"
repeat
set PID to unix id of process "JPEGmini"
set getCpuPercent to "ps aux | grep " & PID & " | grep -v grep | awk '{print $3}'"
set cpuPercent to (do shell script getCpuPercent) as number
if (cpuPercent) < 5 then
display dialog cpuPercent
end if
end repeat
end tell
The problem here is your ps command is returning too much information. To illustrate what I mean, as I write, my google chrome process has a pid of 916. Using the approach in the question, I could do ps aux | grep 916 to see this process. But this is not sufficient - grep will match any instance of the string "916" in the ps output, so if there happens to be a process with a pid of say 1916 or 9160, then this will match as well. Also ps aux lists lots of other stats, many of which could also contain the string "916". In fact if I run ps aux | grep -c 916 there are currently 58 lines which match!
So all we need to do is tell ps we are only interested in a specific pid:
$ ps -o%cpu -p 916 | grep '[[:digit:]]'
0.5
$
This will list only the cpu% for the process with the given $PID. Piping to grep '[[:digit:]]' is necessary to return just the numeric percentage and strip the "CPU" column header from the ps output.
Wrapping this up into your original script you would have:
tell application "System Events"
repeat
set PID to unix id of process "JPEGmini"
set getCpuPercent to "ps -o%cpu -p " & PID & " | grep '[[:digit:]]'"
set cpuPercent to (do shell script getCpuPercent) as number
if (cpuPercent) < 5 then
display dialog cpuPercent
end if
end repeat
end tell
I don't have JPEGmini installed, but this works fine for me on all other processes I have tried on my OSX 10.8.5 powerbook.
I tried your script with the process Safari and also got the error. It seems there was more than one result being returned from the "do shell script" line... so it couldn't make the result into a number.
I changed the code to this and it worked...
tell application "System Events"
repeat
set PID to unix id of process "Safari"
set getCpuPercentCmd to "ps aux | grep " & PID & " | grep -v grep | awk '{print $3}'"
set getCpuPercent to paragraphs of (do shell script getCpuPercentCmd)
set cpuPercent to (item -1 of getCpuPercent) as number
if cpuPercent < 5 then
display dialog cpuPercent as text
end if
end repeat
end tell
I ran just the grep command and found that on my machine webkit is running a WebProcess that is tied to the Safari PID. You can see toward the end of the second line that the -servicename is com.apple.WebKit.WebProcess-4881-0x1076eb0c0. Because of that, the grep is actually finding two results and returning "0.0\r0.0" which it cannot make into a number.
$user 4881 13.3 1.2 3812416 104840 ?? R 4:27PM 0:09.57 /Applications/Safari.app/Contents/MacOS/Safari -psn_0_1692061
$user 4885 0.1 0.7 3778328 56108 ?? S 4:27PM 0:00.79 /System/Library/StagedFrameworks/Safari/WebKit2.framework/WebProcess.app/Contents/MacOS/WebProcess /System/Library/StagedFrameworks/Safari/WebKit2.framework/WebKit2 -type webprocess -servicename com.apple.WebKit.WebProcess-4881-0x1076eb0c0 -localization en_US -client-identifier com.apple.Safari -ui-process-name Safari
$user 5250 0.0 0.0 2432768 520 ?? R 4:29PM 0:00.00 grep 4881
$user 5248 0.0 0.0 2433432 824 ?? S 4:29PM 0:00.00 sh -c ps aux | grep 4881