Measure the time of a child process without blocking the parent - fork

I want to measure the execution time of a child process. I can use times function
https://linux.die.net/man/3/times. But it requires to block the parent process using wait(). On the other hand, I need to keep the parent process running in parallel to store the data being generated by the child. Once the child has exited, I want to know how much CPU time was spent in executing the child process. Something like below:
int done = FALSE;
struct timespec ts;
struct rusage ru;
struct timeval utime;
struct timeval stime;
int main(int argc, char* argv[]){
pid_t pid1, pid2; int status;
char *args[] = {"./wp"};
struct rusage usage;
int i, j,n;
signal(SIGCHLD,reaper);
pid1 = fork();
if (pid1 == 0) { // child
printf("Its child %d\n",getpid());
printf("child %d: executing target program\n", getpid());
execv(argv[1], argv+1);
}
n = 0;
while (n<20) { //parent has to do some stuff in parallel
printf("doing something...\n"); sleep(1);
if (done){ // child exited
done = FALSE;
pid_t pid2 = wait3(&status, 0, &usage);
printf("...child of %d done executing.\n",getpid());
printf("exit code for %d is %d\n", pid2, status);
if (WIFEXITED(status))
printf("The exit status is %d\n", WEXITSTATUS(status));
utime = usage.ru_utime;
stime = usage.ru_stime;
printf("RUSAGE :ru_utime => %lld [sec] : %lld [usec], :ru_stime => %lld [sec] : %lld [usec] \n",
(int64_t)utime.tv_sec, (int64_t)utime.tv_usec,
(int64_t)stime.tv_sec, (int64_t)stime.tv_usec);
}
n++;
}
printf("got my SIGCHLD, cleaning up!\n");
signal(SIGCHLD,SIG_DFL);
return 0;
}
void reaper(int sig) {
done=TRUE; //
}

Related

How to chunk shell script input by time, not by size?

In a bash script I am using a many-producer single-consumer pattern. Producers are background processes writing lines into a fifo (via GNU Parallel). The consumer reads all lines from the fifo, then sorts, filters, and prints the formatted result to stdout.
However, it could take a long time until the full result is available. Producers are usually fast on the first few results but then would slow down. Here I am more interested to see chunks of data every few seconds, each sorted and filtered individually.
mkfifo fifo
parallel ... >"$fifo" &
while chunk=$(read with timeout 5s and at most 10s <"$fifo"); do
process "$chunk"
done
The loop would run until all producers are done and all input is read. Each chunk is read until there has been no new data for 5s, or until 10s have passed since the chunk was started. A chunk may also be empty if there was no new data for 10s.
I tried to make it work like this:
output=$(mktemp)
while true; do
wasTimeout=0 interruptAt=$(( $(date '+%s') + 10 ))
while true; do
IFS= read -r -t5 <>"${fifo}"
rc="$?"
if [[ "${rc}" -gt 0 ]]; then
[[ "${rc}" -gt 128 ]] && wasTimeout=1
break
fi
echo "$REPLY" >>"${output}"
if [[ $(date '+%s') -ge "${interruptAt}" ]]; then
wasTimeout=1
break
fi
done
echo '---' >>"${output}"
[[ "${wasTimeout}" -eq 0 ]] && break
done
Tried some variations of this. In the form above it reads the first chunk but then loops forever. If I use <"${fifo}" (no read/write as above) it blocks after the first chunk. Maybe all of this could be simplified with buffer and/or stdbuf? But both of them define blocks by size, not by time.
This is not a trivial problem to resolve. As I hinted, a C program (or a program in some programming language other than the shell) is probably the best solution. Some of the complicating factors are:
Reading with timeouts.
If data arrives soon enough, the timeout changes.
Different systems have different sets of interval timing functions:
alarm() is likely available everywhere, but has only 1-second resolution which is liable to accumulated rounding errors. (Compile this version with make UFLAGS=-DUSE_ALARM; on macOS, use make UFLAGS=-DUSE_ALARM LDLIB2=.)
setitimer()
uses microsecond timing and the struct timeval type. (Compile this version with make UFLAGS=-DUSE_SETITIMER; on macOS, compile with make UFLAGS=-DUSE_SETITIMER LDLIB2=.)
timer_create() and
timer_settime() etc use the modern nanosecond type struct timespec. This is available on Linux; it is not available on macOS 10.14.5 Mojave or earlier. (Compile this version with make; it won't work on macOS.)
The program usage message is:
$ chunker79 -h
Usage: chunker79 [-hvV][-c chunk][-d delay][-f file]
-c chunk Maximum time to wait for data in a chunk (default 10)
-d delay Maximum delay after line read (default: 5)
-f file Read from file instead of standard input
-h Print this help message and exit
-v Verbose mode: print timing information to stderr
-V Print version information and exit
$
This code is available in my SOQ (Stack Overflow Questions) repository on GitHub as file chunker79.c in the src/so-5631-4784 sub-directory. You will need some of the support code from the src/libsoq directory too.
/*
#(#)File: chunker79.c
#(#)Purpose: Chunk Reader for SO 5631-4784
#(#)Author: J Leffler
#(#)Copyright: (C) JLSS 2019
*/
/*TABSTOP=4*/
/*
** Problem specification from the Stack Overflow question
**
** In a bash script I am using a many-producer single-consumer pattern.
** Producers are background processes writing lines into a fifo (via GNU
** Parallel). The consumer reads all lines from the fifo, then sorts,
** filters, and prints the formatted result to stdout.
**
** However, it could take a long time until the full result is
** available. Producers are usually fast on the first few results but
** then would slow down. Here I am more interested to see chunks of
** data every few seconds, each sorted and filtered individually.
**
** mkfifo fifo
** parallel ... >"$fifo" &
** while chunk=$(read with timeout 5s and at most 10s <"$fifo"); do
** process "$chunk"
** done
**
** The loop would run until all producers are done and all input is
** read. Each chunk is read until there has been no new data for 5s, or
** until 10s have passed since the chunk was started. A chunk may also
** be empty if there was no new data for 10s.
*/
/*
** Analysis
**
** 1. If no data arrives at all for 10 seconds, then the program should
** terminate producing no output. This timeout is controlled by the
** value of time_chunk in the code.
** 2. If data arrives more or less consistently, then the collection
** should continue for 10s and then finish. This timeout is also
** controlled by the value of time_chunk in the code.
** 3. If a line of data arrives before 5 seconds have elapsed, and no
** more arrives for 5 seconds, then the collection should finish.
** (If the first line arrives after 5 seconds and no more arrives
** for more than 5 seconds, then the 10 second timeout cuts in.)
** This timeout is controlled by the value of time_delay in the code.
** 4. This means that we want two separate timers at work:
** - Chunk timer (started when the program starts).
** - Delay timer (started each time a line is read).
**
** It doesn't matter which timer goes off, but further timer signals
** should be ignored. External signals will confuse things; tough!
**
** -- Using alarm(2) is tricky because it provides only one time, not two.
** -- Using getitimer(2), setitimer(2) uses obsolescent POSIX functions,
** but these are available on macOS.
** -- Using timer_create(2), timer_destroy(2), timer_settime(2),
** timer_gettime(2) uses current POSIX function but is not available
** on macOS.
*/
#include "posixver.h"
#include "stderr.h"
#include "timespec_io.h"
#include <assert.h>
#include <signal.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/uio.h>
#include <time.h>
#include <unistd.h>
#ifdef USE_SETITIMER
#include "timeval_math.h"
#include "timeval_io.h"
#include <sys/time.h>
#endif /* USE_SETITIMER */
static const char optstr[] = "hvVc:d:f:";
static const char usestr[] = "[-hvV][-c chunk][-d delay][-f file]";
static const char hlpstr[] =
" -c chunk Maximum time to wait for data in a chunk (default 10)\n"
" -d delay Maximum delay after line read (default: 5)\n"
" -f file Read from file instead of standard input\n"
" -h Print this help message and exit\n"
" -v Verbose mode: print timing information to stderr\n"
" -V Print version information and exit\n"
;
static struct timespec time_delay = { .tv_sec = 5, .tv_nsec = 0 };
static struct timespec time_chunk = { .tv_sec = 10, .tv_nsec = 0 };
static struct timespec time_start;
static bool verbose = false;
static void set_chunk_timeout(void);
static void set_delay_timeout(void);
static void cancel_timeout(void);
static void alarm_handler(int signum);
// Using signal() manages to set SA_RESTART on a Mac.
// This is allowed by standard C and POSIX, sadly.
// signal(SIGALRM, alarm_handler);
#if defined(USE_ALARM)
static void set_chunk_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
alarm(time_chunk.tv_sec);
struct sigaction sa;
sa.sa_handler = alarm_handler;
sigemptyset(&sa.sa_mask);
sa.sa_flags = 0;
sigaction(SIGALRM, &sa, NULL);
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
static void set_delay_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
unsigned time_left = alarm(0);
if (time_left > time_delay.tv_sec)
alarm(time_delay.tv_sec);
else
alarm(time_left);
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
static void cancel_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
alarm(0);
signal(SIGALRM, SIG_IGN);
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
#elif defined(USE_SETITIMER)
static inline struct timeval cvt_timespec_to_timeval(struct timespec ts)
{
return (struct timeval){ .tv_sec = ts.tv_sec, .tv_usec = ts.tv_nsec / 1000 };
}
static void set_chunk_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
struct itimerval tv_new = { { 0, 0 }, { 0, 0 } };
tv_new.it_value = cvt_timespec_to_timeval(time_chunk);
struct itimerval tv_old;
if (setitimer(ITIMER_REAL, &tv_new, &tv_old) != 0)
err_syserr("failed to set interval timer: ");
struct sigaction sa;
sa.sa_handler = alarm_handler;
sigemptyset(&sa.sa_mask);
sa.sa_flags = 0;
sigaction(SIGALRM, &sa, NULL);
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
static void set_delay_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
struct itimerval tv_until;
if (getitimer(ITIMER_REAL, &tv_until) != 0)
err_syserr("failed to set interval timer: ");
struct timeval tv_delay = cvt_timespec_to_timeval(time_delay);
if (verbose)
{
char buff1[32];
fmt_timeval(&tv_delay, 6, buff1, sizeof(buff1));
char buff2[32];
fmt_timeval(&tv_until.it_value, 6, buff2, sizeof(buff2));
err_remark("---- %s(): delay %s, left %s\n", __func__, buff1, buff2);
}
if (cmp_timeval(tv_until.it_value, tv_delay) <= 0)
{
if (verbose)
err_remark("---- %s(): no need for delay timer\n", __func__);
}
else
{
struct itimerval tv_new = { { 0, 0 }, { 0, 0 } };
tv_new.it_value = cvt_timespec_to_timeval(time_delay);
struct itimerval tv_old;
if (setitimer(ITIMER_REAL, &tv_new, &tv_old) != 0)
err_syserr("failed to set interval timer: ");
if (verbose)
err_remark("---- %s(): set delay timer\n", __func__);
}
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
static void cancel_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
struct itimerval tv_new =
{
.it_value = { .tv_sec = 0, .tv_usec = 0 },
.it_interval = { .tv_sec = 0, .tv_usec = 0 },
};
struct itimerval tv_old;
if (setitimer(ITIMER_REAL, &tv_new, &tv_old) != 0)
err_syserr("failed to set interval timer: ");
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
#else /* USE_TIMER_GETTIME */
#include "timespec_math.h"
static timer_t t0 = { 0 };
static void set_chunk_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
struct sigevent ev =
{
.sigev_notify = SIGEV_SIGNAL,
.sigev_signo = SIGALRM,
.sigev_value.sival_int = 0,
.sigev_notify_function = 0,
.sigev_notify_attributes = 0,
};
if (timer_create(CLOCK_REALTIME, &ev, &t0) < 0)
err_syserr("failed to create a timer: ");
struct itimerspec it =
{
.it_interval = { .tv_sec = 0, .tv_nsec = 0 },
.it_value = time_chunk,
};
struct itimerspec ot;
if (timer_settime(t0, 0, &it, &ot) != 0)
err_syserr("failed to activate timer: ");
struct sigaction sa;
sa.sa_handler = alarm_handler;
sigemptyset(&sa.sa_mask);
sa.sa_flags = 0;
sigaction(SIGALRM, &sa, NULL);
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
static void set_delay_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
struct itimerspec time_until;
if (timer_gettime(t0, &time_until) != 0)
err_syserr("failed to set per-process timer: ");
char buff1[32];
fmt_timespec(&time_delay, 6, buff1, sizeof(buff1));
char buff2[32];
fmt_timespec(&time_until.it_value, 6, buff2, sizeof(buff2));
err_remark("---- %s(): delay %s, left %s\n", __func__, buff1, buff2);
if (cmp_timespec(time_until.it_value, time_delay) <= 0)
{
if (verbose)
err_remark("---- %s(): no need for delay timer\n", __func__);
}
else
{
struct itimerspec time_new =
{
.it_interval = { .tv_sec = 0, .tv_nsec = 0 },
.it_value = time_delay,
};
struct itimerspec time_old;
if (timer_settime(t0, 0, &time_new, &time_old) != 0)
err_syserr("failed to set per-process timer: ");
if (verbose)
err_remark("---- %s(): set delay timer\n", __func__);
}
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
static void cancel_timeout(void)
{
if (timer_delete(t0) != 0)
err_syserr("failed to delete timer: ");
}
#endif /* Timing mode */
/* Writing to stderr via err_remark() is not officially supported */
static void alarm_handler(int signum)
{
assert(signum == SIGALRM);
if (verbose)
err_remark("---- %s(): signal %d\n", __func__, signum);
}
static void read_chunks(FILE *fp)
{
size_t num_data = 0;
size_t max_data = 0;
struct iovec *data = 0;
size_t buflen = 0;
char *buffer = 0;
ssize_t length;
size_t chunk_len = 0;
clock_gettime(CLOCK_REALTIME, &time_start);
set_chunk_timeout();
while ((length = getline(&buffer, &buflen, fp)) != -1)
{
if (num_data >= max_data)
{
size_t new_size = (num_data * 2) + 2;
void *newspace = realloc(data, new_size * sizeof(data[0]));
if (newspace == 0)
err_syserr("failed to allocate %zu bytes data: ", new_size * sizeof(data[0]));
data = newspace;
max_data = new_size;
}
data[num_data].iov_base = buffer;
data[num_data].iov_len = length;
num_data++;
if (verbose)
err_remark("Received line %zu\n", num_data);
chunk_len += length;
buffer = 0;
buflen = 0;
set_delay_timeout();
}
cancel_timeout();
if (chunk_len > 0)
{
if ((length = writev(STDOUT_FILENO, data, num_data)) < 0)
err_syserr("failed to write %zu bytes to standard output: ", chunk_len);
else if ((size_t)length != chunk_len)
err_error("failed to write %zu bytes to standard output "
"(short write of %zu bytes)\n", chunk_len, (size_t)length);
}
if (verbose)
err_remark("---- %s(): data written (%zu bytes)\n", __func__, length);
for (size_t i = 0; i < num_data; i++)
free(data[i].iov_base);
free(data);
free(buffer);
}
int main(int argc, char **argv)
{
const char *name = "(standard input)";
FILE *fp = stdin;
err_setarg0(argv[0]);
err_setlogopts(ERR_MICRO);
int opt;
while ((opt = getopt(argc, argv, optstr)) != -1)
{
switch (opt)
{
case 'c':
if (scn_timespec(optarg, &time_chunk) != 0)
err_error("Failed to convert '%s' into a time value\n", optarg);
break;
case 'd':
if (scn_timespec(optarg, &time_delay) != 0)
err_error("Failed to convert '%s' into a time value\n", optarg);
break;
case 'f':
if ((fp = fopen(optarg, "r")) == 0)
err_syserr("Failed to open file '%s' for reading: ", optarg);
name = optarg;
break;
case 'h':
err_help(usestr, hlpstr);
/*NOTREACHED*/
case 'v':
verbose = true;
break;
case 'V':
err_version("CHUNKER79", &"#(#)$Revision$ ($Date$)"[4]);
/*NOTREACHED*/
default:
err_usage(usestr);
/*NOTREACHED*/
}
}
if (optind != argc)
err_usage(usestr);
if (verbose)
{
err_remark("chunk: %3lld.%09ld\n", (long long)time_chunk.tv_sec, time_chunk.tv_nsec);
err_remark("delay: %3lld.%09ld\n", (long long)time_delay.tv_sec, time_delay.tv_nsec);
err_remark("file: %s\n", name);
}
read_chunks(fp);
return 0;
}
My SOQ repository also has a script gen-data.sh which makes use of some custom programs to generate a data stream such as this (the seed value is written to standard error, not standard output):
$ gen-data.sh
# Seed: 1313715286
2019-06-03 23:04:16.653: Zunmieoprri Rdviqymcho 5878 2017-03-29 03:59:15 Udransnadioiaeamprirteo
2019-06-03 23:04:18.525: Rndflseoevhgs Etlaevieripeoetrnwkn 9500 2015-12-18 10:49:15 Ebyrcoebeezatiagpleieoefyc
2019-06-03 23:04:20.526: Nrzsuiakrooab Nbvliinfqidbujoops 1974 2020-05-13 08:05:14 Lgithearril
2019-06-03 23:04:21.777: Eeagop Aieneose 6533 2016-11-06 22:51:58 Aoejlwebbssroncmeovtuuueigraa
2019-06-03 23:04:23.876: Izirdoeektau Atesltiybysaclee 4557 2020-09-13 02:24:46 Igrooiaauiwtna
2019-06-03 23:04:26.145: Yhioit Eamrexuabagsaraiw 9703 2014-09-13 07:44:12 Dyiiienglolqopnrbneerltnmsdn
^C
$
When fed into chunker79 with default options, I get output like:
$ gen-data.sh | chunker79
# Seed: 722907235
2019-06-03 23:06:20.570: Aluaezkgiebeewal Oyvahee 1022 2015-08-12 07:45:54 Weuababeeduklleym
2019-06-03 23:06:24.100: Gmujvoyevihvoilc Negeiiuvleem 8196 2015-08-29 21:15:15 Nztkrvsadeoeagjgoyotvertavedi
$
If you analyze the time intervals (look at the first two fields in the output lines), that output meets the specification. A still more detailed analysis is shown by:
$ timecmd -mr -- gen-data.sh | timecmd -mr -- chunker79
2019-06-03 23:09:14.246 [PID 57159] gen-data.sh
2019-06-03 23:09:14.246 [PID 57160] chunker79
# Seed: -1077610201
2019-06-03 23:09:14.269: Woreio Rdtpimvoscttbyhxim 7893 2017-03-12 12:46:57 Uywaietirkekes
2019-06-03 23:09:16.939: Uigaba Nzoxdeuisofai 3630 2017-11-16 09:28:59 Jnsncgoesycsevdscugoathusaoq
2019-06-03 23:09:17.845: Sscreua Aloaoonnsuur 5163 2016-08-13 19:47:15 Injhsiifqovbnyeooiimitaaoir
2019-06-03 23:09:19.272 [PID 57160; status 0x0000] - 5.026s - chunker79
2019-06-03 23:09:22.084 [PID 57159; status 0x8D00] - 7.838s - gen-data.sh
$
There is a noticeable pause in this setup between when the output from chunker79 appears and when gen-data.sh completes. That's due to Bash waiting on all processes in the pipeline to complete, and gen-data.sh doesn't complete until the next time it writes to the pipe after the message that finishes chunker79. This is an artefact of this test setup; it wouldn't be a factor in the shell script outlined in the question.
I would consider writing a safe multi-threaded program with queues.
I know Java better, but there might be more modern suitable languages like Go and Kotlin.
Something like this:
#!/usr/bin/perl
$timeout = 3;
while(<STDIN>) {
# Make sure there is some input
push #out,$_;
eval {
local $SIG{ALRM} = sub { die };
alarm $timeout;
while(<STDIN>) {
alarm $timeout;
push #out,$_;
}
alarm 0;
};
system "echo","process",#out;
}
GNU Parallel 20200122 introduced --blocktimeout (--bt):
find ~ | parallel -j3 --bt 2s --pipe wc
This works like normal GNU Parallel except if it takes > 2 seconds to fill a block. In that case the block read so far is simply passed to wc (unless it is empty).
It has a slightly odd startup behaviour: You have to wait 3*2s (jobslots*timeout) before the output stabilizes, and you get an output at least every 2s.

Difference between mutual exclusion and blocked-IO in kernel programming?

I am unable to understand the difference between the follwing two codes. Can any body explain the difference between the following codes & also explain the differnece between semaphore and mutex with example....
Mutual exclusion:
DEFINE_SEMAPHORE(mysem);
static ssize_t dev_read(struct file *file,char *buf, size_t lbuf, loff_t *ppos)
{
int maxbytes, bytes_to_do, nbytes;
maxbytes = SIZE - *ppos;
if(maxbytes < lbuf) bytes_to_do = maxbytes;
else bytes_to_do = lbuf;
if(bytes_to_do == 0){
printk("reached end of device\n");
return -ENOSPC;
}
if(down_interruptible(&mysem))
return -ERESTARTSYS;
nbytes = bytes_to_do - copy_to_user(buf,dev_buf+*ppos,bytes_to_do);
up(&mysem);
*ppos += nbytes;
return nbytes;
}
static ssize_t dev_write(struct file *file,const char *buf, size_t lbuf,
loff_t *ppos)
{
int maxbytes, bytes_to_do, nbytes;
maxbytes = SIZE - *ppos;
if(maxbytes < lbuf) bytes_to_do = maxbytes;
else bytes_to_do = lbuf;
if(bytes_to_do == 0){
printk("reached end of device\n");
return -ENOSPC;
}
if(down_interruptible(&mysem))
return -ERESTARTSYS;
nbytes = bytes_to_do - copy_from_user(dev_buf+*ppos,buf,bytes_to_do);
ssleep(10);
up(&mysem);
*ppos += nbytes;
return nbytes;
}
Blocked IO
init_MUTEX_LOCKED(&mysem);
static ssize_t dev_read(struct file *file,char *buf, size_t lbuf, loff_t *ppos)
{
int maxbytes, bytes_to_do, nbytes;
maxbytes = SIZE - *ppos;
if(maxbytes < lbuf) bytes_to_do = maxbytes;
else bytes_to_do = lbuf;
if(bytes_to_do == 0){
printk("reached end of device\n");
return -ENOSPC;
}
if(down_interruptible(&mysem))
return -ERESTARTSYS;
nbytes = bytes_to_do - copy_to_user(buf,dev_buf+*ppos,bytes_to_do);
*ppos += nbytes;
return nbytes;
}
static ssize_t dev_write(struct file *file,const char *buf, size_t lbuf,
loff_t *ppos)
{
int maxbytes, bytes_to_do, nbytes;
maxbytes = SIZE - *ppos;
if(maxbytes < lbuf) bytes_to_do = maxbytes;
else bytes_to_do = lbuf;
if(bytes_to_do == 0){
printk("reached end of device\n");
return -ENOSPC;
}
nbytes = bytes_to_do - copy_from_user(dev_buf+*ppos,buf,bytes_to_do);
ssleep(10);
up(&mysem);
*ppos += nbytes;
return nbytes;
}
Mutex is nothing but a binary semaphore. It means that mutex can have only two states : locked and unlocked. But semaphore can have more than two counts. So number of processes which can acquire the semaphore lock is equal to the count with which semaphore is initialized.
In your example, in first code snippet, whether it is read or write, whichever is acquiring the lock is itself releasing the lock also after it completes its respective read or write. Both can not work simultaneously due to mutex.
While in second code snippet, the code exhibits blocking I/O concept which is designed to solve a problem explained in a book Linux Device Drivers(LDD) : "what to do when there's no data yet to read, but we're not at end-of-file. The default answer is go to sleep waiting for data". As you can see in the code, lock is declared as Mutex and that also in locked state. So, if any read comes when there is no data, it can not acquire a lock as mutex is already in locked state, so it will go to sleep (In short read is blocked). Whenever any write come, it first writes to device and then it releases the mutex. So, now blocked read can acquire that lock and can complete its read process. Here also, both can not work simultaneously, but lock acquiring and releasing mechanism is synchronized in such a manner that read can not progress until write does not write anything to device.

Sending child stdout to parent stdout

I am working on a simple application that takes the output from another one and writes it to a socket.
Below is the code for the application that I am execvp'ing
this code is called simple App
int main( int argc, char * argv[])
{
int count = 0;
while(count < 5)
//Attempt fork
if((pid2 = fork()) < 0) //Failed to fork
{
printf("\n Failed to fork in Dump \n");
exit(1);
}
else if (pid2 == 0) //Child Code
{
dup2(STDOUT_FILENO,pipefd[1]);
close(pipefd[0]);
close(STDOUT_FILENO);
if(execvp("../App/App", dumpParam) < 0) //execute the app; returns -1 if failed
{
printf("\nFailed to execute App\n");
exit(1);
}
}
else //Parent
{
close(pipefd[1]);
memset(buff,'0',sizeof(buff));
printf("here\n");
while( (r = read(pipefd[0],buff,sizeof(buff))) >= 0)
{
printf("\nSuccess in read r = %d\n",r);
printf("\nBuff = %s\n",buff);
memset(buff,'0',sizeof(buff));
}
printf("nowHere\n");
}
return pipefd[0];
so far when i run it, it prints out just a ton of 0's. if i set read > 0 then it doesn't run at all. Thoughts?
The read is returning 0 (end of file) because the child end of the pipe is closed.
You've got the dup2 the wrong way round, it should be
dup2(pipefd[1],STDOUT_FILENO)
also remove
close(STDOUT_FILENO)

Task switching using a queue

i'm developing my own hobby os, and now i'm stuck with a problem on the scheduler/task switching.
I planned to use a FIFO queue as structure to hold processes. I implemented it using linked list.
I also decided to use the iret method to switch from a task to another (so when the os was serving an interrupt request just before the iret i change the ESP register in order to move to the new task).
But i have a problem.
When the os start it launch two tasks:
idle
shell
And with these two i have no problem.
But if i try to launch two other tasks (with a simply printf inside), the task queue was corrupted.
If after that i try to print the queue it print only two tasks that are the 2 just created and with idle and shell disappeared, but the os continues to work (i think that in a specific moment the esp field of the new tasks was replaced with the esp content of the shell).
The task data structure is:
typedef struct task_t{
pid_t pid;
char name[NAME_LENGTH];
void (*start_function)();
task_state status;
task_register_t *registers;
unsigned int cur_quants;
unsigned int eip;
long int esp;
unsigned int pdir;
unsigned int ptable;
struct task_t *next;
}task_t;
and the tss is:
typedef struct {
unsigned int edi; //+0
unsigned int esi; //+1
unsigned int ebp; //+2
unsigned int esp; //+3 (can be null)
unsigned int ebx; //+4
unsigned int edx; //+5
unsigned int ecx; //+6
unsigned int eax; //+7
unsigned int eip; //+8
unsigned int cs; //+9
unsigned int eflags; //+10
unsigned int end;
} task_register_t;
The scheduler function is the following:
void schedule(unsigned int *stack){
asm("cli");
if(active == TRUE){
task_t* cur_task = dequeue_task();
if(cur_task != NULL){
cur_pid = cur_task->pid;
dbg_bochs_print("#######");
dbg_bochs_print(cur_task->name);
if(cur_task->status!=NEW){
cur_task->esp=*stack;
} else {
cur_task->status=READY;
((task_register_t *)(cur_task->esp))->eip = cur_task->eip;
}
enqueue_task(cur_task->pid, cur_task);
cur_task=get_task();
if(cur_task->status==NEW){
cur_task->status=READY;
}
dbg_bochs_print(" -- ");
dbg_bochs_print(cur_task->name);
dbg_bochs_print("\n");
//load_pdbr(cur_taskp->pdir);
*stack = cur_task->esp;
} else {
enqueue_task(cur_task->pid, cur_task);
}
}
active = FALSE;
return;
asm("sti");
}
The tss is initalized with the following values:
void new_tss(task_register_t* tss, void (*func)()){
tss->eax=0;
tss->ebx=0;
tss->ecx=0;
tss->edx=0;
tss->edi =0;
tss->esi =0;
tss->cs = 8;
tss->eip = (unsigned)func;
tss->eflags = 0x202;
tss->end = (unsigned) suicide;
//tss->fine = (unsigned)end; //per metterci il suicide
return;
}
And the function that creates a new task is the following:
pid_t new_task(char *task_name, void (*start_function)()){
asm("cli");
task_t *new_task;
table_address_t local_table;
unsigned int new_pid = request_pid();
new_task = (task_t*)kmalloc(sizeof(task_t));
strcpy(new_task->name, task_name);
new_task->next = NULL;
new_task->start_function = start_function;
new_task->cur_quants=0;
new_task->pid = new_pid;
new_task->eip = (unsigned int)start_function;
new_task->esp = (unsigned int)kmalloc(STACK_SIZE) + STACK_SIZE-100;
new_task->status = NEW;
new_task->registers = (task_register_t*)new_task->esp;
new_tss(new_task->registers, start_function);
local_table = map_kernel();
new_task->pdir = local_table.page_dir;
new_task->ptable = local_table.page_table;
//new_task->pdir = 0;
//new_task->ptable = 0;
enqueue_task(new_task->pid, new_task);
//(task_list.current)->cur_quants = MAX_TICKS;
asm("sti");
return new_pid;
}
I'm sure that i just forgot something, or i miss some consideration. But i cannot figure what i'm missing.
Actually i'm working only in kernel mode, and inside the same address space (pagiing is enabled, but actually i use the same pagedir for all tasks).
The ISR macros are defined here:
https://github.com/inuyasha82/DreamOs/blob/master/include/processore/handlers.h
I declared four kinds of function in order to handle ISR:
EXCEPTION
EXCEPTION_EC (an exception with an error code)
IRQ
SYSCALL
Obviously the scheduler is called by an IRQ routine, so the macro looks like:
__asm__("INT_"#n":"\
"pushad;" \
"movl %esp, %eax;"\
"pushl %eax;"\
"call _irqinterrupt;"\
"popl %eax;"\
"movl %eax, %esp;"\
"popad;"\
"iret;")
the irq handler function is:
void _irqinterrupt(unsigned int esp){
asm("cli;");
int irqn;
irqn = get_current_irq();
IRQ_s* tmpHandler;
if(irqn>=0) {
tmpHandler = shareHandler[irqn];
if(tmpHandler!=0) {
tmpHandler->IRQ_func();
#ifdef DEBUG
printf("2 - IRQ_func: %d, %d\n", tmpHandler->IRQ_func, tmpHandler);
#endif
while(tmpHandler->next!=NULL) {
tmpHandler = tmpHandler->next;
#ifdef DEBUG
printf("1 - IRQ_func (_prova): %d, %d\n", tmpHandler->IRQ_func, tmpHandler);
#endif
if(tmpHandler!=0) tmpHandler->IRQ_func();
}
} else printf("irqn: %d\n", irqn);
}
else printf("IRQ N: %d E' arrivato qualcosa che non so gestire ", irqn);
if(irqn<=8 && irqn!=2) outportb(MASTER_PORT, EOI);
else if(irqn<=16 || irqn==2){
outportb(SLAVE_PORT, EOI);
outportb(MASTER_PORT, EOI);
}
schedule(&esp);
asm("sti;");
return;
}
And these are the enqueue_task and dequeue_task functions:
void enqueue_task(pid_t pid, task_t* n_task){
n_task->next=NULL;
if(task_list.tail == NULL){
task_list.head = n_task;
task_list.tail = task_list.head;
} else {
task_list.head->next=n_task;
task_list.head = n_task;
}
}
task_t* dequeue_task(){
if(task_list.head==NULL){
return NULL;
} else {
task_t* _task;
_task = task_list.tail;
task_list.tail=_task->next;
return _task;
}
return;
}
Thanks in advance,
and let me know if you need more details!
It is hard to tell. How does your assembly part of the isr look like? What makes me think is the problem (since you can save and restore two tasks but not more) is that you don't push and pop all registers properly. You do use pusha and popa for the isr right?
I also want to add that having cli and sti like you have done there can be dangerous. In your isrs set cli as the first opcode. Then you wont need to use sti at all because iret will automatically flip this on for you (it is actually a bit in the eflags register).
Good luck!

How to get an X11 Window from a Process ID?

Under Linux, my C++ application is using fork() and execv() to launch multiple instances of OpenOffice so as to view some powerpoint slide shows. This part works.
Next I want to be able to move the OpenOffice windows to specific locations on the display. I can do that with the XMoveResizeWindow() function but I need to find the Window for each instance.
I have the process ID of each instance, how can I find the X11 Window from that ?
UPDATE - Thanks to Andy's suggestion, I have pulled this off. I'm posting the code here to share it with the Stack Overflow community.
Unfortunately Open Office does not seem to set the _NET_WM_PID property so this doesn't ultimately solve my problem but it does answer the question.
// Attempt to identify a window by name or attribute.
// by Adam Pierce <adam#doctort.org>
#include <X11/Xlib.h>
#include <X11/Xatom.h>
#include <iostream>
#include <list>
using namespace std;
class WindowsMatchingPid
{
public:
WindowsMatchingPid(Display *display, Window wRoot, unsigned long pid)
: _display(display)
, _pid(pid)
{
// Get the PID property atom.
_atomPID = XInternAtom(display, "_NET_WM_PID", True);
if(_atomPID == None)
{
cout << "No such atom" << endl;
return;
}
search(wRoot);
}
const list<Window> &result() const { return _result; }
private:
unsigned long _pid;
Atom _atomPID;
Display *_display;
list<Window> _result;
void search(Window w)
{
// Get the PID for the current Window.
Atom type;
int format;
unsigned long nItems;
unsigned long bytesAfter;
unsigned char *propPID = 0;
if(Success == XGetWindowProperty(_display, w, _atomPID, 0, 1, False, XA_CARDINAL,
&type, &format, &nItems, &bytesAfter, &propPID))
{
if(propPID != 0)
{
// If the PID matches, add this window to the result set.
if(_pid == *((unsigned long *)propPID))
_result.push_back(w);
XFree(propPID);
}
}
// Recurse into child windows.
Window wRoot;
Window wParent;
Window *wChild;
unsigned nChildren;
if(0 != XQueryTree(_display, w, &wRoot, &wParent, &wChild, &nChildren))
{
for(unsigned i = 0; i < nChildren; i++)
search(wChild[i]);
}
}
};
int main(int argc, char **argv)
{
if(argc < 2)
return 1;
int pid = atoi(argv[1]);
cout << "Searching for windows associated with PID " << pid << endl;
// Start with the root window.
Display *display = XOpenDisplay(0);
WindowsMatchingPid match(display, XDefaultRootWindow(display), pid);
// Print the result.
const list<Window> &result = match.result();
for(list<Window>::const_iterator it = result.begin(); it != result.end(); it++)
cout << "Window #" << (unsigned long)(*it) << endl;
return 0;
}
The only way I know to do this is to traverse the tree of windows until you find what you're looking for. Traversing isn't hard (just see what xwininfo -root -tree does by looking at xwininfo.c if you need an example).
But how do you identify the window you are looking for? Some applications set a window property called _NET_WM_PID.
I believe that OpenOffice is one of the applications that sets that property (as do most Gnome apps), so you're in luck.
Check if /proc/PID/environ contains a variable called WINDOWID
Bit late to the party. However:
Back in 2004, Harald Welte posted a code snippet that wraps the XCreateWindow() call via LD_PRELOAD and stores the process id in _NET_WM_PID. This makes sure that each window created has a PID entry.
http://www.mail-archive.com/devel#xfree86.org/msg05806.html
Try installing xdotool, then:
#!/bin/bash
# --any and --name present only as a work-around, see: https://github.com/jordansissel/xdotool/issues/14
ids=$(xdotool search --any --pid "$1" --name "dummy")
I do get a lot of ids. I use this to set a terminal window as urgent when it is done with a long command, with the program seturgent. I just loop through all the ids I get from xdotool and run seturgent on them.
There is no good way. The only real options I see, are:
You could look around in the process's address space to find the connection information and window ID.
You could try to use netstat or lsof or ipcs to map the connections to the Xserver, and then (somehow! you'll need root at least) look at its connection info to find them.
When spawning an instance you can wait until another window is mapped, assume it's the right one, and `move on.
I took the freedom to re-implement the OP's code using some modern C++ features. It maintains the same functionalities but I think that it reads a bit better. Also it does not leak even if the vector insertion happens to throw.
// Attempt to identify a window by name or attribute.
// originally written by Adam Pierce <adam#doctort.org>
// revised by Dario Pellegrini <pellegrini.dario#gmail.com>
#include <X11/Xlib.h>
#include <X11/Xatom.h>
#include <iostream>
#include <vector>
std::vector<Window> pid2windows(pid_t pid, Display* display, Window w) {
struct implementation {
struct FreeWrapRAII {
void * data;
FreeWrapRAII(void * data): data(data) {}
~FreeWrapRAII(){ XFree(data); }
};
std::vector<Window> result;
pid_t pid;
Display* display;
Atom atomPID;
implementation(pid_t pid, Display* display): pid(pid), display(display) {
// Get the PID property atom
atomPID = XInternAtom(display, "_NET_WM_PID", True);
if(atomPID == None) {
throw std::runtime_error("pid2windows: no such atom");
}
}
std::vector<Window> getChildren(Window w) {
Window wRoot;
Window wParent;
Window *wChild;
unsigned nChildren;
std::vector<Window> children;
if(0 != XQueryTree(display, w, &wRoot, &wParent, &wChild, &nChildren)) {
FreeWrapRAII tmp( wChild );
children.insert(children.end(), wChild, wChild+nChildren);
}
return children;
}
void emplaceIfMatches(Window w) {
// Get the PID for the given Window
Atom type;
int format;
unsigned long nItems;
unsigned long bytesAfter;
unsigned char *propPID = 0;
if(Success == XGetWindowProperty(display, w, atomPID, 0, 1, False, XA_CARDINAL,
&type, &format, &nItems, &bytesAfter, &propPID)) {
if(propPID != 0) {
FreeWrapRAII tmp( propPID );
if(pid == *reinterpret_cast<pid_t*>(propPID)) {
result.emplace_back(w);
}
}
}
}
void recurse( Window w) {
emplaceIfMatches(w);
for (auto & child: getChildren(w)) {
recurse(child);
}
}
std::vector<Window> operator()( Window w ) {
result.clear();
recurse(w);
return result;
}
};
//back to pid2windows function
return implementation{pid, display}(w);
}
std::vector<Window> pid2windows(const size_t pid, Display* display) {
return pid2windows(pid, display, XDefaultRootWindow(display));
}
int main(int argc, char **argv) {
if(argc < 2)
return 1;
int pid = atoi(argv[1]);
std::cout << "Searching for windows associated with PID " << pid << std::endl;
// Start with the root window.
Display *display = XOpenDisplay(0);
auto res = pid2windows(pid, display);
// Print the result.
for( auto & w: res) {
std::cout << "Window #" << static_cast<unsigned long>(w) << std::endl;
}
XCloseDisplay(display);
return 0;
}
Are you sure you have the process ID of each instance? My experience with OOo has been that trying to run a second instance of OOo merely converses with the first instance of OOo, and tells that to open the additional file.
I think you're going to need to use the message-sending capabilities of X to ask it nicely for its window. I would hope that OOo documents its coversations somewhere.
If you use python, I found a way here, the idea is from BurntSushi
If you launched the application, then you should know its cmd string, with which you can reduce calls to xprop, you can always loop through all the xids and check if the pid is the same as the pid you want
import subprocess
import re
import struct
import xcffib as xcb
import xcffib.xproto
def get_property_value(property_reply):
assert isinstance(property_reply, xcb.xproto.GetPropertyReply)
if property_reply.format == 8:
if 0 in property_reply.value:
ret = []
s = ''
for o in property_reply.value:
if o == 0:
ret.append(s)
s = ''
else:
s += chr(o)
else:
ret = str(property_reply.value.buf())
return ret
elif property_reply.format in (16, 32):
return list(struct.unpack('I' * property_reply.value_len,
property_reply.value.buf()))
return None
def getProperty(connection, ident, propertyName):
propertyType = eval(' xcb.xproto.Atom.%s' % propertyName)
try:
return connection.core.GetProperty(False, ident, propertyType,
xcb.xproto.GetPropertyType.Any,
0, 2 ** 32 - 1)
except:
return None
c = xcb.connect()
root = c.get_setup().roots[0].root
_NET_CLIENT_LIST = c.core.InternAtom(True, len('_NET_CLIENT_LIST'),
'_NET_CLIENT_LIST').reply().atom
raw_clientlist = c.core.GetProperty(False, root, _NET_CLIENT_LIST,
xcb.xproto.GetPropertyType.Any,
0, 2 ** 32 - 1).reply()
clientlist = get_property_value(raw_clientlist)
cookies = {}
for ident in clientlist:
wm_command = getProperty(c, ident, 'WM_COMMAND')
cookies[ident] = (wm_command)
xids=[]
for ident in cookies:
cmd = get_property_value(cookies[ident].reply())
if cmd and spref in cmd:
xids.append(ident)
for xid in xids:
pid = subprocess.check_output('xprop -id %s _NET_WM_PID' % xid, shell=True)
pid = re.search('(?<=\s=\s)\d+', pid).group()
if int(pid) == self.pid:
print 'found pid:', pid
break
print 'your xid:', xid

Resources