In a bash script I am using a many-producer single-consumer pattern. Producers are background processes writing lines into a fifo (via GNU Parallel). The consumer reads all lines from the fifo, then sorts, filters, and prints the formatted result to stdout.
However, it could take a long time until the full result is available. Producers are usually fast on the first few results but then would slow down. Here I am more interested to see chunks of data every few seconds, each sorted and filtered individually.
mkfifo fifo
parallel ... >"$fifo" &
while chunk=$(read with timeout 5s and at most 10s <"$fifo"); do
process "$chunk"
done
The loop would run until all producers are done and all input is read. Each chunk is read until there has been no new data for 5s, or until 10s have passed since the chunk was started. A chunk may also be empty if there was no new data for 10s.
I tried to make it work like this:
output=$(mktemp)
while true; do
wasTimeout=0 interruptAt=$(( $(date '+%s') + 10 ))
while true; do
IFS= read -r -t5 <>"${fifo}"
rc="$?"
if [[ "${rc}" -gt 0 ]]; then
[[ "${rc}" -gt 128 ]] && wasTimeout=1
break
fi
echo "$REPLY" >>"${output}"
if [[ $(date '+%s') -ge "${interruptAt}" ]]; then
wasTimeout=1
break
fi
done
echo '---' >>"${output}"
[[ "${wasTimeout}" -eq 0 ]] && break
done
Tried some variations of this. In the form above it reads the first chunk but then loops forever. If I use <"${fifo}" (no read/write as above) it blocks after the first chunk. Maybe all of this could be simplified with buffer and/or stdbuf? But both of them define blocks by size, not by time.
This is not a trivial problem to resolve. As I hinted, a C program (or a program in some programming language other than the shell) is probably the best solution. Some of the complicating factors are:
Reading with timeouts.
If data arrives soon enough, the timeout changes.
Different systems have different sets of interval timing functions:
alarm() is likely available everywhere, but has only 1-second resolution which is liable to accumulated rounding errors. (Compile this version with make UFLAGS=-DUSE_ALARM; on macOS, use make UFLAGS=-DUSE_ALARM LDLIB2=.)
setitimer()
uses microsecond timing and the struct timeval type. (Compile this version with make UFLAGS=-DUSE_SETITIMER; on macOS, compile with make UFLAGS=-DUSE_SETITIMER LDLIB2=.)
timer_create() and
timer_settime() etc use the modern nanosecond type struct timespec. This is available on Linux; it is not available on macOS 10.14.5 Mojave or earlier. (Compile this version with make; it won't work on macOS.)
The program usage message is:
$ chunker79 -h
Usage: chunker79 [-hvV][-c chunk][-d delay][-f file]
-c chunk Maximum time to wait for data in a chunk (default 10)
-d delay Maximum delay after line read (default: 5)
-f file Read from file instead of standard input
-h Print this help message and exit
-v Verbose mode: print timing information to stderr
-V Print version information and exit
$
This code is available in my SOQ (Stack Overflow Questions) repository on GitHub as file chunker79.c in the src/so-5631-4784 sub-directory. You will need some of the support code from the src/libsoq directory too.
/*
#(#)File: chunker79.c
#(#)Purpose: Chunk Reader for SO 5631-4784
#(#)Author: J Leffler
#(#)Copyright: (C) JLSS 2019
*/
/*TABSTOP=4*/
/*
** Problem specification from the Stack Overflow question
**
** In a bash script I am using a many-producer single-consumer pattern.
** Producers are background processes writing lines into a fifo (via GNU
** Parallel). The consumer reads all lines from the fifo, then sorts,
** filters, and prints the formatted result to stdout.
**
** However, it could take a long time until the full result is
** available. Producers are usually fast on the first few results but
** then would slow down. Here I am more interested to see chunks of
** data every few seconds, each sorted and filtered individually.
**
** mkfifo fifo
** parallel ... >"$fifo" &
** while chunk=$(read with timeout 5s and at most 10s <"$fifo"); do
** process "$chunk"
** done
**
** The loop would run until all producers are done and all input is
** read. Each chunk is read until there has been no new data for 5s, or
** until 10s have passed since the chunk was started. A chunk may also
** be empty if there was no new data for 10s.
*/
/*
** Analysis
**
** 1. If no data arrives at all for 10 seconds, then the program should
** terminate producing no output. This timeout is controlled by the
** value of time_chunk in the code.
** 2. If data arrives more or less consistently, then the collection
** should continue for 10s and then finish. This timeout is also
** controlled by the value of time_chunk in the code.
** 3. If a line of data arrives before 5 seconds have elapsed, and no
** more arrives for 5 seconds, then the collection should finish.
** (If the first line arrives after 5 seconds and no more arrives
** for more than 5 seconds, then the 10 second timeout cuts in.)
** This timeout is controlled by the value of time_delay in the code.
** 4. This means that we want two separate timers at work:
** - Chunk timer (started when the program starts).
** - Delay timer (started each time a line is read).
**
** It doesn't matter which timer goes off, but further timer signals
** should be ignored. External signals will confuse things; tough!
**
** -- Using alarm(2) is tricky because it provides only one time, not two.
** -- Using getitimer(2), setitimer(2) uses obsolescent POSIX functions,
** but these are available on macOS.
** -- Using timer_create(2), timer_destroy(2), timer_settime(2),
** timer_gettime(2) uses current POSIX function but is not available
** on macOS.
*/
#include "posixver.h"
#include "stderr.h"
#include "timespec_io.h"
#include <assert.h>
#include <signal.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/uio.h>
#include <time.h>
#include <unistd.h>
#ifdef USE_SETITIMER
#include "timeval_math.h"
#include "timeval_io.h"
#include <sys/time.h>
#endif /* USE_SETITIMER */
static const char optstr[] = "hvVc:d:f:";
static const char usestr[] = "[-hvV][-c chunk][-d delay][-f file]";
static const char hlpstr[] =
" -c chunk Maximum time to wait for data in a chunk (default 10)\n"
" -d delay Maximum delay after line read (default: 5)\n"
" -f file Read from file instead of standard input\n"
" -h Print this help message and exit\n"
" -v Verbose mode: print timing information to stderr\n"
" -V Print version information and exit\n"
;
static struct timespec time_delay = { .tv_sec = 5, .tv_nsec = 0 };
static struct timespec time_chunk = { .tv_sec = 10, .tv_nsec = 0 };
static struct timespec time_start;
static bool verbose = false;
static void set_chunk_timeout(void);
static void set_delay_timeout(void);
static void cancel_timeout(void);
static void alarm_handler(int signum);
// Using signal() manages to set SA_RESTART on a Mac.
// This is allowed by standard C and POSIX, sadly.
// signal(SIGALRM, alarm_handler);
#if defined(USE_ALARM)
static void set_chunk_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
alarm(time_chunk.tv_sec);
struct sigaction sa;
sa.sa_handler = alarm_handler;
sigemptyset(&sa.sa_mask);
sa.sa_flags = 0;
sigaction(SIGALRM, &sa, NULL);
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
static void set_delay_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
unsigned time_left = alarm(0);
if (time_left > time_delay.tv_sec)
alarm(time_delay.tv_sec);
else
alarm(time_left);
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
static void cancel_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
alarm(0);
signal(SIGALRM, SIG_IGN);
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
#elif defined(USE_SETITIMER)
static inline struct timeval cvt_timespec_to_timeval(struct timespec ts)
{
return (struct timeval){ .tv_sec = ts.tv_sec, .tv_usec = ts.tv_nsec / 1000 };
}
static void set_chunk_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
struct itimerval tv_new = { { 0, 0 }, { 0, 0 } };
tv_new.it_value = cvt_timespec_to_timeval(time_chunk);
struct itimerval tv_old;
if (setitimer(ITIMER_REAL, &tv_new, &tv_old) != 0)
err_syserr("failed to set interval timer: ");
struct sigaction sa;
sa.sa_handler = alarm_handler;
sigemptyset(&sa.sa_mask);
sa.sa_flags = 0;
sigaction(SIGALRM, &sa, NULL);
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
static void set_delay_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
struct itimerval tv_until;
if (getitimer(ITIMER_REAL, &tv_until) != 0)
err_syserr("failed to set interval timer: ");
struct timeval tv_delay = cvt_timespec_to_timeval(time_delay);
if (verbose)
{
char buff1[32];
fmt_timeval(&tv_delay, 6, buff1, sizeof(buff1));
char buff2[32];
fmt_timeval(&tv_until.it_value, 6, buff2, sizeof(buff2));
err_remark("---- %s(): delay %s, left %s\n", __func__, buff1, buff2);
}
if (cmp_timeval(tv_until.it_value, tv_delay) <= 0)
{
if (verbose)
err_remark("---- %s(): no need for delay timer\n", __func__);
}
else
{
struct itimerval tv_new = { { 0, 0 }, { 0, 0 } };
tv_new.it_value = cvt_timespec_to_timeval(time_delay);
struct itimerval tv_old;
if (setitimer(ITIMER_REAL, &tv_new, &tv_old) != 0)
err_syserr("failed to set interval timer: ");
if (verbose)
err_remark("---- %s(): set delay timer\n", __func__);
}
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
static void cancel_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
struct itimerval tv_new =
{
.it_value = { .tv_sec = 0, .tv_usec = 0 },
.it_interval = { .tv_sec = 0, .tv_usec = 0 },
};
struct itimerval tv_old;
if (setitimer(ITIMER_REAL, &tv_new, &tv_old) != 0)
err_syserr("failed to set interval timer: ");
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
#else /* USE_TIMER_GETTIME */
#include "timespec_math.h"
static timer_t t0 = { 0 };
static void set_chunk_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
struct sigevent ev =
{
.sigev_notify = SIGEV_SIGNAL,
.sigev_signo = SIGALRM,
.sigev_value.sival_int = 0,
.sigev_notify_function = 0,
.sigev_notify_attributes = 0,
};
if (timer_create(CLOCK_REALTIME, &ev, &t0) < 0)
err_syserr("failed to create a timer: ");
struct itimerspec it =
{
.it_interval = { .tv_sec = 0, .tv_nsec = 0 },
.it_value = time_chunk,
};
struct itimerspec ot;
if (timer_settime(t0, 0, &it, &ot) != 0)
err_syserr("failed to activate timer: ");
struct sigaction sa;
sa.sa_handler = alarm_handler;
sigemptyset(&sa.sa_mask);
sa.sa_flags = 0;
sigaction(SIGALRM, &sa, NULL);
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
static void set_delay_timeout(void)
{
if (verbose)
err_remark("-->> %s()\n", __func__);
struct itimerspec time_until;
if (timer_gettime(t0, &time_until) != 0)
err_syserr("failed to set per-process timer: ");
char buff1[32];
fmt_timespec(&time_delay, 6, buff1, sizeof(buff1));
char buff2[32];
fmt_timespec(&time_until.it_value, 6, buff2, sizeof(buff2));
err_remark("---- %s(): delay %s, left %s\n", __func__, buff1, buff2);
if (cmp_timespec(time_until.it_value, time_delay) <= 0)
{
if (verbose)
err_remark("---- %s(): no need for delay timer\n", __func__);
}
else
{
struct itimerspec time_new =
{
.it_interval = { .tv_sec = 0, .tv_nsec = 0 },
.it_value = time_delay,
};
struct itimerspec time_old;
if (timer_settime(t0, 0, &time_new, &time_old) != 0)
err_syserr("failed to set per-process timer: ");
if (verbose)
err_remark("---- %s(): set delay timer\n", __func__);
}
if (verbose)
err_remark("<<-- %s()\n", __func__);
}
static void cancel_timeout(void)
{
if (timer_delete(t0) != 0)
err_syserr("failed to delete timer: ");
}
#endif /* Timing mode */
/* Writing to stderr via err_remark() is not officially supported */
static void alarm_handler(int signum)
{
assert(signum == SIGALRM);
if (verbose)
err_remark("---- %s(): signal %d\n", __func__, signum);
}
static void read_chunks(FILE *fp)
{
size_t num_data = 0;
size_t max_data = 0;
struct iovec *data = 0;
size_t buflen = 0;
char *buffer = 0;
ssize_t length;
size_t chunk_len = 0;
clock_gettime(CLOCK_REALTIME, &time_start);
set_chunk_timeout();
while ((length = getline(&buffer, &buflen, fp)) != -1)
{
if (num_data >= max_data)
{
size_t new_size = (num_data * 2) + 2;
void *newspace = realloc(data, new_size * sizeof(data[0]));
if (newspace == 0)
err_syserr("failed to allocate %zu bytes data: ", new_size * sizeof(data[0]));
data = newspace;
max_data = new_size;
}
data[num_data].iov_base = buffer;
data[num_data].iov_len = length;
num_data++;
if (verbose)
err_remark("Received line %zu\n", num_data);
chunk_len += length;
buffer = 0;
buflen = 0;
set_delay_timeout();
}
cancel_timeout();
if (chunk_len > 0)
{
if ((length = writev(STDOUT_FILENO, data, num_data)) < 0)
err_syserr("failed to write %zu bytes to standard output: ", chunk_len);
else if ((size_t)length != chunk_len)
err_error("failed to write %zu bytes to standard output "
"(short write of %zu bytes)\n", chunk_len, (size_t)length);
}
if (verbose)
err_remark("---- %s(): data written (%zu bytes)\n", __func__, length);
for (size_t i = 0; i < num_data; i++)
free(data[i].iov_base);
free(data);
free(buffer);
}
int main(int argc, char **argv)
{
const char *name = "(standard input)";
FILE *fp = stdin;
err_setarg0(argv[0]);
err_setlogopts(ERR_MICRO);
int opt;
while ((opt = getopt(argc, argv, optstr)) != -1)
{
switch (opt)
{
case 'c':
if (scn_timespec(optarg, &time_chunk) != 0)
err_error("Failed to convert '%s' into a time value\n", optarg);
break;
case 'd':
if (scn_timespec(optarg, &time_delay) != 0)
err_error("Failed to convert '%s' into a time value\n", optarg);
break;
case 'f':
if ((fp = fopen(optarg, "r")) == 0)
err_syserr("Failed to open file '%s' for reading: ", optarg);
name = optarg;
break;
case 'h':
err_help(usestr, hlpstr);
/*NOTREACHED*/
case 'v':
verbose = true;
break;
case 'V':
err_version("CHUNKER79", &"#(#)$Revision$ ($Date$)"[4]);
/*NOTREACHED*/
default:
err_usage(usestr);
/*NOTREACHED*/
}
}
if (optind != argc)
err_usage(usestr);
if (verbose)
{
err_remark("chunk: %3lld.%09ld\n", (long long)time_chunk.tv_sec, time_chunk.tv_nsec);
err_remark("delay: %3lld.%09ld\n", (long long)time_delay.tv_sec, time_delay.tv_nsec);
err_remark("file: %s\n", name);
}
read_chunks(fp);
return 0;
}
My SOQ repository also has a script gen-data.sh which makes use of some custom programs to generate a data stream such as this (the seed value is written to standard error, not standard output):
$ gen-data.sh
# Seed: 1313715286
2019-06-03 23:04:16.653: Zunmieoprri Rdviqymcho 5878 2017-03-29 03:59:15 Udransnadioiaeamprirteo
2019-06-03 23:04:18.525: Rndflseoevhgs Etlaevieripeoetrnwkn 9500 2015-12-18 10:49:15 Ebyrcoebeezatiagpleieoefyc
2019-06-03 23:04:20.526: Nrzsuiakrooab Nbvliinfqidbujoops 1974 2020-05-13 08:05:14 Lgithearril
2019-06-03 23:04:21.777: Eeagop Aieneose 6533 2016-11-06 22:51:58 Aoejlwebbssroncmeovtuuueigraa
2019-06-03 23:04:23.876: Izirdoeektau Atesltiybysaclee 4557 2020-09-13 02:24:46 Igrooiaauiwtna
2019-06-03 23:04:26.145: Yhioit Eamrexuabagsaraiw 9703 2014-09-13 07:44:12 Dyiiienglolqopnrbneerltnmsdn
^C
$
When fed into chunker79 with default options, I get output like:
$ gen-data.sh | chunker79
# Seed: 722907235
2019-06-03 23:06:20.570: Aluaezkgiebeewal Oyvahee 1022 2015-08-12 07:45:54 Weuababeeduklleym
2019-06-03 23:06:24.100: Gmujvoyevihvoilc Negeiiuvleem 8196 2015-08-29 21:15:15 Nztkrvsadeoeagjgoyotvertavedi
$
If you analyze the time intervals (look at the first two fields in the output lines), that output meets the specification. A still more detailed analysis is shown by:
$ timecmd -mr -- gen-data.sh | timecmd -mr -- chunker79
2019-06-03 23:09:14.246 [PID 57159] gen-data.sh
2019-06-03 23:09:14.246 [PID 57160] chunker79
# Seed: -1077610201
2019-06-03 23:09:14.269: Woreio Rdtpimvoscttbyhxim 7893 2017-03-12 12:46:57 Uywaietirkekes
2019-06-03 23:09:16.939: Uigaba Nzoxdeuisofai 3630 2017-11-16 09:28:59 Jnsncgoesycsevdscugoathusaoq
2019-06-03 23:09:17.845: Sscreua Aloaoonnsuur 5163 2016-08-13 19:47:15 Injhsiifqovbnyeooiimitaaoir
2019-06-03 23:09:19.272 [PID 57160; status 0x0000] - 5.026s - chunker79
2019-06-03 23:09:22.084 [PID 57159; status 0x8D00] - 7.838s - gen-data.sh
$
There is a noticeable pause in this setup between when the output from chunker79 appears and when gen-data.sh completes. That's due to Bash waiting on all processes in the pipeline to complete, and gen-data.sh doesn't complete until the next time it writes to the pipe after the message that finishes chunker79. This is an artefact of this test setup; it wouldn't be a factor in the shell script outlined in the question.
I would consider writing a safe multi-threaded program with queues.
I know Java better, but there might be more modern suitable languages like Go and Kotlin.
Something like this:
#!/usr/bin/perl
$timeout = 3;
while(<STDIN>) {
# Make sure there is some input
push #out,$_;
eval {
local $SIG{ALRM} = sub { die };
alarm $timeout;
while(<STDIN>) {
alarm $timeout;
push #out,$_;
}
alarm 0;
};
system "echo","process",#out;
}
GNU Parallel 20200122 introduced --blocktimeout (--bt):
find ~ | parallel -j3 --bt 2s --pipe wc
This works like normal GNU Parallel except if it takes > 2 seconds to fill a block. In that case the block read so far is simply passed to wc (unless it is empty).
It has a slightly odd startup behaviour: You have to wait 3*2s (jobslots*timeout) before the output stabilizes, and you get an output at least every 2s.
I have two programs communicating via named pipes (on a Mac), but the buffer size of named pipes is too small. Program 1 writes 50K bytes to pipe 1 before reading pipe 2. Named pipes are 8K (on my system) so program 1 blocks until the data is consumed. Program 2 reads 20K bytes from pipe 1 and then writes 20K bytes to pipe2. Pipe2 can't hold 20K so program 2 now blocks. It will only be released when program 1 does its reads. But program 1 is blocked waiting for program 2. deadlock
I thought I could fix the problem by creating a gasket program that reads stdin non-blocking and writes stdout non-blocking, temporarily storing the data in a large buffer. I tested the program using cat data | ./gasket 0 | ./gasket 1 > out, expecting out to be a copy of data. However, while the first invocation of gasket works as expected, the read in the second program returns 0 before all the data is consumed and never returns anything other than 0 in follow on calls.
I tried the code below both on a MAC and Linux. Both behave the same. I've added logging so that I can see that the fread from the second invocation of gasket starts getting no data even though it has not read all the data written by the first invocation.
#include <stdio.h>
#include <fcntl.h>
#include <time.h>
#include <stdlib.h>
#include <unistd.h>
#define BUFFER_SIZE 100000
char buffer[BUFFER_SIZE];
int elements=0;
int main(int argc, char **argv)
{
int total_read=0, total_write=0;
FILE *logfile=fopen(argv[1],"w");
int flags = fcntl(fileno(stdin), F_GETFL, 0);
fcntl(fileno(stdin), F_SETFL, flags | O_NONBLOCK);
flags = fcntl(fileno(stdout), F_GETFL, 0);
fcntl(fileno(stdout), F_SETFL, flags | O_NONBLOCK);
while (1) {
int num_read=0;
if (elements < (BUFFER_SIZE-1024)) { // space in buffer
num_read = fread(&buffer[elements], sizeof(char), 1024, stdin);
elements += num_read;
total_read += num_read;
fprintf(logfile,"read %d (%d) elements \n",num_read, total_read); fflush(logfile);
}
if (elements > 0) { // something in buffer that we can write
int num_written = fwrite(&buffer[0],sizeof(char),elements, stdout); fflush(stdout);
total_write += num_written;
fprintf(logfile,"wrote %d (%d) elements \n",num_written, total_write); fflush(logfile);
if (num_written > 0) { // copy data to top of buffer
for (int i=0; i<(elements-num_written); i++) {
buffer[i] = buffer[i+num_written];
}
elements -= num_written;
}
}
}
}
I guess I could make the gasket multi-threaded and use blocking reads in one thread and blocking writes in the other, but I would like to understand why non-blocking IO seems to break for me.
Thanks!
My general solution to any IPC project is to make the client and server non-blocking I/O. To do so requires queuing data both on writing and reading, to handle cases where the OS can't read/write, or can only read/write a portion of your message.
The code below will probably seem like EXTREME overkill, but if you get it working, you can use it the rest of your career, whether for named pipes, sockets, network, you name it.
In pseudo-code:
typedef struct {
const char* pcData, * pcToFree; // pcData may no longer point to malloc'd region
int iToSend;
} DataToSend_T;
queue of DataToSend_T qdts;
// Caller will use malloc() to allocate storage, and create the message in
// that buffer. MyWrite() will free it now, or WritableCB() will free it
// later. Either way, the app must NOT free it, and must not even refer to
// it again.
MyWrite( const char* pcData, int iToSend ) {
iSent = 0;
// Normally the OS will tell select() if the socket is writable, but if were hugely
// compute-bound, then it won't have a chance to. So let's call WritableCB() to
// send anything in our queue that is now sendable. We have to send the data in
// order, of course, so can't send the new data until the entire queue is done.
WritableCB();
if ( qdts has no entries ) {
iSent = write( pcData, iToSend );
// TODO: check error
// Did we send it all? We're done.
if ( iSent == iToSend ) {
free( pcData );
return;
}
}
// OK, either 1) we had stuff queued already meaning we can't send, or 2)
// we tried to send but couldn't send it all.
add to queue qdts the DataToSend ( pcData + iSent, pcData, iToSend - iSent );
}
WritableCB() {
while ( qdts has entries ) {
DataToSend_T* pdts = qdts head;
int iSent = write( pdts->cData, pdts->iToSend );
// TODO: check error
if ( iSent == pdts->iToSend ) {
free( pdts->pcToFree );
pop the front node off qdts
else {
pdts->pcData += iSent;
pdts->iToSend -= iSent;
return;
}
}
}
// Off-subject but I like a TINY buffer as an original value, that will always
// exercise the "buffer growth" code for almost all usage, so we're sure it works.
// If the initial buffer size is like 1M, and almost never grows, then the grow code
// may be buggy and we won't know until there's a crash years later.
int iBufSize = 1, iEnd = 0; iEnd is the first byte NOT in a message
char* pcBuf = malloc( iBufSize );
ReadableCB() {
// Keep reading the socket until there's no more data. Grow buffer if necessary.
while (1) {
int iRead = read( pcBuf + iEnd, iBufSize - iEnd);
// TODO: check error
iEnd += iRead;
// If we read less than we had space for, then read returned because this is
// all the available data, not because the buffer was too small.
if ( iRead < iBufSize - iEnd )
break;
// Otherwise, double the buffer and try reading some more.
iBufSize *= 2;
pcBuf = realloc( pcBuf, iBufSize );
}
iStart = 0;
while (1) {
if ( pcBuf[ iStart ] until iEnd-1 is less than a message ) {
// If our partial message isn't at the front of the buffer move it there.
if ( iStart ) {
memmove( pcBuf, pcBuf + iStart, iEnd - iStart );
iEnd -= iStart;
}
return;
}
// process a message, and advance iStart by the size of that message.
}
}
main() {
// Do your initial processing, and call MyWrite() to send and/or queue data.
while (1) {
select() // see man page
if ( the file handle is readable )
ReadableCB();
if ( the file handle is writable )
WritableCB();
if ( the file handle is in error )
// handle it;
if ( application is finished )
exit( EXIT_SUCCESS );
}
}
If we take a very simple counter using ncurses:
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <ncurses.h>
int main(void) {
struct timespec start;
clock_gettime(CLOCK_REALTIME, &start);
initscr();
cbreak();
nodelay(stdscr, TRUE);
{
int key = -1;
struct timespec delay, now;
do {
clock_gettime(CLOCK_REALTIME, &delay);
delay.tv_sec = 0;
delay.tv_nsec = 1000L * 1000L * 1000L - delay.tv_nsec;
nanosleep(&delay, NULL);
clock_gettime(CLOCK_REALTIME, &now);
mvprintw(1, 1, "%ld\n", (long)(now.tv_sec - start.tv_sec));
refresh();
key = getch();
if (key >= 0)
break;
} while (now.tv_sec - start.tv_sec < 60);
}
endwin();
return 0;
}
it aborts after pressing any key (OK, because of cbreak() using ctrl-C would always work without any extra effort...).
But we can make this more complicated, like adding a function to pause the counter or resetting it on the fly (+/- 1 second).
We definitely need a non-blocking keyboard input for this.
I wonder if it possible to do this in Gforth? OK, I know how to catch interrupts like SIGINT there, but something like above, working for any key or a any predetermined key?
Use key?, it returns a flag which is true if new input is available.
You can augment the following code as you see fit, but I think it explains the basic idea of running in a loop until a key is pressed.
: run-until-key ( -- )
0
begin
\ place your terminal code here
." Num:" dup . cr
1+
key? until drop ;
If you want to wait for a specific key, just add an if before the until:
...
key? if key 13 = else false then until
...
You can also add your timer there.
SO,
There are many similar questions, however none that I have been able to use. My code snippet is as follows:
for(int j=0; j<N; j++) {
pid_t pid = fork();
if (pid == -1) {
exit(-1); //err
} else if (pid == 0) {//kid
stringstream ss;
ss<<j;
execlp("./sub","sub",ss.str().c_str(),NULL);
exit(0);
} else {
/* parent */
}
}
my executing code in sub(.cpp) is:
int main( int argc, char **argv )
{
cout<<argv[i]<<endl;
exit(0);
}
my output is as such:
[terminal prompt '$'] 4
2
3
etc.
Is there a way I could prevent the prompt from displaying on the exec call? and why is it ONLY displaying on the first exec call, and not on every one?
What you see is the normal prompt of your shell, because the parent process terminates very quickly. It is not the output of the exec call. The forked processes print their output after the parent process has terminated.
You can use waitpid() in the parent process to "wait" until all forked process have terminated.
so, I have an assignment for my Operating Systems class wherein i am to create a ring of processes connected with pipes in order to pass messages between them. i found some example code which i was looking to adapt (or at least understand) for my needs. the example code (slightly modified) is:
/* Program 4.1 */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
/* Sample C program for generating a unidirectional ring of processes.Invoke this program
with a command-line arg ument indicating the number of processes on the ring. Communication
is done via pipes that connect the standard output of a process to the standard input of
its successor on the ring. After the ring is created, each process identifies itself with
its process ID and the process ID of its parent. Each process then exits. */
void main(int argc, char *argv[ ])
{
int master_pid = getpid();
printf("master pid: %i\n", master_pid);
int i; /* number of this process (starting with 1) */
int childpid; /* indicates process should spawn another */
int nprocs; /* total number of processes in ring */
int fd[2]; /* file descriptors returned by pipe */
int error; /* return value from dup2 call */
/* check command line for a valid number of processes to generate */
if ( (argc != 2) || ((nprocs = atoi (argv[1])) <= 0) ) {
fprintf (stderr, "Usage: %s nprocs\n", argv[0]);
exit(1);
}
/* connect std input to std output via a pipe */
if (pipe (fd) == -1) {
perror("Could not create pipe");
exit(1);
}
printf("%s\n", "test");
//this section is blocking printf()?
if ((dup2(fd[0], STDIN_FILENO) == -1) ||
(dup2(fd[1], STDOUT_FILENO) == -1)) {
perror("Could not dup pipes");
exit(1);
}
printf("%s\n", "test");
if ((close(fd[0]) == -1) || (close(fd[1]) == -1)) {
perror("Could not close extra descriptors");
exit(1);
}
/* create the remaining processes with their connecting pipes */
for (i = 1; i < nprocs; i++) {
if (pipe (fd) == -1) {
fprintf(stderr,"Could not create pipe %d: %s\n",
i, strerror(errno));
exit(1);
}
if ((childpid = fork()) == -1) {
fprintf(stderr, "Could not create child %d: %s\n",
i, strerror(errno));
exit(1);
}
if (childpid > 0) /* for parent process, reassign stdout */
error = dup2(fd[1], STDOUT_FILENO);
else
error = dup2(fd[0], STDIN_FILENO);
if (error == -1) {
fprintf(stderr, "Could not dup pipes for iteration %d: %s\n",
i, strerror(errno));
exit(1);
}
if ((close(fd[0]) == -1) || (close(fd[1]) == -1)) {
fprintf(stderr, "Could not close extra descriptors %d: %s\n",
i, strerror(errno));
exit(1);
}
if (childpid)
break;
}
/* say hello to the world */
fprintf(stderr,"This is process %d with ID %d and parent id %d\n",
i, (int)getpid(), (int)getppid());
wait(1);
exit (0);
} /* end of main program here */
which outputs:
master pid: 30593
test
This is process 1 with ID 30593 and parent id 30286
This is process 2 with ID 30594 and parent id 30593
when i give is 2 as argv[1]
so, I'm wondering, why would the dup2 section prevent the printf() from executing? if i cant even print something, i'm not sure if i could even pass the message correctly. also, why would the fprintf() already there work, but not one that i would put there?
edit: i would take this to my professor/TA, but theyre both out of town and will be unreachable between now and the deadline...
printf prints to stdout, which is file descriptor 1 (or equivalently STDOUT_FILENO). dup2(3) is duplicating the pipe's file descriptor on top of the current stdout, which has the side effect of closing the current stdout. So, when you try to printf after calling that particular dup2, you're really printing the data into the pipe you just created, which doesn't go to your terminal output.
fprintf(stderr, ...) still works because that prints to stderr, not stdout, and the stderr file descriptor (2, or equivalently STDERR_FILENO) does not change during the program, so it continues to print out to the terminal.
printf() does not send data to path 0, it sends buffered data using stdout. It would seem that when you disrupt path 0 by dup2'ing something to it, you're disrupting stdout in the process.
From the man page on dup2: dup2() makes newfd be the copy of oldfd, closing newfd first if necessary. Thus when you call dup2(fd[0], STDIN_FILENO) you are breaking stdout.
You state that fprintf() is working but printf() is not... what path are you using for fprintf()? If you're using stderr then it makes perfect sense that it would continue to work, since you haven't done anything with that path.