I am attempting to kick off multiple commands in a bash script but wait for them to complete
It looks something like this:
A &
B &
C &
D
Unfortunately i don't know which of these processes will finish first. But i need the whole script to complete when finished with all processes.
So like a rookie i tried:
(A &
B &
C &
D) && E
Unfortunately E only execs after D completes. I would like it if i could get E to happen after A - D exec
Hopefully that sums the problem.
Thanks
A &
B &
C &
D &
wait
E
From the help listing:
wait: wait [-n] [id ...]
Wait for job completion and return exit status.
Waits for each process identified by an ID, which may be a process ID or a
job specification, and reports its termination status. If ID is not
given, waits for all currently active child processes, and the return
status is zero. If ID is a a job specification, waits for all processes
in that job's pipeline.
If the -n option is supplied, waits for the next job to terminate and
returns its exit status.
Exit Status:
Returns the status of the last ID; fails if ID is invalid or an invalid
option is given.
wait(1) is the canonical solution, but I've used a q&d solution in the past:
( A & B & C & D & ) | cat; E
Related
I'm trying to parallelize work. I'm trying to have subprocesses each take a piece of the work, then wait for the work to complete from the parent process.
The following solution seems to work for small examples:
let par_iter (items: 'i list) ~(f: 'i -> unit): unit =
let orig_pid = Unix.getpid () in
let rec loop = function
| [] -> ()
| h :: t ->
match Unix.fork () with
| 0 -> f h
| _ -> loop t
in
loop items;
if Unix.getpid () = orig_pid then
List.iter items ~f:(fun _ -> ignore ## Unix.wait ())
else
exit 0
Usage:
let () =
par_iter [1; 2; 3; 4] ~f:do_something_with_an_int;
The contract of Unix.wait when there are multiple subprocesses is not very clear to me. Here I'm relying on the behavior where waiting n times will wait for all and only n subprocesses to finish.
Does this solution rely on undefined behavior? Is there an alternative way to correctly implement par_iter such that it spawns work on child processes and waits for the work to finish?
tested on OCaml 4.14.0
If the original parent process already had some subprocesses, this can fail. Thus, IMHO it's not usable as a general library. You should use Unix.waitpid to wait specifically for the processes you created.
I'm trying to understand why I'm not getting an Err() result when using bash -c expression to run a command.
Here is an example and the output below. I expect out2 and out3 to be an Err() but out3 is an Ok() with the failed status.
I'm using bash -c to execute a command from a given string, which is easy.
It's possible to get an Err() result using bash -c syntax?
#![allow(unused)]
use std::process::{Command, Output};
fn main() {
let out1 = Command::new("bash").arg("-c").arg("ls").output();
println!("out1: {:?}", out1);
let out2 = Command::new("wrongcommand").arg("-c").arg("ls").output();
println!("out2: {:?}", out2);
let out3 = Command::new("bash").arg("-c").arg("wrongcommand").output();
println!("out3: {:?}", out3);
}
Output:
out1: Ok(Output { status: ExitStatus(ExitStatus(0)), stdout: "Cargo.lock\nCargo.toml\ncrate-information.json\nsrc\ntarget\n", stderr: "" })
out2: Err(Os { code: 2, kind: NotFound, message: "No such file or directory" })
out3: Ok(Output { status: ExitStatus(ExitStatus(32512)), stdout: "", stderr: "bash: wrongcommand: command not found\n" })
I tried from command line and
$ bash -c wrongcommand
and
$ wrongcommand
and both return the same exit code (127). That's why I expected that Command failed in the same way.
That can be explained pretty easily. Follow along on the list of error codes
out1 is Ok for obvious reasons
out2 is an Err type because Command directly looked for the process, could not find it, and returned ENOENT (code 2). This error happened within rust, nothing was actually executed.
out3 is Ok because the process that Command ran is bash, and it returned its status. In this case, it wouldn't have found the command, so by bash statuses, it'll have returned 127. However, it's not that easy, because there is an additional layer of information contained in ExitStatus. On unix, when a process fails, it actually returns a 16bit/32bit integer (based on platform/libc/etc), separated in two:
The highest bits are what signal triggered the return
The lowest 8 bits are the return status of the process
And, to no surprise, if we shift 32512 8 bits to the right (it was an u16), we get... 127!
The takeaway is simple:
Err definitely means the process didn't run
Ok means the main process ran, but you'll need to check ExitStatus::success() to confirm that it actually did (i.e. 0 as exit status)
You could recover it like so:
Command::new("bash").arg("-c").arg("wrongcommand").output().and_then(|r| match r.status.success() {
true => Ok(r),
false => Err(io::Error::new(io::ErrorKind::InvalidData, "Process error"))
});
You can play with this on the playground. success() is a decently reliable indicator of child process success; all it does is check if the least significant 8 bits of the exit status is non-zero.
Obviously, this does not help if a process returns non-zero on success, but that's a different problem.
can some one explain this code ?
int main ( ){
int i=0 ;
while (fork() !=0 && i<2)
i=i+1;
printf(" this is the process %d and ends with i=%d \n", getpid(), i);
return 0;
}
what I have understand that a process father has 3 children !
but according to this execution output I am not sure that I have understood the fork function :
[root#www Desktop]# ./prog1
this is the process 8025 and ends with i=2
[root#www Desktop]# this is the process 8027 and ends with i=1
this is the process 8028 and ends with i=2
this is the process 8026 and ends with i=0
Thank You !
Remember fork() forks your process, resulting in two more-or-less identical processes, the difference in each is the return value of fork() is 0 for the child, and the child's pid for the parent.
Your while loop only iterates for the parent processes (it ends for the child processes since in those processes the return value of fork() is 0). So the first time through (i==0), the child process falls through, prints its pid, and exits. The parent remains.
The parent increments i, forks again, the child (i==1) falls through, prints its pid and exits. So that's one exit with i==0 and one exit with i==1.
The parent increments i, forks again, but i is now 2, so the while loop exits for both parent and child processes. Both exit with i==2. So in total that's one exit i==0, one exit with i==1 and two exits with i==2.
A couple of other points to bear in mind:
processes are not guaranteed to be sequential, so the output may be out-of-(expected)-order (as it is in your example)
an optimising compiler may also mess with sequencing. Compiling with -O0 may make the output (sequence) more what you expect.
$ gcc -w -O0 forktest.c && ./a.out
this is the process 5028 and ends with i=0
this is the process 5029 and ends with i=1
this is the process 5030 and ends with i=2
this is the process 5027 and ends with i=2
In Ruby, I'm using Process.spawn to run a command in a new process. I've opened a bidirectional pipe to capture stdout and stderr from the spawned process. This works great until the bytes written to the pipe (stdout from the command) exceed 64Kb, at which point the command never finishes. I'm thinking the pipe buffer size has been hit, and writes to the pipe are now blocked, causing the process to never finish. In my actual application, i'm running a long command that has lots of stdout that I need to capture and save when the process has finished. Is there a way to raise the buffer size, or better yet have the buffer flushed so the limit is not hit?
cmd = "for i in {1..6600}; do echo '123456789'; done" #works fine at 6500 iterations.
pipe_cmd_in, pipe_cmd_out = IO.pipe
cmd_pid = Process.spawn(cmd, :out => pipe_cmd_out, :err => pipe_cmd_out)
Process.wait(cmd_pid)
pipe_cmd_out.close
out = pipe_cmd_in.read
puts "child: cmd out length = #{out.length}"
UPDATE
Open3::capture2e does seem to work for the simple example I showed. Unfortunately for my actual application, I need to be able to get the pid of the spawned process as well, and have control of when I block execution. The general idea is I fork a non blocking process. In this fork, I spawn a command. I send the command pid back to the parent process, then I wait on the command to finish to get the exit status. When command is finished, exit status is sent back to parent. In the parent, a loop is iterating every 1 second checking the DB for control actions such as pause and resume. If it gets a control action, it sends the appropriate signal to the command pid to stop, continue. When the command eventually finishes, the parent hits the rescue block and reads the exit status pipe, and saves to DB. Here's what my actual flow looks like:
#pipes for communicating the command pid, and exit status from child to parent
pipe_parent_in, pipe_child_out = IO.pipe
pipe_exitstatus_read, pipe_exitstatus_write = IO.pipe
child_pid = fork do
pipe_cmd_in, pipe_cmd_out = IO.pipe
cmd_pid = Process.spawn(cmd, :out => pipe_cmd_out, :err => pipe_cmd_out)
pipe_child_out.write cmd_pid #send command pid to parent
pipe_child_out.close
Process.wait(cmd_pid)
exitstatus = $?.exitstatus
pipe_exitstatus_write.write exitstatus #send exitstatus to parent
pipe_exitstatus_write.close
pipe_cmd_out.close
out = pipe_cmd_in.read
#save out to DB
end
#blocking read to get the command pid from the child
pipe_child_out.close
cmd_pid = pipe_parent_in.read.to_i
loop do
begin
Process.getpgid(cmd_pid) #when command is done, this will except
#job.reload #refresh from DB
#based on status in the DB, pause / resume command
if #job.status == 'pausing'
Process.kill('SIGSTOP', cmd_pid)
elsif #job.status == 'resuming'
Process.kill('SIGCONT', cmd_pid)
end
rescue
#command is no longer running
pipe_exitstatus_write.close
exitstatus = pipe_exitstatus_read.read
#save exit status to DB
break
end
sleep 1
end
NOTE: I cannot have the parent poll the command output pipe because the parent would then be blocked waiting for the pipe to close. It would not be able to pause and resume the command via the control loop.
This code seems to do what you want, and may be illustrative.
cmd = "for i in {1..6600}; do echo '123456789'; done"
pipe_cmd_in, pipe_cmd_out = IO.pipe
cmd_pid = Process.spawn(cmd, :out => pipe_cmd_out, :err => pipe_cmd_out)
#exitstatus = :not_done
Thread.new do
Process.wait(cmd_pid);
#exitstatus = $?.exitstatus
end
pipe_cmd_out.close
out = pipe_cmd_in.read;
sleep(0.1) while #exitstatus == :not_done
puts "child: cmd out length = #{out.length}; Exit status: #{#exitstatus}"
In general, sharing data between threads (#exitstatus) requires more care, but it works
here because it's only written once, by the thread, after initialization. (It turns out
$?.exitstatus can return nil, which is why I initialized it to something else.) The call
to sleep() is unlikely to execute even once since the read() just above it won't complete
until the spawned process has closed its stdout.
Indeed, your diagnosis is likely correct. You could implement a select and read loop on the pipe while waiting for the process to end, but likely you can get what you want more simply with stdlib Open3::capture2e.
I'm writing a fake shell, where I create a child process and then call execvp(). In the normal shell, when I enter an unknown command such as 'hello' it returns 'hello: Command not found.' However, when I pass hello into execvp(), it doesn't return any error by default and just continues running the rest of my program like nothing happened. What's the easiest way to find out if nothing was actually run? here's my code:
if(fork() == 0)
{
execvp(cmd, args);
}
else
{
int status = 0;
int corpse = wait(&status);
printf(Child %d exited with a status of %d\n", corpse, status);
}
I know that if corpse < 0, then it's an unknown command, but there are other conditions in my code not listed where I don't want to wait (such as if & is entered at the end of a command). Any suggestions?
All of the exec methods can return -1 if there was an error (errno is set appropriately). You aren't checking the result of execvp so if it fails, the rest of your program will continue executing. You could have something like this to prevent the rest of your program from executing:
if (execvp(cmd, args) == -1)
exit(EXIT_FAILURE);
You also want to check the result of fork() for <0.
You have two independent concerns.
1) is the return value of execvp. It shouldn't return. If it does there is a problem. Here's what I get execvp'ing a bad command. You don't want to wait if execvp fails. Always check the return values.
int res = execvp(argv[1], argv);
printf ("res is %i %s\n", res, strerror(errno));
// => res is -1 No such file or directory
2) The other concern is background processes and such. That's the job of a shell and you're going to need to figure out when your program should wait immediately and when you want to save the pid from fork and wait on it later.