Ordering problem when using Ruby popen2 to interact with C++ iostreams - ruby

I'm using Open3's popen2 to interact with a simple C++ program's iostreams. My understanding is that std::cin and std::cout are independent, but the order in which I have my popen2 block's IO objects read/write calls seems to make a difference. My C++ program is:
int main(int argc, char** argv) {
std::string input;
std::cout<<"EXECUTE TASK"<<std::endl;
std::cin>>input;
std::cout<<"END"<<std::endl;
}
My ruby script is:
require 'open3'
expected_string = "EXECUTE TASK"
Open3.popen2('~/Sandbox/a.out') { |stdin, stdout|
stdin.write('\n')
stdin.close
results = stdout.readlines
puts results
}
The above works fine, but if I move the stdout.readlines before the stdin.close, the ruby script will hang. My intent is to conditionally write the \n to stdin if the C++ program writes expected_string to standard out first, but I'm forced to close the stdin stream before I can execute readlines. Like I said, my understanding was that the two streams are independent, and the file descriptors returned by popen2 appear to be independent as well, so why would the order matter?
Any help is appreciated. Thanks.
Solution with full scope of what I was trying to accomplish (someone may find this helpful):
int main(int argc, char** argv) {
std::string input;
std::cout<<"1"<<std::endl;
std::cout<<"2"<<std::endl;
std::cout<<"3"<<std::endl;
std::cout<<"4"<<std::endl;
std::cout<<"5"<<std::endl;
std::cout<<"EXECUTE TASK"<<std::endl;
std::cout.flush();
std::cin>>input;
std::cout<<"END"<<std::endl;
}
require 'open3'
expected_string = "EXECUTE TASK"
Open3.popen2('~/Sandbox/a.out') { |stdin, stdout|
found = false
begin
while(result = stdout.readline)
puts result
if(result.include?(expected_string))
found = true
break
end
end
rescue
raise "Exception caught while reading lines"
end
stdin.write('\n')
stdin.close
}

Looks like a deadlock:
The ruby script is calling IO::readlines, which will not return until the entire stream has been read.
The C++ program does not actually terminate until it receives a carriage return.
You may want to call IO::readline, which will return each line as it is received, or reorder the two scripts so that there is not a deadlock.

It is highly likely that you just need stdout.flush after anything that you expect your correspondent to act on.
Because the pipe set up by #popen is not a tty, it will not default to line buffering. It will do block buffering. You will need to force your stream to act in datagram fashion by calling #flush on "record" boundaries.
Also, see IO#sync= for a way to flush all I/O automatically.

Related

I am trying to write a shell program to execute more than one command at a time

My Code
#include<stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/wait.h>
int main()
{
char * arg_list[3];
arg_list[0] = "ls";
arg_list[1] = "-l";
arg_list[2] = 0;
char *arg_list2[3];
arg_list2[0] = " ps";
arg_list2[1] = "-ef";
arg_list2[2] = 0;
for(int i=0;i<5;i++){ // loop will run n times (n=5)
if(fork() == 0) {
if (i == 0){
execvp("ls", arg_list);
}else if(i==1){
execvp("ps" , arg_list2);
}else if(i>1){
printf("[son] pid %d from [parent] pid %d\n",getpid(),getppid());
exit(0);
}
}
}
for(int i=0;i<5;i++) // loop will run n times (n=5)
wait(NULL);
}
ME trying to modify it
#include<stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/wait.h>
int main()
{
for(int i=0;i<5;i++){ // loop will run n times (n=5)
if(fork() == 0) {
printf("[son] pid %d from [parent] pid %d\n",getpid(),getppid());
execlp(argv[i],argv[i],argv[i+1],(char*)NULL);
exit(0);
}
}
for(int i=0;i<5;i++) // loop will run n times (n=5)
wait(NULL);
}
-- NEED GUIDANCE AND UNDERSTANDING
I am trying to make my own tiny little shell program. When I run my first code works fine, runs all commands on the command line. But I cannot know and define all commands the user might enter. So i am trying to get a base code which could run any commands single or multiple entered by user. I tried using execlp where it does not compile saying argv is not defined which is true as i don't want to specifically define it.
I am trying to make my own tiny little shell program. When I run my first code works fine, runs all commands on the command line. But I cannot know and define all commands the user might enter.
For sure.... A shell program purpose is basically:
Read user input
Execute user input
Return result of execution.
There's nothing in your code that read user input....
So i am trying to get a base code which could run any commands single or multiple entered by user.
So read user input ;-)
I tried using execlp where it does not compile saying argv is not defined which is true as i don't want to specifically define it.
For sure ... but how would GCC guessed that `argv[]̀ must be automaticallty filled with user input ?
There's nothing automatic when coding in C language. You have to manage this manually.
Also, note that argc, argv et envp are usually reserved for main() function:
main(int argc, char **argv, char **envp)
So you may use something else to build your command array.
In pseudo code, what you must implement is:
quit=0
while (quit = 0) {
command_to_run = read_user_input();
if (command_to_run == "exit") {
quit = 1;
} else {
execute(command_to_run);
}
}
Some advices:
Try to use more functions. For example, implement a fork_and_run(char **cmd) function to fork and then execute command provided by the user. Il will make your code more readable and easy to maintain.
Read carefully manpages: everything you should know (like, for example, the fact that array provided to execvp() must be NULL-terminated) is written in it.
Your debugging messages should be printed to stderr. The result of the command run must be printed to stdin, so use fprintf() instead of printf() to write to the correct stream.
I would use a #define debug(x) fprintf(stderr, x) or something similar for debugging output so that you can easily disable later ;-)

Boost process continuously read output

I'm trying to read outputs/logs from different processes and display them in a GUI. The processes will be running for long time and produce huge output. I'm planning to stream the output from those processes and display them according to my needs. All the while allow my gui application to take user inputs and perform other actions.
What I've done here is, from main thread launch two threads for each process. One for launching the process and another for reading output from the process.
This is the solution I've come up thus far.
// Process Class
class MyProcess {
namespace bp = boost::process;
boost::asio::io_service mService; // member variable of the class
bp::ipstream mStream // member variable of the class
std::thread mProcessThread, mReaderThread // member variables of the class.
public void launch();
};
void
MyProcess::launch()
{
mReaderThread = std::thread([&](){
std::string line;
while(getline(mStream, line)) {
std::cout << line << std::endl;
}
});
mProcessThread = std::thread([&]() {
auto c = boost::child ("/path/of/executable", bp::std_out > mStream, mService);
mService.run();
mStream.pipe().close();
}
}
// Main Gui class
class MyGui
{
MyProcess process;
void launchProcess();
}
MyGui::launchProcess()
{
process.launch();
doSomethingElse();
}
The program is working as expected so far. But I'm not sure if this is the correct solution. Please let me know if there's any alternative/better/correct solution
Thanks,
Surya
The most striking conceptual issues I see are
Process are asynchronous, no need to add a thread to run them.¹
You prematurely close the pipe:
mService.run();
mStream.pipe().close();
Run is not "blocking" in the sense that it will not wait for the child to exit. You could use wait to achieve that. Other than that, you can just remove the close() call.
With the close means you will lose all or part of the output. You might not see any of the output if the child process takes a while before it outputs the first data.
You are accessing the mStream from multiple threads without synchronization. This invokes Undefined Behaviour because it opens a Data Race.
In this case you can remove the immediate problem by removing the mStream.close() call mentioned before, but you must take care to start the reader-thread only after the child has been initialized.
Strictly speaking the same caution should be taken for std::cout.
You are passing the io_service reference, but it's not being used. Just dropping it seems like a good idea.
The destructor of MyProcess needs to detach or join the threads. To prevent Zombies, it needs to detach or reap the child pid too.
In combination with the lifetime of mStream detaching the reader thread is not really an option, as mStream is being used from the thread.
Let's put out the first fixes first, and after that I'll suggest show some more simplifications that make sense in the scope of your sample.
First Fixes
I used a simple bash command to emulate a command generating 1000 lines of ping:
Live On Coliru
#include <boost/process.hpp>
#include <thread>
#include <iostream>
namespace bp = boost::process;
/////////////////////////
class MyProcess {
bp::ipstream mStream;
bp::child mChild;
std::thread mReaderThread;
public:
~MyProcess();
void launch();
};
void MyProcess::launch() {
mChild = bp::child("/bin/bash", std::vector<std::string> {"-c", "yes ping | head -n 1000" }, bp::std_out > mStream);
mReaderThread = std::thread([&]() {
std::string line;
while (getline(mStream, line)) {
std::cout << line << std::endl;
}
});
}
MyProcess::~MyProcess() {
if (mReaderThread.joinable()) mReaderThread.join();
if (mChild.running()) mChild.wait();
}
/////////////////////////
class MyGui {
MyProcess _process;
public:
void launchProcess();
};
void MyGui::launchProcess() {
_process.launch();
// doSomethingElse();
}
int main() {
MyGui gui;
gui.launchProcess();
}
Simplify!
In the current model, the thread doesn't pull it's weight.
I you'd use io_service with asynchronous IO instead, you could even do away with the whole thread to begin with, by polling the service from inside your GUI event loop².
If you're gonna have it, and since child processes naturally execute asynchronously³ you could simply do:
Live On Coliru
#include <boost/process.hpp>
#include <thread>
#include <iostream>
std::thread launch(std::string const& command, std::vector<std::string> args = {}) {
namespace bp = boost::process;
return std::thread([=] {
bp::ipstream stream;
bp::child c(command, args, bp::std_out > stream);
std::string line;
while (getline(stream, line)) {
// TODO likely post to some kind of queue for processing
std::cout << line << std::endl;
}
c.wait(); // reap PID
});
}
The demo displays exactly the same output as earlier.
¹ In fact, adding threads is asking for trouble with fork
² or perhaps idle tick or similar idea. Qt has a ready-made integration (How to integrate Boost.Asio main loop in GUI framework like Qt4 or GTK)
³ on all platforms supported by Boost Process

Why does libncurses ui not display when capturing to Bash variable?

I've written a libcurses-based ascii ui that writes text to stdout when the program exits.
If I execute the program alone, like so...
> ./test
...the ui displays.
However, if I try to capture the program output to a Bash variable, like so...
> foo=$(./test)
...the ui does not display, however the Bash variable captures the expected output.
Does anyone know why this behavior is as such? Is there a way to get the ui to show up when trying to capture its stdout to a Bash variable?
The Code
#include <iostream>
#include <curses.h>
#include <unistd.h>
int main(int argc, char* argv[])
{
WINDOW* pWindow = initscr();
keypad(pWindow, TRUE);
curs_set(0);
nodelay(pWindow, FALSE);
mvwprintw(pWindow, 5, 5, "hello, world!");
mvwprintw(pWindow, 6, 5, "hello, fold!");
mvwprintw(pWindow, 7, 5, "hello, toad!");
for (int i = 0; i < 5; ++i)
{
mvwprintw(pWindow, 5 + i, 1, "==>");
refresh();
usleep(500000);
mvwprintw(pWindow, 5 + i, 1, " ");
refresh();
}
endwin();
std::cout << "bar" << std::endl;
}
Simply because redirecting the std output (>, a=$(…)) just redirects the standard output – ncurses, on the other hand, directly talks to the terminal and displays characters, that never are part of stdout.
Short: it doesn't capture the output because there is none. Instead, ncurses programs directly talk to the underlying terminal.
Is there a way to get the ui to show up when trying to capture its stdout to a Bash variable?
I don't recommend that. Because you're mixing non-interactive usage (getting std output) and interactive, and that can't really go well in the end, but:
you can end your ncurses session and just use printf like any other C programmer. Then you'd actually be producing std output.
I'd much rather just add an option to my program that takes a file to which I write my output. Then the bash script could open that file after my program has run.
When you initialize curses using initscr, it will use the standard output for display. (You could use newterm to specify another output). So when you redirect the output of the program you will not see the user-interface.
Adapting your example,
#!/bin/bash
g++ -o test foo.c $(ncursesw6-config --cflags --libs)
foo=$(./test)
set >foo.log
and looking at what bash puts in $foo, I see the expected control characters which are written in the user interface, e.g.,
foo=$'\E[?1049h\E[1;40r\E(B\E[m\E[4l\E[?7h\E[?1h\E=\E[?25l\E[H\E[2J\E[6d ==> hello, world!\n\E[6Ghello, fold!\n\E[6Ghello, toad!\E[6;5H\r \r\n ==>\r \r\n ==>\r \r\n ==>\r\E[J \r\n ==>\r\E[J \E[40;1H\E[?12l\E[?25h\E[?1049l\r\E[?1l\E>bar'

main() program won't exit normally

My C++ 2011 main() program for DiGSE is:
int main(int argc, char* argv[]) {
. . .
return EXIT_SUCCESS;
} // this } DOES match the opening { above
It compiles and executes correctly. A print statement immediately before the return outputs normally. However, a Windows 7.1 notification pops up saying "DiGSE.exe has stopped working." It then graciously offers to search the web for a solution.
I tried replacing the return with return 0; exit(0); and nothing so execution falls out the bottom (which, as I understand, is acceptable). However, in all cases I still get the pop-up.
What do I do to get the main() to exit gracefully?
DiGSE is just the name of the Windows 7 executable compiled on MinGW 4.9.2. The "full" program is already stripped down:
int main(int argc, char* argv[]) {
try {
DiGSE::log_init(DiGSE::log_dest_T::console_dest, "dig.log", true,
DiGSE::log_lvl_T::trace_lvl);
}//try
catch (const std::exception& ex) {
std::cerr << FMSG("\n"
"Executing '%1%' raised this exception:\n"
" %2%", % DiGSE::Partition::productName()
% ex.what())
<< std::endl;
return EXIT_FAILURE;
}//exception
catch (...) {
std::cerr << FMSG("\n"
"Executing '%1%' instance raised an unknown exception.",
% DiGSE::Partition::productName())
<< std::endl;
return EXIT_FAILURE;
}//exception
L_INFO(FMSG("'%1% v%2%' terminated normally.",
% DiGSE::Partition::productName()
% DiGSE::Partition::productVersion()))
return EXIT_SUCCESS;
}//main()
The L_INFO() is a logging call, which outputs as it should. The log_init() at the top initializes the log. Commenting out log_init() and L_INFO() has the same result as originally reported.
Program received signal SIGSEGV, Segmentation fault.
0x000000006fc8da9d in libstdc++-6!_ZNSo6sentryC1ERSo ()
from D:\Program Files\mingw-w64\x86_64-4.9.2-posix-seh-rt_v3-rev0\mingw64\bin
\libstdc++-6.dll
This is what gdb returns while mail() is exiting. It does this even with the log_init() and L_LNFO() commented out. So the problem is probably in one of globals of something it's linked to.
It is completely possible for a program to crash after the end of main -- the program isn't over yet. The following items execute after main() returns:
Registered at_exit handlers
Destructors for main()'s own automatic variables, and all variables with static storage duration (globals and function-static) (C++ only)
DllMain(PROCESS_DETACH) code in all dynamic libraries you are using (Windows only)
In addition to that, various events can occur outside your program and cause failures which you might mistake for a failure of your program (especially if your program forks or spawns copies of itself):
SIGCHLD is raised (on *nix). Process handles become signaled and cause wait functions to return (on Windows)
All open handles (file descriptors) get abandoned, and the close handler in the driver is invoked
The other end of connections (pipes, sockets) shift into a disconnected state (reads return 0, writes fail, on *nix SIGHUP may be raised)
I suggest attaching a debugger, set a breakpoint at the end of main, and then single-step through the cleanup code to find out where the failure is occurring. Divide and conquer may also be helpful (cut out some global variables, or all usage of a particular DLL).

get all running processes info using QProcess

few days ago i asked about how to get all running processes in the system using QProcess.
i found a command line that can output all processes to a file:
C:\WINDOWS\system32\wbem\wmic.exe" /OUTPUT:C:\ProcessList.txt PROCESS get Caption
this will create C:\ProcessList.txt file contains all running processes in the system.
i wonder how can i run it using QProcess and take its output to a variable.
it seems every time i try to run it and read nothing happens:
QString program = "C:\\WINDOWS\\system32\\wbem\\wmic.exe";
QStringList arguments;
arguments << "/OUTPUT:C:\\ProcessList.txt" <<"PROCESS"<< "get"<< "Caption";
process->setStandardOutputFile("process.txt");
process->start(program,arguments);
QByteArray result = process->readAll();
i prefer not to create process.txt at all and to take all the output to a variable...
You can run wmic.exe with "/OUTPUT:STDOUT" switch to print the process info directly to stdout. However, I was unable to read this info through QProcess API and save it in variable. Here's the code I used:
#include <QtCore/QCoreApplication>
#include <QProcess>
#include <QDebug>
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
QProcess process;
process.setReadChannel(QProcess::StandardOutput);
process.setReadChannelMode(QProcess::MergedChannels);
// process.start("cmd.exe /C echo test");
process.start("wmic.exe /OUTPUT:STDOUT PROCESS get Caption");
process.waitForStarted(1000);
process.waitForFinished(1000);
QByteArray list = process.readAll();
qDebug() << "Read" << list.length() << "bytes";
qDebug() << list;
}
This code successfully captures output of "cmd.exe /C echo test", but doesn't work on wmic.exe. It seems that process wmic.exe is never finished, and I suppose it's stdout is never flushed so you don't receive anything throught QProcess::readAll().
That's all help I can give you. Maybe you, or some other SO user will find bug in the snippet above.
Try this it will work well.
process.start("cmd", QStringList() << "/C" << "echo" << "process" << "get" << "caption" << "|" << "wmic");

Resources