I am trying to parallelize my program with MPI.
MPI_File fh;
MPI_File_open(MPI_COMM_WORLD,"input.txt",MPI_MODE_CREATE|MPI_MODE_RDONLY, MPI_INFO_NULL, &fh);
if(rank == 0){
nwords = -1;
do {
err = fscanf(fh, "%[^\n]\n", word[++nwords]);
} while( err != EOF && nwords < maxwords);
printf("Read in %d words\n", nwords);
}
then I got this error.
warning #167: argument of type "MPI_File" is incompatible with parameter of type "FILE *__restrict__"
err = fscanf(fh, "%[^\n]\n", word[++nwords]);
how can I read a file using MPI_File_open?
MPI_File_open works with MPI_File, and fscanf() works with FILE * and there is no interoperability.
You have to either
- MPI_File_open() and MPI_File_read()
- or stick to fopen() and fscanf()
MPI-IO true potential is unleashed when doing collective IO (e.g. MPI_File_read_all()) and there is no such thing as MPI_File_scanf() so unless you are willing to MPI_File_read_all() and sscanf(), you might want to stick with non MPI subroutines.
Related
Now I'm trying to download file from server (Linux) to my host (windows).
It looks working well (other functions) but when I'm trying to read file from server. Data in the file is not proper.
for example)
server text file:
A
B
C
D
E
Reading file and saving to buffer:
A
B
C
I checked small size text file. It was working well. However now I'm in trouble because text file is bigger than before.
Here is my code.
int scp_receive(ssh_session session)
{
ssh_scp scp;
int rc;
int size, mode;
char *filename;
char *buffer;
int fd;
scp = ssh_scp_new(session, SSH_SCP_READ, "/home/abc/27");
if (scp == NULL)
{
fprintf(stderr, "Error allocating scp session: %s\n",
ssh_get_error(session)); return SSH_ERROR;
}
rc = ssh_scp_init(scp);
if (rc != SSH_OK)
{
fprintf(stderr, "Error initializing scp session: %s\n",ssh_get_error(session));
ssh_scp_free(scp);
return rc;
}
rc = ssh_scp_pull_request(scp);
if (rc != SSH_SCP_REQUEST_NEWFILE)
{
fprintf(stderr, "Error receiving information about file: %s\n",ssh_get_error(session));
return SSH_ERROR;
}
size = ssh_scp_request_get_size(scp);
std::cout << "size is:" << size << std::endl;
filename = strdup(ssh_scp_request_get_filename(scp));
mode = ssh_scp_request_get_permissions(scp);
printf("Receiving file %s, size %d, permisssions 0%o\n", filename, size, mode);
free(filename);
buffer = (char *)malloc(sizeof(char)*size);
if (buffer == NULL)
{
fprintf(stderr, "Memory allocation error\n");
return SSH_ERROR;
}
ssh_scp_accept_request(scp);
rc = ssh_scp_read(scp, buffer, sizeof(char)*size);
if(rc == SSH_ERROR)
{
fprintf(stderr, "Error receiving file data: %s\n", ssh_get_error(session));
free(buffer);
return rc;
}
std::cout << buffer << std::endl;
printf("Done\n");
//write(fd, buffer, size);
char path[30];
sprintf(path, "C:/Check/%s", filename);
if (access(path, F_OK) != 0) {
//TODO: file
remove(path);
}
if(0 < (fd=open(path, O_RDWR | O_CREAT | O_TRUNC, mode)))
{
write(fd, buffer, sizeof(char)*size);
close(fd);
}
else
{
std::cerr << "failed to open" << std::endl;
}
free(buffer);
//rc = ssh_scp_pull_request(scp);
//if(rc != SSH_SCP_REQUEST_EOF)
//{
// fprintf(stderr, "Unexpected request: %s\n",ssh_get_error(session));
// return SSH_ERROR;
//}
ssh_scp_close(scp);
ssh_scp_free(scp);
return SSH_OK;
}
The issue here is with the libssh read function. It only reads a max of 65536 bytes at once. It won't read anything large than that block size.
If you take a look at the libssh SCP read function here: http://xtalopt.openmolecules.net/wiki/index.fcgi/browser/src/libssh/scp.c?rev=2b0288492ad2481ee8bdbb8c1f9d5c453a044eee
You will see the size is limited to 65536.
SOLUTION
In your application, you should attempt to read multiple smaller chunks of memory till you finish reading the large text file. And since you have the size of the large text file, this can be done easily via a loop.
I know this is a late answer, but hope this can help someone else having this issue.
I am writing a C++ (Windows) client console application which reads from an anonymous pipe on STDIN. I would like to be able to use my program as follows:
echo input text here | my_app.exe
and do something in the app with the text that is piped in
OR
my_app.exe
and then use some default text inside of the app instead of the input from the pipe.
I currently have code that successfully reads from the pipe on STDIN given the first situation:
#include <Windows.h>
#include <iostream>
#include <string>
#define BUFSIZE 4096
int main(int argc, const char *argv[]) {
char char_buffer[BUFSIZE];
DWORD bytes_read;
HANDLE stdin_handle;
BOOL continue_reading;
unsigned int required_size;
bool read_successful = true;
stdin_handle = GetStdHandle(STD_INPUT_HANDLE);
if (stdin_handle == INVALID_HANDLE_VALUE) {
std::cout << "Error: invalid handle value!\n\n";
} else {
continue_reading = true;
while (continue_reading) {
continue_reading = ReadFile(stdin_handle, char_buffer, BUFSIZE,
&bytes_read, NULL);
if (continue_reading) {
if (bytes_read != 0) {
// Output what we have read so far
for (unsigned int i = 0; i < bytes_read; i++) {
std::cout << char_buffer[i];
}
} else {
continue_reading = false;
}
}
}
}
return 0;
}
I know that my only option with anonymous pipes is to do a blocking read with ReadFile. If I understand correctly, in regard to how I am invoking it, ReadFile will continue to read from the buffer on STDIN until it detects an end of write operation on the other end of the pipe (perhapse reads some sort of "end of write" token??). I would like to know if there is some sort of "beginning write" token that will be in the buffer if something is being piped in which I can check on STDIN BEFORE I call ReadFile. If this were the case I could just skip calling ReadFile and use some default text.
If there is not a way to do this, I can always pass in a command line argument that denotes that I should not check the pipe and just use the default text (or the other way around), but I would much prefer to do it the way that I specified.
Look at PeekNamedPipe(). Despite its name, it works for both named and anonymous pipes.
int main(int argc, const char *argv[])
{
char char_buffer[BUFSIZE];
DWORD bytes_read;
DWORD bytes_avail;
DWORD dw;
HANDLE stdin_handle;
bool is_pipe;
stdin_handle = GetStdHandle(STD_INPUT_HANDLE);
is_pipe = !GetConsoleMode(stdin_handle, &dw);
if (stdin_handle == INVALID_HANDLE_VALUE) {
std::cout << "Error: invalid handle value!\n\n";
} else {
while (1) {
if (is_pipe) {
if (PeekNamedPipe(stdin_handle, NULL, 0, NULL, &bytes_avail, NULL)) {
if (bytes_avail == 0) {
Sleep(100);
continue;
}
}
}
if (!ReadFile(stdin_handle, char_buffer, min(bytes_avail, BUFSIZE), &bytes_read, NULL)) {
break;
}
if (bytes_read == 0) {
break;
}
// Output what we have read so far
for (unsigned int i = 0; i < bytes_read; i++) {
std::cout << char_buffer[i];
}
}
}
return 0;
}
It looks like what you're really trying to do here is to determine whether you've got console input (where you use default value) vs pipe input (where you use input from the pipe).
Suggest testing that directly instead of trying to check if there's input ready: the catch with trying to sniff whether there's data in the pipe is that if the source app is slow in generating output, your app might make an incorrect assumption just because there isn't input yet available. (It might also be possible that, due to typeahead, there's a user could have typed in characters that area ready to be read from console STDIN before your app gets around to checking if input is available.)
Also, keep in mind that it might be useful to allow your app to be used with file redirection, not just pipes - eg:
myapp.exe < some_input_file
The classic way to do this "interactive mode, vs used with redirected input" test on unix is using isatty(); and luckily there's an equivalent in the Windows CRT - see function _isatty(); or use GetFileType() checking for FILE_TYPE_CHAR on GetStdHandle(STD_INPUT_HANDLE) - or use say GetConsoleMode as Remy does, which will only succeed on a real console handle.
This also works without overlapped I/O while using a second thread, that does the synchronous ReadFile-call. Then the main thread waits an arbitrary amount of time and acts like above...
Hope this helps...
I'd like to read a file line-by-line. I have fgets() working okay, but am not sure what to do if a line is longer than the buffer sizes I've passed to fgets()? And furthermore, since fgets() doesn't seem to be Unicode-aware, and I want to allow UTF-8 files, it might miss line endings and read the whole file, no?
Then I thought I'd use getline(). However, I'm on Mac OS X, and while getline() is specified in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/usr/include/stdio.h, it's not in /usr/include/stdio, so gcc doesn't find it in the shell. And it's not particularly portable, obviously, and I'd like the library I'm developing to be generally useful.
So what's the best way to read a file line-by-line in C?
First of all, it's very unlikely that you need to worry about non-standard line terminators like U+2028. Normal text files are not expected to contain them, and the very overwhelming majority of all existing software that reads normal text files doesn't support them. You mention getline() which is available in glibc but not in MacOS's libc, and it would surprise me if getline() did support such fancy line terminators. It's almost a certainly that you can get away with just supporting LF (U+000A) and maybe also CR+LF (U+000D U+000A). To do that, you don't need to care about UTF-8. That's the beauty of UTF-8's ASCII compatibility and is by design.
As for supporting lines that are longer than the buffer you pass to fgets(), you can do this with a little extra logic around fgets. In pseudocode:
while true {
fgets(buffer, size, stream);
dynamically_allocated_string = strdup(buffer);
while the last char (before the terminating NUL) in the buffer is not '\n' {
concatenate the contents of buffer to the dynamically allocated string
/* the current line is not finished. read more of it */
fgets(buffer, size, stream);
}
process the whole line, as found in the dynamically allocated string
}
But again, I think you will find that there's really quite a lot of software out there that simply doesn't bother with that, from software that parses system config files like /etc/passwd to (some) scripting languages. Depending on your use case, it may very well be good enough to use a "big enough" buffer (e.g. 4096 bytes) and declare that you don't support lines longer than that. You can even call it a security feature (a line length limit is protection against resource exhaustion attacks from a crafted input file).
Based on this answer, here's what I've come up with:
#define LINE_BUF_SIZE 1024
char * getline_from(FILE *fp) {
char * line = malloc(LINE_BUF_SIZE), * linep = line;
size_t lenmax = LINE_BUF_SIZE, len = lenmax;
int c;
if(line == NULL)
return NULL;
for(;;) {
c = fgetc(fp);
if(c == EOF)
break;
if(--len == 0) {
len = lenmax;
char * linen = realloc(linep, lenmax *= 2);
if(linen == NULL) {
// Fail.
free(linep);
return NULL;
}
line = linen + (line - linep);
linep = linen;
}
if((*line++ = c) == '\n')
break;
}
*line = '\0';
return linep;
}
To read stdin:
char *line;
while ( line = getline_from(stdin) ) {
// do stuff
free(line);
}
To read some other file, I first open it with fopen():
FILE *fp;
fp = fopen ( filename, "rb" );
if (!fp) {
fprintf(stderr, "Cannot open %s: ", argv[1]);
perror(NULL);
exit(1);
}
char *line;
while ( line = getline_from(fp) ) {
// do stuff
free(line);
}
This works very nicely for me. I'd love to see an alternative that uses fgets() as suggested by #paul-tomblin, but I don't have the energy to figure it out tonight.
so, I have an assignment for my Operating Systems class wherein i am to create a ring of processes connected with pipes in order to pass messages between them. i found some example code which i was looking to adapt (or at least understand) for my needs. the example code (slightly modified) is:
/* Program 4.1 */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
/* Sample C program for generating a unidirectional ring of processes.Invoke this program
with a command-line arg ument indicating the number of processes on the ring. Communication
is done via pipes that connect the standard output of a process to the standard input of
its successor on the ring. After the ring is created, each process identifies itself with
its process ID and the process ID of its parent. Each process then exits. */
void main(int argc, char *argv[ ])
{
int master_pid = getpid();
printf("master pid: %i\n", master_pid);
int i; /* number of this process (starting with 1) */
int childpid; /* indicates process should spawn another */
int nprocs; /* total number of processes in ring */
int fd[2]; /* file descriptors returned by pipe */
int error; /* return value from dup2 call */
/* check command line for a valid number of processes to generate */
if ( (argc != 2) || ((nprocs = atoi (argv[1])) <= 0) ) {
fprintf (stderr, "Usage: %s nprocs\n", argv[0]);
exit(1);
}
/* connect std input to std output via a pipe */
if (pipe (fd) == -1) {
perror("Could not create pipe");
exit(1);
}
printf("%s\n", "test");
//this section is blocking printf()?
if ((dup2(fd[0], STDIN_FILENO) == -1) ||
(dup2(fd[1], STDOUT_FILENO) == -1)) {
perror("Could not dup pipes");
exit(1);
}
printf("%s\n", "test");
if ((close(fd[0]) == -1) || (close(fd[1]) == -1)) {
perror("Could not close extra descriptors");
exit(1);
}
/* create the remaining processes with their connecting pipes */
for (i = 1; i < nprocs; i++) {
if (pipe (fd) == -1) {
fprintf(stderr,"Could not create pipe %d: %s\n",
i, strerror(errno));
exit(1);
}
if ((childpid = fork()) == -1) {
fprintf(stderr, "Could not create child %d: %s\n",
i, strerror(errno));
exit(1);
}
if (childpid > 0) /* for parent process, reassign stdout */
error = dup2(fd[1], STDOUT_FILENO);
else
error = dup2(fd[0], STDIN_FILENO);
if (error == -1) {
fprintf(stderr, "Could not dup pipes for iteration %d: %s\n",
i, strerror(errno));
exit(1);
}
if ((close(fd[0]) == -1) || (close(fd[1]) == -1)) {
fprintf(stderr, "Could not close extra descriptors %d: %s\n",
i, strerror(errno));
exit(1);
}
if (childpid)
break;
}
/* say hello to the world */
fprintf(stderr,"This is process %d with ID %d and parent id %d\n",
i, (int)getpid(), (int)getppid());
wait(1);
exit (0);
} /* end of main program here */
which outputs:
master pid: 30593
test
This is process 1 with ID 30593 and parent id 30286
This is process 2 with ID 30594 and parent id 30593
when i give is 2 as argv[1]
so, I'm wondering, why would the dup2 section prevent the printf() from executing? if i cant even print something, i'm not sure if i could even pass the message correctly. also, why would the fprintf() already there work, but not one that i would put there?
edit: i would take this to my professor/TA, but theyre both out of town and will be unreachable between now and the deadline...
printf prints to stdout, which is file descriptor 1 (or equivalently STDOUT_FILENO). dup2(3) is duplicating the pipe's file descriptor on top of the current stdout, which has the side effect of closing the current stdout. So, when you try to printf after calling that particular dup2, you're really printing the data into the pipe you just created, which doesn't go to your terminal output.
fprintf(stderr, ...) still works because that prints to stderr, not stdout, and the stderr file descriptor (2, or equivalently STDERR_FILENO) does not change during the program, so it continues to print out to the terminal.
printf() does not send data to path 0, it sends buffered data using stdout. It would seem that when you disrupt path 0 by dup2'ing something to it, you're disrupting stdout in the process.
From the man page on dup2: dup2() makes newfd be the copy of oldfd, closing newfd first if necessary. Thus when you call dup2(fd[0], STDIN_FILENO) you are breaking stdout.
You state that fprintf() is working but printf() is not... what path are you using for fprintf()? If you're using stderr then it makes perfect sense that it would continue to work, since you haven't done anything with that path.
Description:
Obtain output from an executable
Note:
Will not compile, due to fgets() declaration
Question:
What is the best alternative to fgets, as fgets requires char *?
Is there a better alternative?
Illustration:
void Q_analysis (const char *data)
{
string buffer;
size_t found;
found = buffer.find_first_of (*data);
FILE *condorData = _popen ("condor_q", "r");
while (fgets (buffer.c_str(), buffer.max_size(), condorData) != NULL)
{
if (found == string::npos)
{
Sleep(2000);
} else {
break;
}
}
return;
}
You should be using the string.getline function for strings
cppreference
however in your case, you should be using a char[] to read into.
eg
string s;
char buffer[ 4096 ];
fgets(buffer, sizeof( buffer ), condorData);
s.assign( buffer, strlen( buffer ));
or your code:
void Q_analysis( const char *data )
{
char buffer[ 4096 ];
FILE *condorData = _popen ("condor_q", "r");
while( fgets( buffer, sizeof( buffer ), condorData ) != NULL )
{
if( strstr( buffer, data ) == NULL )
{
Sleep(2000);
}
else
{
break;
}
}
}
Instead of declaring you buffer as a string declare it as something like:
char buffer[MY_MAX_SIZE]
call fgets with that, and then build the string from the buffer if you need in that form instead of going the other way.
The reason what you're doing doesn't work is that you're getting a copy of the buffer contents as a c-style string, not a pointer into the gut of the buffer. It is, by design, read only.
-- MarkusQ
You're right that you can't read directly into a std::string because its c_str and data methods both return const pointers. You could read into a std::vector<char> instead.
You could also use the getline function. But it requires an iostream object, not a C FILE pointer. You can get from one to the other, though, in a vendor-specific way. See "A Handy Guide To Handling Handles" for a diagram and some suggestions on how to get from one file type to another. Call fileno on your FILE* to get a numeric file descriptor, and then use fstream::attach to associate it with an fstream object. Then you can use getline.
Try the boost library - I believe it has a function to create an fstream from a FILE*
or you could use fileno() to get a standard C file handle from the FILE, then use fstream::attach to attach a stream to that file. From there you can use getline(), etc. Something like this:
FILE *condorData = _popen ("condor_q", "r");
std::ifstream &stream = new std::ifstream();
stream.attach(_fileno(condorData));
I haven't tested it all too well, but the below appears to do the job:
//! read a line of text from a FILE* to a std::string, returns false on 'no data'
bool stringfgets(FILE* fp, std::string& line)
{
char buffer[1024];
line.clear();
do {
if(!fgets(buffer, sizeof(buffer), fp))
return !line.empty();
line.append(buffer);
} while(!strchr(buffer, '\n'));
return true;
}
Be aware however that this will happily read a 100G line of text, so care must be taken that this is not a DoS-vector from untrusted source files or sockets.