As I understand, on Windows, the process that creates a named pipe, owns the "server" handle (one side of the pipe).
Is it possible to split pipe creation from actually using it (opening it for reading and writing)?
I need to create a named pipe with one process, but use two other processes to open it for reading and writing.
But I'm concerned, that when the process that creates the pipe exits, the pipe gets removed from the file system.
Named pipes exist solely because of the sort of scenario you're talking about.
With anonymous pipes (see CreatePipe), it is necessary that there be a "factory" somewhere that creates the pipe, and then at least one of the handles is handed off to a "client" somewhere (within the same process or not).
With named pipes (see CreateNamedPipe), it is not necessary to have the "factory" which creates and distributes the handles. Instead, the "server" side of the named pipe directly does a CreateNamedPipe with the correct pipe name, and the "client" side of the named pipe directly uses CreateFile with the pipe name to open the client side of the named pipe. Each side deals with its error conditions appropriately. If implemented correctly, no additional coordination is necessary.
If your application requires a "factory" process for the named pipe (it shouldn't), then you'll have to figure out your own way to provide the handles to the "client" processes in such a way that the pipe is not destroyed in the interim. Or you can use the pipe name instead of distributing the handles, but you may have to explicitly coordinate between the processes using something like a named event (see CreateEvent).
If this doesn't answer your question, please update your question with more details.
Related
On Unix-like operating systems, the file descriptors (file handles) used for things like pipes and sockets are small integers. It's common for a parent process to communicate with its child process by opening a pipe, socket or file at a particular file descriptor number. The number will either be agreed upon by the programmers of the parent and child program, or passed to the child program via the command line or environment variables. The child can then access the inherited file descriptor using the same number as the parent.
Is there a convention for passing handle numbers in a similar way on Windows? As far as I can tell, the relevant MSDN articles only talk about such conventions for the "standard" file handles (standard input, standard output, standard error). What would be a robust and idiomatic way to do something like "pass a pipe on file descriptor 3" as we sometimes do on Unix?
The MSDN article on Inheritance says (emphasis mine):
An inherited handle refers to the same object in the child process as it does in the parent process. It also has the same value and access privileges.
Does this mean that I can just cast the HANDLE value to an integer, convert that integer into a string and pass that string as a command line argument for the child process to parse and get the same handle back, and is this reliable or customary? Is there some other IPC mechanism that Windows programmers would normally use instead?
The main motivation for me would be to avoid temporary files and use pipes instead. This is conventional when all you have is stdin/stdout but less so when you have multiple pipes.
Firstly, is it possible to write to a named pipe from two processes at once, using the same handle (or one duplicated using DuplicateHandle)?
And secondly, would it make any difference if one process performs asynchronous IO on the pipe, while the other performs synchronous IO?
From what I've tried, the process that reads from the pipe crashes.
I need to send data from child processes to parent. Some of this data is HTML, plain text, etc. but it may also be necessary send image data, zip file data, etc.
As I understand it, anonymous pipes use the child process standard input and standard output. Conventionally stdin and stdout only convey textual data: would there be any problem with sending non printable characters using this mechanism?
There is no relation between anonymous pipes and stdin/out. As one process has only one stdin/out, you could create only one anonymous pipe per process that way, which sounds stupid, doesn't it? You can redirect stdin/out of a child process to the pipe, yes. But you don't have to, if the child process is able to report itself by another means (like logfile or network activity). A call to CreatePipe gives you reading and writing handles and it's up to you how you use them. Sending arbitrary binary data is indeed possible. Anonymous pipe is in no way different from named pipe in that respect.
Even if you do choose to use stdin/stdout redirection to pass the pipe handle(s) to the child process, you shouldn't have any problems provided the child process uses the Windows API to send the data rather than the C runtime library functions.
That is, WriteFile will work perfectly, but printf would not be a good idea.
You can use GetStdHandle to get the handle(s) to the pipe(s) for use with the Windows API functions.
If I'm writing a simple text log file from multiple processes, can they overwrite/corrupt each other's entries?
(Basically, this question Is file append atomic in UNIX? but for Windows/NTFS.)
You can get atomic append on local files. Open the file with FILE_APPEND_DATA access (Documented in WDK). When you omit FILE_WRITE_DATA access then all writes will ignore the the current file pointer and be done at the end-of file. Or you may use FILE_WRITE_DATA access and for append writes specify it in overlapped structure (Offset = FILE_WRITE_TO_END_OF_FILE and OffsetHigh = -1 Documented in WDK).
The append behavior is properly synchronized between writes via different handles. I use that regularly for logging by multiple processes. I do write BOM at every open to offset 0 and all other writes are appended. The timestamps are not a problem, they can be sorted when needed.
Even if append is atomic (which I don't believe it is), it may not give you the results you want. For example, assuming a log includes a timestamp, it seems reasonable to expect more recent logs to be appended after older logs. With concurrency, this guarantee doesn't hold - if multiple processes are waiting to write to the same file, any one of them might get the write lock - not just the oldest one waiting. Thus, logs can be written out of sequence.
If this is not desirable behaviour, you can avoid it by publishing logs entries from all processes to a shared queue, such as a named pipe. You then have a single process that writes from this queue to the log file. This avoids the conccurrency issues, ensures that logs are written in order, and works when file appends are not atomic, since the file is only written to directly by one process.
From this MSDN page on creating and opening Files:
An application also uses CreateFile to specify whether it wants to share the file for reading, writing, both, or neither. This is known as the sharing mode. An open file that is not shared (dwShareMode set to zero) cannot be opened again, either by the application that opened it or by another application, until its handle has been closed. This is also referred to as exclusive access.
and:
If you specify an access or sharing mode that conflicts with the modes specified in the previous call, CreateFile fails.
So if you use CreateFile rather than say File.Open which doesn't have the same level of control over the file access, you should be able to open a file in such a way that it can't get corrupted by other processes.
You'll obviously have to add code to your processes to cope with the case where they can't get exclusive access to the log file.
No it isn't. If you need this there is Transactional NTFS in Windows Vista/7.
I have some basic questions about pipes I am unsure about.
a) What is the standard behavior if a process writing to a pipe gets killed (ie. SIGKILL SIGINT) Does it close the pipe? Does it flush the pipe? Or is the behavior undefined?
b) What is the standard behavior if a process returns normally? Is it guaranteed to flush the pipe and close the pipe? (without explicitly doing so of course).
I would like these answers to be as general as possible, but in reality if it depends entirely on the OS specs I can accept that! However, if there is a Posix standard or a current defined Windows behavior I would be very grateful to know.
Thanks.
a. What is the standard behavior if a process writing to a pipe gets killed (ie. SIGKILL SIGINT) Does it close the pipe? Does it flush the pipe? Or is the behavior undefined?
SIGKILL never allows any cleanup - the process dies, dead. With SIGINT, it depends on whether the process handles the signal. If so, it is likely to exit via exit(2), which flushes standard I/O file handles. The question is - was the pipe connected to standard output or via popen()? If so, outstanding buffered data may be flushed; if not, there is no buffered data so flushing is immaterial.
If there is unread data in the pipe, that data remains in the pipe, ready for the reader to collect - assuming there is a reader.
b. What is the standard behavior if a process returns normally? Is it guaranteed to flush the pipe and close the pipe? (without explicitly doing so of course).
It depends on whether the pipe was connected via standard I/O or not. If not, there is nothing pending. If so, then yes, any material in the buffers will be flushed as the standard I/O stream is closed.
c. Thanks for the info on signals and the unread data, but I'm a little confused about the standard I/O pipe connection. After you mentioned popen() I looked it up and the man page says its return value identical to an I/O stream and the streams are fully buffered by default. I'm just not clear on the difference between the two nor do I understand where the difference comes from.
The basic system call for creating pipes is pipe(2). It creates two file descriptors, one for the read end of the pipe, one for the write end. If you do nothing else with them, then they remain as file descriptors, with unbuffered output (via write(2) and related system calls). If the process terminates, there is no buffering in the application; the pipe is closed.
If you use popen(3), then it does a whole lot more work for you. It still invokes pipe(2) to create the pipes, but it then does a fork(2). The child arranges the correct configuration of the pipes and launches the child process. The parent also closes the unused end of the pipe, and uses fdopen(3) to create a standard I/O file stream for the calling process to use.
With the file stream, if there is data in the I/O buffer, then a close or equivalent will ensure that the outstanding data is flushed and the file descriptor is closed.
The normal behaviour is that all file descriptors are closed when a process terminates. This means that a pipe, like any other open file descriptor, is closed normally.
One interesting thing about pipes, though: in POSIX, if a process writes to a pipe that has been closed, the writer will get a signal, SIGPIPE.
Edit:
A caveat: The difference between s SIGx termination and a normal termination is that, like any other file write, you may loose data that has been buffered (via a FILE write) and not yet written to the file descriptor.