I need to create a new process with redirected standard error stream to some file. The code from which child process is being created has no console available, so there are cases when GetStdHandle(any) will return 0. Child process will try to duplicate all of its standard IO handles for some reason (source code for child process is unavailable) so all of it's handles should be valid.
So I need to run that process in the same manner as it's can be ran from the console with:
someproc <nul >nul 2>err
I see some ways for this:
1. Create two pair of pipes. This is possibly good solution, but it will be too complex for me.
2. Open "nul" file with CreateFile("nul", ...) function call. No file is being created by this call, but this looks weird too me.
3. Use INVALID_HANDLE_VALUE. This works too, but I think there can be different problems with another child processes.
I believe there are better ways.
As originally phrased, you have already answered your own question. To open a "nul" file, you simply specify "nul" when you call CreateFile. It only looks weird because hardly anyone ever uses that file name. (I don't see it used nearly as often as I see /dev/null.) It's perfectly valid, though.
But if you've found that Invalid_Handle_Value works, too, then go ahead and use that instead. It's certainly easiest. I wouldn't have expected it to work, initially, since I wouldn't expect it to be duplicable.
Yes, "nul" is doing what you think. If you move to unix, it will be "/dev/null". The funky name is a holdover from DOS days, along with "prn" and "com1", etc.
Using INVALID_HANDLE_VALUE with DuplicateHandle is improper: the documentation states that you need PROCESS_DUP_HANDLE access right on the handle. You don't have that righton INVALID_HANDLE_VALUE. Device NUL (symbolic link to /device/null) will work fine, though.
Related
Which is better?
cmd >>file 2>&1
cmd 1>>file 2>>file
Is there even a difference?
I know two reasons to choose the first one: It does also work with > instead of >>. It is more popular, therefore someone knowing shell-scripts would except it right away.
But, I still feel like the second one is better readable, and works without having to know the [n]>&[n] syntax, which IMHO is kinda confusing.
What is the difference?
Let's examine what each of these commands means. I will assume that the POSIX shell specification applies since the question doesn't ask about anything more specific.
The first command is cmd >>file 2>&1. This runs cmd after setting up the specified redirections.
The redirection >>file opens the named file with O_APPEND. As explained in the specification of open, this creates a new Open File Description, which notably contains the current file offset, and arranges for File Descriptor 1 to refer to that description. The meaning of O_APPEND is "the file offset shall be set to the end of the file prior to each write".
The redirection 2>&1 says that file descriptor 2 "shall be made to be a copy" of file descriptor 1. That specification is a little vague, but I think the only sensible interpretation (and what shells actually do) is it means to call dup2(1, 2), which "shall cause the file descriptor [2] to refer to the same open file description as the file descriptor [1]". Crucially, we get another file descriptor, but continue to use the same file description, meaning they both have the same file offset.
The second command is cmd 1>>file 2>>file. Based on the specifications cited above, this creates two separate file descriptions for file, each with their own offset.
Now, if the only thing that cmd does to file descriptors 1 and 2 is to call write, then these two situations are equivalent, because every call to write will atomically update the offset to point to the end of the file before performing the write, and therefore the existence of two separate offsets in the second command will not have any observable effect.
However, if cmd performs some other operation, for example lseek, then the two cases are not equivalent because that will reveal that the first command has one shared offset while the second command has two independent offsets.
Additionally, the above assumes the POSIX-specified semantics of O_APPEND. But real computer systems do not always implement that; for example, NFS does not have atomic append. Without atomic append, the second command may behave differently (most likely corrupting the output) even when only write is performed.
Which is better?
As the two commands do not mean the same thing, which is better presumably depends on which meaning is closer to what you intend. I speculate that, in almost all cases, the intent is to append to file both the standard output and standard error from cmd, which is presumed to only write to these descriptors. That is precisely the meaning of the first command (cmd >>file 2>&1), and hence is the better choice.
While the second command does use fewer shell features, and hence might be easier to understand for some people, it would probably seem odd to those who do have greater familiarity with redirection syntax, and might even behave differently than intended in some circumstances. Therefore I would advise against it, and if I found it in some code I was maintaining, would be inclined to change it to the first form.
Of course, if you truly want separate file descriptions, and hence separate file offsets, then the second command makes sense, so long as you put a comment nearby explaining the rationale for the unusual construction.
I am working on making a program that will act in a similar way as a shell, but supports only foreground processes and pipes. I have multiple processes writing to the same pipe and some other properties that differ from the normal usage of pipes. Anyhow, my question is,
Is there any easy (automatic) way to close all file descriptors of a process except the three basic ones?
I am asking this question since I have a lot of difficulties keeping track of all file descriptors for every process. And sometimes they act in some unpredictable ways to me. It could be also because of the fact that I don't have a very thorough understanding of them.
Is there any easy way(automatic) to close all file descriptors of a process except the three basic ones?
The normal way to do this is to simply iterate over all of them and close them:
for (i = getdtablesize(); i > 3;) close(--i);
That's already a one-liner. It doesn't get any more "automatic" than that.
I am asking this question since I have a lot of difficulty keeping track of all file descriptors for every process.
It will be worth your time to think about the life cycle of each file descriptor you open, when it gets duplicated (e.g. dup2() and fork()), how it gets used, and make sure you account for how each one is going to get closed when it is no longer needed. Papering over a problem of leaked file descriptors by indiscriminately closing them all is not going to be sustainable.
I have multiple processes writing to the same pipe
If you do this, then you need to be aware that the order in which data arrive at the other end of the pipe is going to be unpredictable. It will be difficult to avoid corrupting the data stream.
Use the closefrom(3) C library function.
From the manpage:
The closefrom() system call deletes all open file descriptors greater
than or equal to lowfd from the per-process object reference table.
Any errors encountered while closing file descriptors are ignored.
Example usage:
#include <unistd.h>
int main() {
// Close everything except stdin, stdout and stderr
closefrom(3); // Were 3 is the lowest file descriptor you wish to close
printf("Clear of all, but the three basic file descriptors!\n");
return 0;
}
This works in most unices, but requires the libbsd support library for Linux.
This is bugging me for several days now. I know about the standard stream redirection to the NUL device, but this isn't the case. node.js uses CreateFileW under its fs native/libuv bindings.
Unfortunately using something like:
require('fs').writeFileSync('NUL', 'foo')
creates a NUL file into the cwd that has 3 bytes.
I tried writing to the \Device\Null, but since I'm pretty much a *nix head where everything is a file, I failed to actually find a working path for \Device\Null. Such as \\.\Device\Null that throws ENOENT.
Any ideas about how to make this work under Windows?
This seems to be related, but I can not track the whole flow from lib/fs.js to uv/src/win/fs.c to check that the path argument doesn't suffer from some kind of relative to absolute path resolution.
Valid path to NUL device is "\\\\.\\NUL", not NUL, so the usage is: fs.writeFileSync("\\\\.\\NUL", "foo"). This issue was raised against Node.js on GitHub: https://github.com/nodejs/node-v0.x-archive/issues/9271
Since NUL is a device, not a file, it has to be accessed via device namespace - this requires putting \\.\ in the beginning (the other slashes are for escaping) - see https://msdn.microsoft.com/en-gb/library/windows/desktop/aa365247.aspx#Win32_Device_Namespaces.
There is also a simple dev-null library on NPM that can be used with streams: https://www.npmjs.com/package/dev-null (not with .writeFile though).
There is a long workaround, like code it yourself. But you can take an idea from here https://github.com/hanshuebner/node-hid/blob/master/src/HID.cc and write a wrapper over C library.
I have a Win32 application written in C that can have its console output via printf() redirected to a log file.
It would be nice if I could have my app. detect if it had been started with or without a redirect '>'.
Any ideas?
Tom, thanks for the input.
I did some experiments and found this works for me..
fpost_t pos ;
fgetpos (stdout, & pos) ;
When an application's output is being redirected to a file, fgetpos() sets 'pos' to zero. It makes sense since its freshly opened stderr for you. EDIT: Actually, the value returned may be a positive integer if text has already been redirected to the log/file. So in your code you'd have something like "if (pos >= 0) bfRedirected = TRUE ;"
When an application's output is not being redirected - it's going to the console device - not a file, so fgetpos() will set 'pos' to -1.
I think that pipes are blind by nature, so you don't have a built-in way to tell whether you're being redirected. Moreover, trying to find out what's happening behind an abstraction layer is a bad habit.
If you really need to control your output, add the log file name as a command line parameter!
That being said, you can make some smart guesswork to find out:
A program can query the shell command history to find out the most recent commands executed.
If you know the path to the logfiles, you can scan that directory and see if a file has been created or changed its size.
Benchmark writing speed when redirected and not redirected. This would work only if your system is ultra-stable, and environment condition won't change.
There may be a way to do this - a quick google yielded this hit that might give you the hint in the right direction.
Method from Microsoft: WriteConsole fails if it is used with a standard handle that is redirected to a file. If an application processes multilingual output that can be redirected, determine whether the output handle is a console handle (one method is to call the GetConsoleMode function and check whether it succeeds).
AFAIK the answer to this is that you can't. Output redirection works by simply reading the stream of output from a given program and redirecting it to another pipe / stream. The design of file / streams is such that the writer is ignorant of the reader to the point that you shouldn't know you are being read.
Even detecting that there was a reader would be of no use because there is one in the normal case. The console is reading the output of your program and displaying it to the screen already.
I was hacking away the source code for plink to make it compatible with unison.
If you don't know, unison is a file synchronization tool, it runs an "ssh" command to connect to a remote server, but there's no ssh.exe for windows; there's plink, which is very close but not close enough (it doesn't behave like unison expects it to), so people usually make wrappers around it, like this one.
one of the problems is that unison expects the password prompt to print to stderr (but plink prints it to stdout, and causes unison to be confused), so I thought, well, should be simple enough, hack my thru plink's code and make it print the prompt to stdout. so I hacked my way through and did that.
Next problem: I can't respond to the prompt!! no matter what I type, it has no effect.
the code for getting input is roughly like this:
hin = GetStdHandle(STD_INPUT_HANDLE);
....
r = ReadFile(hin, .....);
I'm not sure why it's done this way, but I'm not an expert in designing command line tools for windows, so what do I know! But I figure something is missing in setting up the input handle.
I looked at the source code for the above wrapper tool and I see this:
hconin=CreateFile("CONIN$",GENERIC_READ|GENERIC_WRITE,FILE_SHARE_READ,0,OPEN_EXISTING,0,0)
and I try it (just for the heck of it)
hin=CreateFile("CONIN$",GENERIC_READ|GENERIC_WRITE,FILE_SHARE_READ,0,OPEN_EXISTING,0,0);
....
r = ReadFile( hin ...... )
and surprisingly it works! I can now respond to the prompt!
Why is this? what is "CONIN$"? and why is it different from the STD_INPUT_HANDLE?
I can sort of "guess" that FILE_SHARE_READ and OPEN_EXISTING are playing a role in this (since ssh is being run from within another process), but I want to understand what's going on here, and make sure that this code doesn't have some unwanted side effects or security holes or something scary like that!
CONIN$ is the console input device. Normally, stdin is an open file handle to this, but if stdin is redirected for some reason, then using CONIN$ will allow you to get access to the console despite the redirection. Reference.