Simple question here (though perhaps not such a simple answer):
Is it possible to specify a path for an (existing) named pipe that can be used by programs as if they were opening on a normal file?
According to this MSDN page, name pipes on the local computer can be referrenced using the following path syntax: \\.\pipe\PipeName, yet I'm having no luck using this from standard Windows programs.
As a side point, if anyone has any suggestions for interfacing with programs that are only capable of using the file-system in a more efficient manner than physical I/O (e.g. named pipes), I would be glad to take them.
It would only work if the programs are using the Win32 API CreateFile() function to open the files.
Related
Hi according to this post, unbuffer connects to a command via a pseudo-terminal (pty), which makes the system treat it as an interactive process, therefore not using any stdout buffering.
I would like to use this function on Windows. May I know what is the equivalent of unbuffer program on Windows? Thanks.
I spent some time on this and succeeded. I found this blog during research, and decided to return and provide my solution to save the next guy some time. I'm responding as a guest with a false email so I won't be interacting, but no further information should be required.
On Jul 18 '12 at 19:41 Harry Johnston wrote:
"In principle, if you know how much data to expect, you could use the console API functions to create a console for the application to write to, and then read the output from the console. But you can't do that from Java, you would need to write a C application to do it for you."
Thing is, there is already a utility that does this. It's written for a slightly different use, but it can be coxed into providing the desired result. Its intended purpose is to enable a windows console app to interact with a Linux style tty terminal. It does this by running a hidden console and accesses the console buffer directly. If you tried to use it – you'd fail. I got lucky and discovered that there are undocumented switches for this utility which will allow it to provide simple unbuffered output. Without the switches it fails with the error – the output is not a tty – when trying to pipe output.
The utility is called winpty. You can get it here:
https://github.com/rprichard/winpty/releases
The undocumented switches are mentioned here:
https://github.com/rprichard/winpty/issues/103
I’m using the MSYS2 version. You’ll need the msys-2.0.dll to use it.
Simply run:
winpty.exe -Xallow-non-tty -Xplain your_program.exe | receive_unbuffered_output.exe
-Xallow-non-tty , will allow piped output
-Xplain , will remove the added Linux terminal escape codes (or whatever they’re called)
Required files are:
winpty.exe
winpty-agent.exe
winpty.dll
msys-2.0.dll
winpty-debugserver.exe – Not needed
The behaviour you're describing is typical of applications using run-time libraries for I/O. By default, most runtime libraries check to see whether the handle is a character mode device such as a console, and if so, they don't do any buffering. (Ideally the run-time library would treat a pipe in the same way as a console, but it seems that most don't.)
I'm not aware of any sensible way to trick such an application into thinking it is writing to a console when it is actually writing to a pipe.
Addendum: seven years later, Windows finally supports pseudoconsoles. If you are running on Windows 10 v1809 or later, this new API should solve your problem.
On older versions of Windows, if you know how much data to expect, you could in principle use the console API functions to create a console for the application to write to, and then read the output from the console. But you can't do that from Java, you would need to write a C application to do it for you.
Similarly, in principle it should presumably be possible to write a device driver equivalent to a Unix pseudo-terminal, one that acts like a pipe but reports itself to be a character-mode device. But writing device drivers requires specific expertise, and they have to be digitally signed, so unless there is an existing product out there this approach isn't likely to be feasible.
Disclaimer: My answer only deals with executables compiled using MSVC.
The buffering policy is coded inside Microsoft C Runtime (CRT) Library. You can learn the details here. This article suggests using console handles and manipulate console buffers to receive unbuffered output.
However, there's an undocumented feature inside Microsoft C Runtime to inherit file handles with some internal flags directly from its parent process using lpReserved2 and cbReserved2 fields of STARTUPINFO structure. You can find the details in the crt source code provided by Microsoft Visual Studio. Or search for something like posfhnd on GitHub.
We can exploit this undocumented feature to provide a pipe handle and specify FOPEN | FDEV flags to the child process, to fool the child process treat that pipe handle the same way as a FILE_TYPE_CHAR handle.
I have a working Python3 script to demonstrate this method.
I'm fairly new to C++ and I want to make a Program Launcher, All it does is launch applications for me based on text I input.
I all ready have the basic code for it, but I can't seem to find out how to change the working directory. I know you use chdir, but how Exactly.
C++ still has very little standardized support for filesystem functions, in part because it runs on embedded devices which may not have working directories, or sometimes even directories or files at all. So we have to look to the OS API.
For POSIX, there is chdir() and getcwd().
For Windows, there is SetCurrentDirectory and GetCurrentDirectory, although if you want to deal with per-drive working directory, you will need to work with environment variables.
There is an example on MSDN titled Changing the Current Directory
Is it possible to create a new, arbitrary, file namespace scheme in Windows?
As best I understand, Windows currently understands two or three file system or file-system-like namespace schemes:
The namespace scheme we all know and love, eg, C:\path\to\file.
UNC paths, eg, \\server\path\to\file
One, perhaps uncommon scheme - the Windows NT Object Manager, eg, \\.\Device\COM1 - see WinObj on SysInternals, usually accessed by programs by calling CreateFile, though this is not really a file system.
Is it possible to implement a custom namespace scheme that would be universally, automatically used by the rest of the operating system? Perhaps a filter driver or some other specialized kernel-mode driver? I'm out of my league here, but I'm genuinely curious.
I don't have anything concrete, but lets say I wanted to implement a kernel driver that, not only understands how to read and write OpenVMS file systems, but also implements some sort of filter driver so that userland programs could use standard File-11 syntax to access such a filesystem.
For example, an existing program calls OpenFile("[DIR1.DIR2.DIR3]FILE.EXT;10"); and somehow a custom handler deals with it transparently, and lo, notepad can read and write VMS files. More importantly, perhaps, some ported program that expects OpenVMS File-11 path strings just works. Simply mapping the OpenVMS file system into the regular windows file system as D:\dir1\dir2\file.ext would be insufficient.
I should clarify that my OpenVMS reference is just an example; I'd be looking for a more generic solution. This could be for OpenVMS File-11, MVS, standard unix syntax ala /path/to/thing, or something I just cooked up myself.
I'm aware of shell-based namespace extensions, and compatibility layers like cygwin, but that's not what I'm looking for.
So SO, what do you think? Is this possible? Where do you start?
What are the differences, and in what cases one or the other would prove superior in some way?
First of all the function fopen can be used only for simple portable operations with files.
CreateFile on the other side can be used not only for operations with files, but also with directories (with use of corresponding options), pipes and various Windows devices.
CreateFile has a lot of additional useful switches, like FILE_FLAG_NO_BUFFERING, FILE_ATTRIBUTE_TEMPORARY and FILE_FLAG_SEQUENTIAL_SCAN, which can be very useful in different scenarios.
You can use CreateFile with a filename longer that MAX_PATH characters. It can be important for some server applications or ones which must be able to open any file (a virus scanner or a backup application for example). This is enabled by using namespace semantics, though this mode has its own concerns, like ability to actually create a file named ".." or L"\xfeff\x20\xd9ab" (good luck trying to delete them later).
You can use CreateFile in different security scenarios. I mean not only usage of security attributes. If current process has SE_BACKUP_NAME or SE_RESTORE_NAME privilege (like Administrators typically have) and enable this privilege, one can use CreateFile to open any file also a file to which you have no access through security descriptor.
If you only want to read the content of a file, you can use CreateFile, CreateFileMapping and MapViewOfFile to create file mapping. Then you can work with a file as with a block of memory, which can possibly increase your application's speed.
There are also other uses of the function, which are described in detail in the corresponding MSDN article.
So I can summarize: only if you have a hard portability requirements or if you need to pass a FILE* to some external library, then you have to use fopen. In all other cases I would recommend you to use CreateFile.
For best results, I would also advise to learn Windows API specifically, as there are many features that you can find a good use for.
UPDATED: Not directly related to your question, but I also recommend you to take a glance at transactional I/O functions which are supported starting with Windows Vista. Using this feature, you can commit a bunch of operation with files, directories or registry as one transaction that cannot be interrupted. It is a very powerful and interesting tool. If you are not ready now to use the transactional I/O functions, you can start with CreateFile and port your application to transactional I/O later.
That really depends on what type of program you are writing. If it is supposed to be portable, fopen will make your life easier. fopen will call CreateFile "behind the scenes".
Some more advanced options (cache control, file access control, etc) are only available if you are using the Win32 API (they depend on the Win32 file handle, as opposed to the FILE pointer in stdio), so if you are writing a pure Win32 application, you may want to use CreateFile.
CreateFile lets you
Open file for asynchronous I/O
Pass optimization hints like FILE_FLAG_SEQUENTIAL_SCAN
Set security and inherit settings without threading issues
They don't return the same handle type, with fopen/FILE object you can call other runtime functions such as fputs (as well as converting it to a "native" file handle)
Whenever possible, prefer object oriented wrappers that support RAII, like fstream or boost file IO objects.
You should, of course, care about the share mode, so fopen() and STL are insufficient.
Is it possible to hook into Windows loading or saving files (no matter how the file is opened like notepad word etc.) to modify the file on the fly?
For example to encode/decode it on the fly?
Would code need administrative permissions to launch?
You probably will have to write a driver. See if you can get a hold of Filemon's source, there is a lot to learn there.
You could also use something like madCodeHook to intercept file read/writes and to install your dll into every process. I've used this technique to record print jobs for billing purposes.
Yes, you need to write an Installable File System driver. The Installable File System Kit from Microsoft contains a couple of sample drivers, including the one used by Filemon. Unfortunately, I do not believe you can access those API's without the IFS Kit.
avoid madCodeHook (not profesdional)
use standard api hooking mechanisms (Richter and Microsoft D mainly)