I want to put some sort of "hook" into windows (only has to work on Windows Server 2008 R2 and above) which when I ask for a file on disk and it's not there it then requests it from a web server and caches it locally.
The files are immutable and have unique file names.
The application which is trying to open these files is written in C and just opens a file using the operating system in the normal way. Say it calls OpenFile asking for c:\scripts\1234.12.script, and that is there then it will just open it normally. If then it asks for c:\scripts\1234.13.script and it isn't then my hook in the operating system will then go and ask my web service for the file, download it and then return that file as it it were there all the time.
I'd prefer to write this as a usermode process (I've never written a windows driver), it should only fire when files are not found in a specific folder, and I'd prefer if possible to write it in a managed language (C# would be perfect). The files are small (< 50kB) and the web service is fast and the internet connection blinding so I'm not expecting it to take more than a second to download the file.
My question is - where do I start looking for information about this kind of thing? And if anyone has done anything similar - do you know what options I have (eg can it be done in C#?)?
You would need to create a kernel-mode filesystem filter driver which would intercept requests for opening such files and would "fake" those files. I should say that this is a very complicated task even for driver development. Our CallbackFilter product would be able to solve your problem however mechanism for "faking" files is not yet ready (we plan this feature for CallbackFilter 3). Until then I don't know any user-mode solutions (frankly speaking, no kernel-mode solutions as well) that would solve your problem.
If you can change the folder the application is accessing, then you can create a virtual file system and map it to the drive letter or a folder on NTFS drive. From the virtual file system you can direct most requests to/from real disk and if the file doesn't exist, you can download the file and cache it. Our other product, Callback File System, lets you do what I described in user-mode. If you have a one-time task you need to accomplish, and don't have a budget for it, please contact us anyway and maybe we can find some solution. There also exists an open-source solution with similar (but not so comprehensive) functionality named Dokan, yet I will refrain from commenting on its quality.
You can also try Dokan , it open source and you can check its discussion group for question and guides.
Related
I just learned by accident that since Windows XP there is a special system directory C:/Windows/Prefetch which contains "prefetch files" used to speed up system boot and application start-up performance. I discovered them by searching my disk with the name of a program of mine and saw a .pf file in that directory.
Naturally, I began to wonder if it is possible for me, as the creator of an application, to pre-create these prefetch files myself. The intention is that the users of my application could benefit from better startup performance since the very first launch of my application, rather than the second one, as I understood that the prefetch file is generated only when needed.
Is it possible to create these special files ahead of time, package them with an app, and then install them, or do they have to be created on the end user's machine?
I have a big collection of folders for projects I'm working on. I've been trying to find a better way to sort them all for a long time and I want to write an app that creates groups based on whatever criteria I say, such as "folders from 2011" or "folders containing a x type of file" etc.
This is fairly straightforward, and wouldn't present much of a problem to code using its own UI in winForms or WPF or something. But I think it would be far better if I could make these folders appear to be part of the filesystem, so other apps (like existing file explorers) can see them.
Is this possible? Would it cause problems I haven't considered? How do I go about doing it if it is possible?
One way I thought of doing it would be to have the app monitor the filesystem and create folder shortcuts every time there's a change, but I'm curious about whether its possible to actually present a fake filesystem to explorer through a 'gateway' folder
EDIT: Ok it's obviously possible since http://www.virtualfolder.net/ can do it, and now that I think of it so can TrueCrypt, although it would be nice if it didn't have to appear as a separate drive. So the question becomes, how do I implement it?
You can create a Shell Namespace Extension that gathers the file information you want and displays it within Windows Explorer any way you wish. You can choose where your extension is located, whether as its own top-level node, a child of another system virtual folder/extension, or as a child of a file system folder.
Writing a SNE is not trivial, but it is a lot easier then writing a lower-level file system driver, and it does not require special driver-oriented compilers. Any compiler that supports developing COM objects will work.
This is accomplished using filesystem drivers or filesystem filter drivers. First let you create a virtual filesystem and mount it to a drive letter and also to a folder on NTFS drive (folder must exist but its contents are "replaced" with a virtual filesystem directory tree). Filesystem filter drivers let you introduce virtual files and folders in existing folders without replacing them.
VirtualFolder uses filesystem driver as it creates a drive letter.
Both types of drivers are written in C and work in kernel-mode. Writing them requires deep knowledge of Windows internals and experience with driver development (since filesystem drivers are one of the most complicated driver types).
We offer several products related to virtual storage. One of them, Callback File System, is a filesystem driver. It calls your user-mode code to perform actual filesystem functions. Another product, CallbackFilter, is an FS filter driver (and it also calls your user-mode code). However, current version of CallbackFilter doesn't let you introduce virtual files and folders (this would be implemented in the next release).
There's also Pismo File Mount product available, they use filter driver techniques. You can check with them if what you need can be accomplished.
From what I gather you are looking for a way to present the results of predefined file queries to appear as though they are located at a specific location in the file system. If that is correct you may want to look into Hard Links and Junctions. There are limits on what you can do with these file system services. However it is really straight forward to implement.
I am trying to figure out when and how does Windows update File Access Times on files.
First of all, most Windows installs come with File Access Times disabled for performance reasons, so before wrapping your head around it here is what you need to do in order to activate last access times on NTFS file systems: modify the key [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem] value name NtfsDisableLastAccessUpdate to DWORD 0 value data(if it is set to 1 of course). If it doesn't exist just create it.
Upon reading File Times article on MSDN i am still in doubt as to how Windows updates access times.
My questions are as follow:
Do access times update upon issuing a WinApi CreateFile() with FILE_READ_ATTRIBUTES ? In my case, while doing it programmatically, it doesn't. Opening up the File Properties dialog of that file through the Explorer Shell does update the access time.
Do access times update upon issuing a WinApi ExtractIconEx() to read an icon from a file?
In my case doing so programatically, it doesn't. Opening up the File Properties dialog of that file through the Explorer Shell does update the access time.
If you ask me, both of those cases should update the file access times, but it seems to me that direct WinApi calls don't update them or Window/NTFS driver really lags behind, while operating on files from Windows Explorer do update pretty well. What do you think is or could be the issue here?
As a side note, i did do CloseHandle() as per:
The only guarantee about a file
timestamp is that the file time is
correctly reflected when the handle
that makes the change is closed.
My end conclusion is that, indeed the opinions lying around the web are true and Windows does update File Access Times in a random fashion and thus one really shouldn't in no way depend on Windows File Access Times.
Off-topic rant: Sorry forensics guys, you'll have to prove access times using another method or you can have your case invalided in seconds. :P
No, accessing the metadata of the file isn't going to change the last access time (name, attributes, timestamps). Wouldn't work well in practice, just looking at the directory with Explorer would change it. You have to actually open the file. ExtractIconEx() would normally be an excellent candidate, except that Windows can play tricks with it. A hidden desktop.ini file can redirect the icon to another file.
Using the last access time is pretty worthless for forensics. You'd need a file system filter driver. Similar to the one embedded in SysInternals' ProcMon utility. It might be using ETW btw, that got pretty powerful at Vista time. Nevertheless, your project just got 10 times more complicated.
For testing and development purposes, it would be nice to somehow simulate (spurious) file access errors to local files. For example, even if an application has correctly opened a file with the appropriate restrictive sharing flags, it still can happen that an attempt to access the file (through any of the Win32 API functions or your favourite framework, which internally will just call any of the Win32 API functions) can fail.
The only example I ever was able to track down was the virus scanner on a machine, but I guess there could be other reasons. (In this question's comment, Luke mentions something about "File system filter drivers".)
FWIW, I know of a few possibilities to "simulate" file problems, that I do not consider good solutions, either because they require to much manual work or because they don't fit for every app/file:
Place a file on a network drive or removable storage device - that way you can just mess up the device (unplug, disk-full, ...).
Open the application process in Process Explorer and close the handle of the file you want to test.
So the question really is if there are any ((semi)automated) tools that can mess up file access (on an NTFS drive) even though an application has already opened a file with appropriate (for the app) sharing flags.
Holodeck purports to allow Win32 API hooking, which would enable you to manipulate return codes as needed for Fault Injection.
If your API set of interest is well-defined, you could probably do this yourself using the Import Address Table approach described here.
I have been trying to find a way to "defragment" the registry on my Windows machine. Firstly, does this make sense? Any benefits in doing this? (Not much love on superuser.com) Secondly, I am looking for a way to rewrite the registry using C/C++ with Windows API. Is there a way to read the registry and write it to a new file getting rid of unused bytes along the way? (I might have to write the new file and then boot into another OS/disk before I can overwrite the original... but I am willing to take that risk.)
Microsoft's PageDefrag does exactly this, as it states on its page "PageDefrag uses the standard file defragmentation APIs to defragment the files."
(A copy of the linked article is here because in typical MSDN style their link is dead.)
http://www.larshederer.homepage.t-online.de/erunt/ - NTREGOPT NT Registry Optimizer
Similar to Windows 9x/Me, the registry files in an NT-based system
can become fragmented over time, occupying more space on your hard
disk than necessary and decreasing overall performance. You should
use the NTREGOPT utility regularly, but especially after installing
or uninstalling a program, to minimize the size of the registry files
and optimize registry access.
The program works by recreating each registry hive "from scratch",
thus removing any slack space that may be left from previously
modified or deleted keys.
http://reboot.pro/index.php?showtopic=11212 - Offreg.dll MS WDK Offline Registry Library
The offline registry library (Offreg.dll) is used to modify a registry hive outside the active system registry. This library is intended for registry update scenarios such as servicing an operating system image. The library supports registry hive formats starting with Windows XP.
Developer Audience
http://reboot.pro/topic/11312-offline-registry/ - Offline Registry MS WDK Command-Line Tool
A command line tool that will allow one to read and write to an offline registry hive.
Reading the values should be possible.
But I've never seen any spec for how the registry files are written to disk, and unless you could find one you'd have to reverse engineer those files in your OS (might be differences between XP and 7 etc). Then you have to remember that the registry isn't just one file, it's multiple files and some of them belongs to certain users and I think they use SIDs rather than user names so even if you move them to a new computer, you have to be sure it's the same OS version with the same users with the same SIDs set up on it.
All this for little or no gain so I'd agree with the superuser users that it wouldn't make sense.