I have an encrypted 7z file which has some binary data files.
I am on windows platform (Windows 10 - Customers are also on same platform).
I am using C++ 17.
I am using a third party library (C standard) that can load these binary files only from disk (not from memory streams) into memory.
Loading the bin files from disk to memory takes a few milliseconds.
I don't want to let the users of my software to be able to read the content of the binary files.
I can't use a online service to host these bin files because the customers should be able to use the software on a standalone computer without any network connectivity.
The way I am planning it now is as follows:
Choose a random folder path at runtime (in the windows temp folder)
Extract the encrypted 7z file to the above random path.
Immediately acquire a exclusive lock on the bin files using https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-lockfileex
Read the bin files
After the reading is done, overwrite the files with zeros
Delete the extracted files
Things that can go wrong in the above approach:
Customers having admin privileges can potentially perfectly time the bin file loading in software and kill the process before the files are overwritten.
Customers can potentially get a memory dump and read the data directly (not sure if this is so easy to pull off)
Potentially more ways..
Is there any better way to deal with the situation?
Should I live with the potential of IP loss.
Related
Files
I learned from school that operating system deal with directory as file and it contains elements like list of files in the directory and extension of its
so how could I open and edit that file in windows for example?
The operating system does not know exactly how files and directories are stored, only the driver/code for a specific filesystem (ntfs, fat, ext4, btrfs, zfs etc.) knows that.
When you create and delete files like normal you are essentially modifying the "directory file". There is no generic way to edit a directory directly, you have to use normal directory and file functions provided by the OS.
Technically it might be possible to get raw access to the disk and play with the raw bytes but if you do this on a live system you are most likely just going to corrupt it.
I want to put some sort of "hook" into windows (only has to work on Windows Server 2008 R2 and above) which when I ask for a file on disk and it's not there it then requests it from a web server and caches it locally.
The files are immutable and have unique file names.
The application which is trying to open these files is written in C and just opens a file using the operating system in the normal way. Say it calls OpenFile asking for c:\scripts\1234.12.script, and that is there then it will just open it normally. If then it asks for c:\scripts\1234.13.script and it isn't then my hook in the operating system will then go and ask my web service for the file, download it and then return that file as it it were there all the time.
I'd prefer to write this as a usermode process (I've never written a windows driver), it should only fire when files are not found in a specific folder, and I'd prefer if possible to write it in a managed language (C# would be perfect). The files are small (< 50kB) and the web service is fast and the internet connection blinding so I'm not expecting it to take more than a second to download the file.
My question is - where do I start looking for information about this kind of thing? And if anyone has done anything similar - do you know what options I have (eg can it be done in C#?)?
You would need to create a kernel-mode filesystem filter driver which would intercept requests for opening such files and would "fake" those files. I should say that this is a very complicated task even for driver development. Our CallbackFilter product would be able to solve your problem however mechanism for "faking" files is not yet ready (we plan this feature for CallbackFilter 3). Until then I don't know any user-mode solutions (frankly speaking, no kernel-mode solutions as well) that would solve your problem.
If you can change the folder the application is accessing, then you can create a virtual file system and map it to the drive letter or a folder on NTFS drive. From the virtual file system you can direct most requests to/from real disk and if the file doesn't exist, you can download the file and cache it. Our other product, Callback File System, lets you do what I described in user-mode. If you have a one-time task you need to accomplish, and don't have a budget for it, please contact us anyway and maybe we can find some solution. There also exists an open-source solution with similar (but not so comprehensive) functionality named Dokan, yet I will refrain from commenting on its quality.
You can also try Dokan , it open source and you can check its discussion group for question and guides.
Update:
Ehh -- Even though this question isn't "answered", I've just emptied my pockets and purchased an SSD. My ramdisk software was going to cost just about as much anyway. I'm not particularly interested in an answer here anymore, so I'll just mark this as "answered" and go on with my life.
Thanks for the help.
I've got a program which is writing files to a ramdisk (in Windows XP) and I need to copy its data from the ramdisk to a directory on my harddrive once it has finished execution. Obviously in a ramdisk the space is limited and I need to free up as much space on the ramdisk as I can between runs. The simple solution is to copy the data folder that my program generates on the ramdisk to a location on the harddisk and recursively delete the "data" folder from the ramdisk.
There is a problem with that solution however; my program looks at the filesystem and filenames to ensure that it doesn't overwrite files (The most recent data file in the directory is 006.dat, so it will write 007.dat instead of overwriting anything). I can't just delete the files once I'm done writing data because it needs that filesystem intact to record data without over-writing the old files when I copy the data back to my hard-drive
I'd like a simple little windows batch script which I can execute after my program has finished writing data files to the ramdisk. This batch script should copy the ramdisk "data" folder to my harddisk and delete all the files from the ramdisk, then it should re-create the filesystem as it was but with all zero-byte files.
How would I go about this?
Could you simply have it delete all files EXCEPT the most recent, then you would still have 006 and your logger would generate 007?
That seems safer than creating a zero length file because you would have to make sure it wasn't copied over the real 006 on the backup.
edit: Sorry can't help with how to do this solely in batch, but there are a bunch of unix utils, specifically find and touch that are perfect for this. There are numerous windows ports of these - search on SO for options.
Robocopy.exe (free download in the windows server resource kit) can do copy from one dir to another AND has the option to watch a dir for new files and copy them when they are closed or changed
What would be some of the test cases involved int testing the copy command in windows environment ?
Holy geez! There are likely MILLIONS of test cases that are both legitimate and unique for the "copy" command. Your best approach is to come up with different domains of inputs and scenarios. Here's the domains I came up with a a handful of scenarios for each classification. Then the fun begins when you start doing testing across different the different domains (e.g. copy from a read-only set of file by wildcard to a network share)
By no means is this an exhaustive list of domains or test scenarios. But should get you started... In the majority of cases, the test should involve comparing the source and destination files to validate the contents match.
Copying between:
network share
A really slow network share across the Internet
partitions
disks
disks of different types (flash, usb, external sata, SSD, etc...)
directories
within the same directory
Naming
Normal file name
Filename that exceeds 8.3 (verylongfilename.whatever)
Copying a very long file name, but referencing it by it's 8.3 name (copy verylo~1.wha d:\)
A full directory path that exeeds MAX_PATH (260) characters (e.g. c:\a\very\long\directory\name\that\goes\on\forever\in\length......foo.txt)
By absolute addressing (\\?\\c:\foo\foo.txt)
wildcards (e.g. *.* *.txt foo?.txt )
A filename with unicode characters
A filename with illegal characters in it (there are creative ways to get these files on disk)
Attributes
Testing with different file attributes (read-only, hidden, system, archive, etc...)
Validate timestamp is preserved across copies
Validate timestamp is preserved across network file share copies when the destination machine is in another timezone
NTFS ACLs are preserved
Addressing types
reference by absolute path (e.g. copy c:\some\directory\foo.txt c:\other\place\foo.txt)
reference by relative path (e.g. copy ..\..\documents\whatever\foo.txt subdirectory/foo.txt)
By absolute drive letter into current working directoroy of destination (with no path (e.g. copy foo.txt d:)
Network share mounted to a drive letter
Failure cases, edge cases, and hack attacks
Try to copy a file onto itself (e.g copy c:\foo.txt c:\foo.txt)
Copy when the network share is down.
Unplug the network cable in the middle of a network file copy
copy to a read only directory
copy when the source file is locked.
copy the when destination file exists but the destination file exists and is read only
Detach the external disk right before the file copy starts
disk is near full (But would be full before the entire copy finishes)
disk is full
Unplug the power cable in the middle of the copy!
During a very long copy, start another copy with the same source file, but to another destination
During a very long copy, start another copy with a different source file, but the the same destination
During a very long copy, start another copy with the same source and destination files!
File types
ascii file
unicode file
binary file
Environments
RAID configurations
FAT and NTFS
Windows XP, Vista, 7, Server 2003, etc... (you can quantify this by asking the requirement of "which OS" up front)
Virtual Machine (VMWare, virtual PC, hypervisor, etc...)
Intel and AMD
I am in the middle of writing a tool that finds lost files of an iTunes library, for both Mac and Windows. On the Mac, I can quickly find files by naming using the wonderful "CatalogSearch" function.
On Windows, however, there seems to be no OS API for searching by file name (or is there?).
After some googling, I learned that there are tools (like TFind, Everything) that read the NTFS directory directly and scan it to find files by name.
I would like to do the same, but without having to start from scratch (although I've written quite a few disk tools in the past, I've never had the energy to dig into NTFS).
I wonder if there are ready-made libs around, possibly as a .dll, that would give me this search feature: Pass in a file name, get back its path.
Alternatively, what about the Windows indexing service? At least when I tried this on a recently installed XP Home system, the Search operation under the Start menu would actually scan all directories, which suggests that it has no complete database. As I'm not a Windows user at all, I wonder why this isn't working.
In the end, the complete solution I need is: I have a list of file names to find, and I need code that searches the entire disk (or uses a DB for it) to get me all results in one go. E.g, the search should not start a new full scan for every file I'm looking up. That's why I think the MFT way would be optimal, as it could quickly iterate over all names, comparing each to my list.
The best way to solve your problem seems to be by using the Windows Change Journal.
Problem: If it is not enabled for a volume or the volume is a non-NTFS you need a fallback (or enable the Change Journal if it is NTFS). You need administrator rights as well to access the Change Journal.
You get the files by using the FSCTL_ENUM_USN_DATA and DeviceIOControll with LowUsn=0. This directly accesses the MFT and writes all filenames into the supplied buffer. Because it sequentially acesses the MFT it is faster than the FindFirstFile API.