Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
I have a separate partition on my disk formatted with FAT32. When I hibernate windows, I want to be able to load another OS, create/modify files that are on that partition, then bring Windows out of hibernation and be able to see the changes that I've made.
I know what you're going to type, "Well, you're not supposed to do that!" and then link me to some specs about how what I'm trying to do is wrong/impossible/going to break EVERYTHING. However, I'm sure there's some way I can get around that. :)
I don't need the FAT32 partition in Windows, except to read the files that were written there, then I'm done - so whatever the solution is, it's acceptable for the disk to be completely inaccessible for a period of time. Unfortunately, I can't take the entire physical disk offline because it is just a partition of the same physical device that windows is installed on -- just the partition.
These are the things I've tried so far...
Google it. I got at least one "this is NEVER going to happen" answer. Unacceptable! :)
Unmount the disk before hibernating. Mount after coming out of hibernation. This seems to have no effect. Windows still thinks the FAT is the same as it was before, so whatever data I wrote to disk is lost, and any files I resized are corrupted. If any of the file was cached, it's even worse.
Use DeviceIoControl to call IOCTL_DISK_UPDATE_PROPERTIES to try and refresh the disk (but the partition table hasn't changed, so this doesn't really do anything).
Is there any way to invalidate the disk/volume read cache to force windows to go back to the disk?
I thought about opening the partition and reading/writing directly by using libfat and bypassing the cache or something is overkill.
So I finally got a solution to my problem. In my mind, I associated Mount Point with Mount. These are NOT the same thing. Removing all of the volume mount points does not make the volume unmounted. It's still mounted but not in the sense that you have a path you can access in explorer.
This is the article that started it all.
It also goes to show that searching for your EXACT problem, as opposed to the perceived problem can help a lot!
So there were a couple of solutions, one was to constantly call NtSetSystemInformation() in a tight loop to set the "SYSTEMCACHEINFORMATION" property to essentially empty/clear the cache whenever the system is going to hibernation. Then stop the loop when you come out. This, to me, seemed like it could affect system performance. So I discarded it.
Even better though, is the recommended solution to a slightly different problem presented in this MSDN article, which provides direction to an even better solution to the problem: Dismounting Volumes in a Hibernate Once/Resume Many Configuration
Now I have a service which will flush the write caches, then lock and dismount the volume whenever the system goes into hibernation/sleep and release the lock on the volume as soon as it comes out.
Here's a little bit of code.
OnHibernate>
volumeHandle = CreateFile(volumePath,
GENERIC_READ|GENERIC_WRITE,
FILE_SHARE_READ|FILE_SHARE_WRITE,
NULL,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
0 );
FlushFileBuffers( volumeHandle );
DeviceIoControl( volumeHandle, FSCTL_LOCK_VOLUME, NULL, 0, NULL, 0, &cbReturned, NULL ) ;
DeviceIoControl( volumeHandle, FSCTL_DISMOUNT_VOLUME, NULL, 0, NULL, 0, &cbReturned, NULL );
//Keep the handle open here.
//System hibernates.
OnResume>
DeviceIoControl( volumeHandle, FSCTL_UNLOCK_VOLUME, NULL, 0, NULL, 0, &cbReturned, NULL )
CloseHandle(volumeHandle)
Hopefully this helps someone else out in the future :)
Well, you're not supposed to do that! ;-)
Since the operating system (Windows in this case, but Linux is the same) writes some of its internal filesystem structures in the hibernation image, it is going to be very confused if the disk contents change while it's "running" (think of hibernation as just a long pause in the operating system's execution).
What I can suggest is that you completely bypass the issue: format the partition as ext2. There are Windows programs to read an ext2 partition, which you can use to get data out of it, and most modern operating systems should be able to read/write it (since it's a quite common Unix-style filesystem). Avoid ext2 IFS drivers; you want to take the filesystem access out of the kernel and into a program which you can open and close at will.
Use Linux to create the partition as a hidden FAT32 partition. Linux will let you mount the partition and write files. Windows will not let you mount the partition and read files, and Windows will not corrupt the partition. But there are third party libraries that will read the partition while Windows is running.
To clarify, hidden means that the partition type is different from an ordinary FAT32 partition type. If your ordinary partition type is 0x0C then the corresponding hidden type is 0x1C.
On a related but different problem, I used the following:
Running cmd as administrator (works from batch file)
DISKPART
SELECT G
REMOVE
ASSIGN LETTER=G
EXIT
This unmounts the volume (G:) and then remounts it. Any read of the disk (in my case a device pretending to be a USB Mass Storage Device formatted as FAT16) will actually read the device, so the read cache is effectively flushed.
Downside is that starting DISKPART takes about 4 seconds, but that's probably not a problem in a hibernating situation.
My memory is that the FAT table is read during the OS boot and mounting of the volume. Can't you do a shutdown, then modify the FAT, then reboot Windows?
As far as I can tell, Windows does caching at the disk level. However, if a partition has a type that Windows refuses to read or write (ext2, hidden FAT32, etc.) then that partition's contents should never get into Windows caches in the first place.
With DOS, it was typing ctrl+c, twice if I recall well.
With Linux, sync; echo 3 > /proc/sys/vm/drop_caches or a script thereof, of course ;-)
With Windows, interpolate ;-) Or install VirtualBox and Ubuntu+Wine to develop compatibly.
Well, I vaguely remember that former Windows used a diskcache program to start a process and that diskcache could be used to send the process a signal to flush and purge the whole cache. If things have evolved gently, you might be able to send such a signal to a Windows process. Sorry I'm no longer using Windows partly because of such obscurity.
Related
I am willing to write an backup application that:
read a ntfs partition without using windows api (done)
write a bootable ntfs partition from the data it saved (planned)
i have a problem with $logfile, im scared that if i simply copy it, it could make windows believe that this partition is in a bad state and try to fix it(and would probably corrupt stuff since this scenario isn't supposed to happen). i currently have very little understanding of how logfile work except that it is some kind of transaction and it use entries sequence numbers.
my question is the following:
what happen if mft entries don't fit $logFile content? (like sequences numbers)
can i bypass this by not copying $logfile? or at least removing a part of it content? (i guess windows wouldn't try to fix anything but i cant be sure)
if it doesn't work, what could i do to make $logfile safe to copy?
How do you request Windows to spin down a hard disk programmatically? Is there any user-mode function I can call (or kernel-mode function to call or IRP to send) in order to make this happen?
I've tried making a program to send an ATA STANDBY command directly to the hard disk, but the problem is that this method doesn't inform the system, and hence whenever the system needs to flush the cache, it'll wake up the hard disk again. How do I tell the system to do this for me? (If the system does it, it'll save up the cache and "burst" the data when it gets too large, instead of writing in small increments.)
(The entire point here is to do this directly, not by changing the system-wide spin-down timeout to a 1-second period and waiting for the disk to spin down. I need a function I can call at a specific moment in time when I'm using my laptop, not something generic that doesn't suit 95% of situations.)
How far I've gotten so far:
I have a feeling that PoCallDriver and IRP_MJ_POWER might be useful for this, but I have very limited kernel-mode programming experience (and pretty much zero driver experience) so I really have no idea.
Please read:
Update:
People seem to be repeatedly mentioning the solutions that I have already mentioned do not work. Like I said above, I've already tried "hacky" solutions that change the timeout value or that directly issue the drive a command, and the entire reason I've asked this question here is that those did not do what I needed. Please read the entire question (especially paragraphs 2 and 3) before repeating what I've already said inside your answers -- that's the entire difficulty in the question.
More info:
I've found this document about Disk Idle Detection to be useful, but my answer isn't in there. It states that the Power Manager sends an IRP to the disk driver (hence why I suspect IRP_MJ_POWER to be useful), but I have no idea how to use the information.
I hope this helps:
This: http://msdn.microsoft.com/en-us/library/aa394173%28VS.85%29.aspx
Leads to this:
http://msdn.microsoft.com/en-us/library/aa394132%28VS.85%29.aspx#properties
Then, you can browse to this:
http://msdn.microsoft.com/en-us/library/aa393485(v=VS.85).aspx
This documentation seems to outline what you are looking for I think.
P.S. Just trying to help, don't shoot the messanger.
Have you tried WMI? Based on MSDN documentation, you should be able to send spindown command to HDD via WMI:
http://msdn.microsoft.com/en-us/library/aa393493%28v=VS.85%29.aspx
uint32 SetPowerState(
[in] uint16 PowerState,
[in] datetime Time
);
EDIT:
This code lists all drives in system and drives that support this API:
strServer = "."
Set objWMI = GetObject("winmgmts://" & strServer & "/root\cimv2")
rem Set objInstances = objWMI.InstancesOf("CIM_DiskDrive",48)
Set objInstances = objWMI.ExecQuery("Select * from CIM_DiskDrive",,48)
On Error Resume Next
For Each objInstance in objInstances
With objInstance
WScript.Echo Join(.Capabilities, ", ")
WScript.Echo Join(.CapabilityDescriptions, ", ")
WScript.Echo .Caption
WScript.Echo .PNPDeviceID
WScript.Echo "PowerManagementCapabilities: " & .PowerManagementCapabilities
WScript.Echo "PowerManagement Supported: " & .PowerManagementSupported
WScript.Echo .Status
WScript.Echo .StatusInfo
End With
On Error Goto 0
Next
Just save this code as a .vbs file and run that from command line.
I do not have an answer to the specific question that Mehrdad asked.
However, to help others who find this page when trying to figure out how to get their disk to standby when it should but doesn't:
I found that on a USB disk, MS PwrTest claims that the disk is off, but actually it is still spinning. This occurs even with really short global disk timeouts in win 7. (This implies that even if the system thinks it has turned the disk off, it might not actually be off. Consequently, Mehrdad's original goal might not work even if the correct way to do it is found. This may relate to how various USB disk controllers implement power state.)
I also found that the program HDDScan successfully can turn off the disk, and can successfully set a timeout value that the disk honors. Also, the disk spins up when it is accessed by the OS, a good thing if you need to use it, but not so good if you are worrying about it spinning up all the time to flush 1kB buffers. (I chose to set the idle timeout in HDDScan to 1 minute more than the system power manager timeout. This hopefully assures that the system will not think the disk is spun up when it is not.)
I note that powercfg has an option to prevent the idle clock from restarting from small infrequent disk writes. (Called "burst ignore time.")
You can get HDDScan here: HDDScan.com and PwrTest here: Windows Driver Kit. Unfortunately, the PwrTest thing forces you to have a lot of other MS stuff installed first, but it is all free if you can figure out how to download it from their confusing web pages.
While there is no apparent way to do what you're asking for (i.e. tell power management "act as if the timer for spinning down the disk has expired"), there may be a couple ways to simulate it:
Call FlushFileBuffers on the drive (you need to be elevated to open \\.\C), then issue the STANDBY command to the drive.
Make the API call that sets the timeout for spinning down the disk to 1 second, then increase it back to its former value after 1 second. Note that you may need to ramp up to the former value rather than immediately jump to it.
I believe the Devcon Command line utility should be able to accomplish what you need to do. If it does - the source code is available in the Windows Ddk.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 2 years ago.
Improve this question
I want to write a program that can recover deleted files from hard drive ( FAT32/NTFS partition Windows). I don't know where to start from. What should be the starting point of this? What should i read to pursue this? Help is required. Which system level structs should i study?
It's entirely a matter of the filesystem layout, how a "file" actually looks on disk, and what remains when a file is deleted. As such, pretty much all you need to understand is the filesystem spec (for each and every filesystem you want to support), and how to get direct block-level access to the HD data. It might be possible to reuse some code from existing filesystem drivers, but it will need to be modified to process structures that, from the point of view of the filesystem, are gone.
NTFS technical reference
NTFS.com
FAT32 spec
You should know first how file deletion is done in FAT32/NTFS, and how other undelete softwares work.
Undelete software understands the internals of the system used to store files on a disk (the file system) and uses this knowledge to locate the disk space that was occupied by a deleted file. Because another file may have used some or all of this disk space there is no guarantee that a deleted file can be recovered or if it is, that it won't have suffered some corruption. But because the space isn't re-used straight away there is a very good chance that you will recover the deleted file 100% intact. People who use deleted file recovery software are often amazed to find that it finds files that were deleted months or even years ago. The best undelete programs give you an indication of the chances of recovering a file intact and even provide file viewers so you can check the contents before recovery.
Here's a good read (but not so technical): http://www.tech-pro.net/how-to-recover-deleted-files.html
This is not as difficult as you think. You need to understand how files are stored in fat32 and NTFS. I recommend you use winhex an application used for digital forensics to check your address calculations are correct.
Ie NTFS uses master file records to store data of the file in clusters. Unlink deletes file in c but if you look at the source code all it does is removes entry from table and updates the records. Use an app like winhex to read information of the master file record. Here are some useful info.
Master boot record - sector 0
Hex 0x55AA is the end of MBR. Next will be mft
File name is mft header.
There is a flag to denote folder or file (not sure where).
The file located flag tells if file is marked deleted. You will need to change this flag if you to recover deleted file.
You need cluster size and number of clusters as well as the cluster number of where your data starts to calculate the start address if you want to access data from the master file table.
Not sure of FAT32 but just use same approach. There is a useful 21 YouTube video which explains how to use winhex to access deleted file data on NTFS. Not sure the video but just type in winhex digital forensics recover deleted file. Once you watch this video it will become much clearer.
good luck
Just watched the 21 min YouTube video on how to recover files deleted in NTFS using winhex. Don't forget resident flag which denotes if the file is resident or not. This gives you some idea of how the file is stored either in clusters or just in the mft data section if small. This may be required if you want to access the deleted data. This video is perfect to start with as it contains all the offset byte position to access most of the required information relative to beginning of the file record. It even shows you how to do the address calculation for the start of the cluster. You will need to access the table in binary format using a pointer and adding offsets to the pointer to access the required information. The only way to do it is go through the whole table and do a binary comparison of the filename byte for byte. Some fields are little eindian so make sure you got winhex to check your address calculations.
To restrict the scope, let assume we are in Windows world only.
Also assume we don't want to play with permission policy.
Is it possible for us to create a file that cannot be copied?
Thank you in advance.
"Trying to make digital files uncopyable is like trying to make water not wet." ~ Bruce Schneier
No. You can't create a file that a SYSADMIN can't copy. You could encrypt it, though.
Well, how about creating a file that uses up more than 50% of the total space on that machine and that is not compressible?
For instance, let us assume that you want to save a boolean (true or false) in such a fashion.
Depending on its value, you could then write a bit stream of ones or zeroes and encrypt said stream using some kind of encryption algorith, such as AES in CBC mode. This gives you the added advantage of error correction. Even in case of massive data corruption, you should be able to recover your boolean by checking whether ones or zeroes are prevalent in the decrypted stream.
In that case you cannot copy it around (completely) on the machine...
Of course, any type of external memory that can be added to the system would pose a problem in this scenario. But the file would be already encrypted, so don't worry about it too much...
Any file that can be read can have its contents written to another location (such as another file, i.e. copied).
The only thing you can do is limit who/what can read the file.
What is the motivation behind? If it is a read-only file, you can have it as embedded resources within your assembly.
Nice try, RIAA.
But seriously, no you can not. It is always possible to copy, you can just make it it more difficult for people to make sense of the file or try to hide it using like encryption. Spotify does it.
If you really try hard thou, you cold make a root-kit for windows and use it to prevent windows from even knowing about the file and also prevent copies. The file will still be there and copy-able by other tools, or Linux accessing the ntfs.
If in a running process you open a file and hold an exclusive lock, then other processes cannot read the file until you close the handle or your process terminates. However, as admin you could forcibly remove the lock handle.
Short answer: No.
You can, of course, use security settings to limit who can read the file. But if someone can read it, then they can copy it. Even if you found some operating system trick to disable "ordinary" copying, if someone can read the file, they can extract the contents, store it in memory, and then write it somewhere else.
You can encrypt the contents so it's only useful to your own program, that knows how to decrypt it.
That's about it.
When using Windows 7 to copy some files from a hard drive, certain files popped up a message saying they could not be copied in their entirety; certain data would be omitted from the copy. I suspect that had something to do with slack space at the end of the files, though I thought the message was curious. I would have expected the copy operation to just ignore the slack space.
If you are running old (OLD) versions of windows, there are certain characters you can put in the filename that make it invalid, not listed in folders, etc. They were used a lot in the old pub ftp days of filesharing ;)
In the old DOS days, you used to be able to flag disk sectors as bad and still read from them. This meant the OS ignored the sector in question but your application would know where to look and be able to get the data. Not sure this would work these days.
Another old MS-DOS trick was to put a space character in the middle of the filename (yes, spaces were valid characters for filenames). Since there was no method on the command line to escape a space, the file couldn't be copied using the DOS commands.
This answer is outside Windows so yeah
Dont know if its already been said but what about a file that is an inseperable part of the firmware so that it is always on AND running, perhaps it has firmware that generates a sequence that is required for the other . AN incedental effect of its running is to prevent any 80% or more of its code from being replicated. Lets say its on an entirely different board, protected by surge protectors, heavy em proof shielding and anything else required to make it completely unerasable.
If its possible to make a program that is ALWAYS on and running as long as the copying software is running then yes.
I have another way and this IS with windows. I will come to your house and give you a disk, i will then proceed to destroy every single computer you put the disk into. This doesnt work on XP
Well technically you could create and write to a write-only network share.
I have backups of files archived in optical media (CDs and DVDs). These all have par2 recovery files, stored on separate media. Even in cases where there are no par2 files, minor errors when reading on one optical drive can be read fine on another drive.
The thing is, when reading faulty media, the read time is very, very long, because devices tend to retry multiple times.
The question is: how can I control the number of retries (ie set to no retries or only one try)? Some system call? A library I can download? Do I have to work on the SCSI layer?
The question is mainly about Linux, but any Win32 pointers will be more than welcome too.
man readom, a program that comes with cdrecord:
-noerror
Do not abort if the high level error checking in readom found an
uncorrectable error in the data stream.
-nocorr
Switch the drive into a mode where it ignores read errors in
data sectors that are a result of uncorrectable ECC/EDC errors
before reading. If readom completes, the error recovery mode of
the drive is switched back to the remembered old mode.
...
retries=#
Set the retry count for high level retries in readom to #. The
default is to do 128 retries which may be too much if you like
to read a CD with many unreadable sectors.
The best tool avaliable is dd_rhelp. Just
dd_rhelp /dev/cdrecorder /home/myself/DVD.img
,take a cup of tea and watch the nice graphics.
The dd_rhelp rpm package info:
dd_rhelp uses ddrescue on your entire disc, and attempts to gather the maximum
valid data before trying for ages on badsectors. If you leave dd_rhelp work
for infinite time, it has a similar effect as a simple dd_rescue. But because
you may not have this infinite time, dd_rhelp jumps over bad sectors and rescue
valid data. In the long run, it parses all your device with dd_rescue.
You can Ctrl-C it whenever you want, and rerun-it at will, dd_rhelp resumes the
job as it depends on the log files dd_rescue creates. In addition, progress
is shown in an ASCII picture of your device being rescued.
I've used it a lot myself and Is very, very realiable.
You can install it from DAG to Red Hat like distributions.
Since dd was suggested, I should note that I know of the existence and have used sg_dd, but my question was not about commands (1) or (1m), but about system calls (2) or libraries (3).
EDIT
Another linux command-line utility that is of help, is sdparm. The following flag seems to disable hardware retries:
sudo sdparm --set=RRC=0 /dev/sr0
where /dev/sr0 is the device for the optical drive in my case.
While checking whether hdparm could modify the number of retries (doesn't seem so), I thought that, depending on the type of error, lowering the CD-ROM speed could potentially reduce the number of read errors, which could actually increase the average read speed. However, if some sectors are completely unreadable, then even lowering the CD-ROM speed won't help.
Since you are asking about driver level access, you should look into SCSI commands, or perhaps an ASPI like API. On windows VSO software (developers of blindread/blindwrite below) have developed a much better API, Patin-Couffin, that provides locked low level access:
http://en.wikipedia.org/wiki/Patin-Couffin
That might get you started. However, at the end of the day, the drive is interfaced with SCSI commands, even if it's actually USB, SATA, ATA, IDE, or otherwise. You might also look up terms related to ATAPI, which was one of the first specifications for this CD-ROM SCSI layer interface.
I'd be surprised if you couldn't find a suitable linux library or example of dealing with the lower level commands using the above search terms and concepts.
Older answer:
Blindread/blindwrite was developed in the heyday of cd-rom protection schemes often using intentionally bad sectors or error information to verify the original CD.
It will allow you to set a whole slew of parameters, including retries. Keep in mind that the CD-ROM drive itself determines how many times to retry, and I'm not sure that this is settable via software for many (most?) CD-ROM drives.
You can copy the disk to ISO format, ignoring the errors, and then use ISO utilities to read the data.
-Adam
Take a look at the ASPI interface. Available on both windows and linux.
dd(1) is your friend.
dd if=/dev/cdrom of=image bs=2352 conv=noerror,notrunc
The drive may still retry a bit, but I don't think you'll get any better without modifying firmware.