Is there a way to guarantee that a file on Windows (using the NTFS file system) will use contiguous sectors on the hard disk? In other words, the first chunk of the file will be stored in a certain sector, the second chunk of the file will be stored in the next sector, and so on.
I should add that I want to be able to create this file programmatically, so I'd rather not just ask the user to defrag their harddrive after creating this file. If there is a way to programmatically defrag just the file that I create, then that would be OK too.
I would start here:
http://technet.microsoft.com/en-us/sysinternals/bb897428.aspx
and follow Mark's documentation of the defrag stuff:
http://technet.microsoft.com/en-us/sysinternals/bb897427.aspx
I know of no such guarantees.
But also keep in mind that NTFS "files" are comprised of multiple data streams. So you are actually looking for a way to guarantee that a stream is contiguous.
I believe there's no way to achieve that. You can only defragment the file after it's been written.
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 2 years ago.
Improve this question
I want to write a program that can recover deleted files from hard drive ( FAT32/NTFS partition Windows). I don't know where to start from. What should be the starting point of this? What should i read to pursue this? Help is required. Which system level structs should i study?
It's entirely a matter of the filesystem layout, how a "file" actually looks on disk, and what remains when a file is deleted. As such, pretty much all you need to understand is the filesystem spec (for each and every filesystem you want to support), and how to get direct block-level access to the HD data. It might be possible to reuse some code from existing filesystem drivers, but it will need to be modified to process structures that, from the point of view of the filesystem, are gone.
NTFS technical reference
NTFS.com
FAT32 spec
You should know first how file deletion is done in FAT32/NTFS, and how other undelete softwares work.
Undelete software understands the internals of the system used to store files on a disk (the file system) and uses this knowledge to locate the disk space that was occupied by a deleted file. Because another file may have used some or all of this disk space there is no guarantee that a deleted file can be recovered or if it is, that it won't have suffered some corruption. But because the space isn't re-used straight away there is a very good chance that you will recover the deleted file 100% intact. People who use deleted file recovery software are often amazed to find that it finds files that were deleted months or even years ago. The best undelete programs give you an indication of the chances of recovering a file intact and even provide file viewers so you can check the contents before recovery.
Here's a good read (but not so technical): http://www.tech-pro.net/how-to-recover-deleted-files.html
This is not as difficult as you think. You need to understand how files are stored in fat32 and NTFS. I recommend you use winhex an application used for digital forensics to check your address calculations are correct.
Ie NTFS uses master file records to store data of the file in clusters. Unlink deletes file in c but if you look at the source code all it does is removes entry from table and updates the records. Use an app like winhex to read information of the master file record. Here are some useful info.
Master boot record - sector 0
Hex 0x55AA is the end of MBR. Next will be mft
File name is mft header.
There is a flag to denote folder or file (not sure where).
The file located flag tells if file is marked deleted. You will need to change this flag if you to recover deleted file.
You need cluster size and number of clusters as well as the cluster number of where your data starts to calculate the start address if you want to access data from the master file table.
Not sure of FAT32 but just use same approach. There is a useful 21 YouTube video which explains how to use winhex to access deleted file data on NTFS. Not sure the video but just type in winhex digital forensics recover deleted file. Once you watch this video it will become much clearer.
good luck
Just watched the 21 min YouTube video on how to recover files deleted in NTFS using winhex. Don't forget resident flag which denotes if the file is resident or not. This gives you some idea of how the file is stored either in clusters or just in the mft data section if small. This may be required if you want to access the deleted data. This video is perfect to start with as it contains all the offset byte position to access most of the required information relative to beginning of the file record. It even shows you how to do the address calculation for the start of the cluster. You will need to access the table in binary format using a pointer and adding offsets to the pointer to access the required information. The only way to do it is go through the whole table and do a binary comparison of the filename byte for byte. Some fields are little eindian so make sure you got winhex to check your address calculations.
Is there a tool that would show me for a specific file on disk, how fragmented it is? (How many seeks does physical disk need to make if I were to read that file in a linear fashion)
The Sysinternals tool contig with parameter -a can do this for a file or all files in a folder and its subfolders.
You can use DeviceIoControl with FSCTL_GET_VOLUME_BITMAP, FSCTL_GET_RETRIEVAL_POINTERS and FSCTL_MOVE_FILE, see Defragmenting Files.
You can also find different code examples if you search for FSCTL_MOVE_FILE.
Here is one in C and another in .NET.
filefrag is the tool you're looking for, if you're using Linux.
Use -v parameter with filename to get detailed list of fragmentation.
http://linux.die.net/man/8/filefrag
And, of course, "fragmentation" is suspect:
The file may be in pieces in the same cylinder. No seek overhead, just rotational latency. Or not as the pieces may be an optimal order (chances are near zero for this one).
The file may be "contiguous" but across several cylinders. Even reading sequentially will result in seeks.
The file may be on a stripe set and you have no idea where the boundaries are. You may skip to another controller, another spindle, or another partition on the same drive.
Be careful about what conclusions you draw.
fsutil file queryallocranges offset=<o> length=<l> <file> will show you the file's extents you will need admin rights.
I am trying to defragment a single file through Windows defragmentation API ( http://msdn.microsoft.com/en-us/library/aa363911(VS.85).aspx ) but if there is no free space block large enough for my file I would like to move other parts of files to make room for it.
The linked article mentions moving parts of other files but I can't find any information about how to find out which files to move. From the free space bitmap I can find an almost large enough space and I know the logical cluster numbers surrounding it, but from this I can't find out which files are surrounding it and a handle to the files is required to do FSCTL_MOVE_FILE which moves parts of files.
Is there any way, through the API or by parsing the MFT, to find out what file a logical cluster number is part of, and what virtual cluster number in the file corresponds to the logical cluster number found through the bitmap?
The slow but compatible method is to recursively scan all directories for files, and use the FSCTL_GET_RETRIEVAL_POINTERS. Then scan the resulting VCN-LCN mapping for the cluster in question.
Another option would be to query the USN Journal of the drive to get the File Reference IDs, then use FSCT_GET_NTFS_FILE_RECORD to get the $MFT file record.
I'm currently working on a simple Defrag program (written in Java) with the aim to pack files of a directory (e.g. all files of a large game) close together to reduce loading times and loading lags.
I use a faster method to retrieve the file mappings on the NTFS or FAT32 drive.
I parse the $MFT file directly (the format has some pitfalls), or the FAT32 file allocation table along with the directories.
The trick is to open the drive (e.g. "c:") with FileCreate for fully shared GENERIC read. The resulting handle can then be read with FileRead and FileSeek on a byte granularity. This works only in administrator mode (or elevated).
On NTFS, the $MFT might be fragmented and is a bit tricky to locate it from the boot sector info. I use the FSCTL_GET_RETRIEVAL_POINTERS on the C:\$MFT file to get its clusters.
On FAT32, one must parse the boot sector to locate the FAT table and the cluster containing root directory file. You need to parse the directory entries and recursively locate the clusters of the sub-directories.
There is no O(1) way of mapping from block # to file. You need to walk the entire MFT looking for files that contain that block.
Of course, in a live system, once you've read that data it's out-of-date and you must be prepared for failures in the move data FSCTL.
I wanted to make some simple file recovery software, where I want to try to recover files which happen to have been deleted by pressing Shift + Delete. I'm working in Windows, can anyone show me any links or documents which can help me to do so programatically? I know C, C++, .NET. Any pointers?
http://www.google.hu/search?q=file+recovery+theory&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a :)
Mainly file recoveries are looking for file headers and/or filenames in the disk as I know, then try to get the whole file by the header information.
This could be a good start: http://geeksaresexy.blogspot.com/2006/02/theory-behind-deleted-files-recovery.html
The principle of all recovery tools is that deleting a file only removes a pointer in a folder and (quick) formatting of a partition only rewrites the first sectors of the partition which contains the headers of the filesystem. An in depth analysis of the partition data (at sector level) can rebuild a big part of the filesystem data, cluster allocation tables, folders, and file cluster chains.
All course if you use a surface test tool while formatting the partition that will rewrite all sectors to make sure that they are correct, nothing will be recoverable - unless you use specialized hardware to look at remanent magnetism on the edges of the actual tracks
In windows when a file is deleted(permanent delete) it's not actually deleted from disk but the file name added with char( _ I guess) in front of it and windows ignores these when showing in explorer... and recovery tools will search these kind of file names in the disk. And your file recover integrity based on some data over written on location of deleted file. Don't know this pattern still used by windows.. but long time back I have read this some where
I need to get the list of all the files on a drive. I am using a recursive solution. But it is taking a lot of time. I was wondering that, is it possible to get the names and location of all the files on a NTFS drive from it's Master File Table? I think it will be very fast. Any suggestions?
There is a tool that will search the mft directly, it's called ndff. I have used it before and it is very fast.
Presumably it is possible to do what you want - there is another tool called "Everything" which I guess does the same thing - it also uses the USN change journal to update it's index.
When you get a list of all the files on an NTFS-formatted drive using a recursive solution, you are getting them from the MFT. There should be little disk IO outside of the MFT when simply retrieving a list of filenames and directories.
Before going down the path of determining the format of the MFT (which is available from a variety of places on the Internet) and writing code to read it directly, you should probably profile your code and determine that you aren't already CPU or IO bound.
I have the impression you're imagining some kind of list-like structure in the MFT which you can read in one go with no or minimal seeking.
This is not the case. The MFT uses a type of b-tree to store pathnames. When you scan the directory structure on your disk, you are in fact walking the MFT b-tree; you are doing what you would have to do if you accessed the MFT directly.
Yes there is, and the program I just open-sourced does exactly this.
You can read the source to find out how it works, but basically, it just looks for FILE_NAME attributes inside the $MFT and then uses the ParentDirectory field to get the parent of every file.
That way it can completely avoid reading the contents of any directory.