I see cautions about treating network drives as looking like lettered drive. Can you turn it on its head?
So. What are the cautions about mapping a drive to a your-choice-of-name share in the usual way. For example share name "01-D-drive". In file explorer under network (for my local machine) I now have another "neater" more logical way to access my drives (shared with appropriate permissions) that gives me ordering and naming possibilities. Cautions might be raised that the performance is not the same--is it prohibitively expensive? Another step might be to map these "local" shares to a network path so I have top level access in file explorer to the "drives" e.g. define network resource to this local path //my-local-computer/01-D-drive. Where will this break down? Here's an example:
Open share on //my-local-computer to see (and use). --note: lettered drives are still there in file explorer
my-local-computer
01-zdrive
02-any-name-any-order
03-hey-system-drive-should-not-be-shared
04-why-not?
don't want to open share (surely even more overhead??) and reference in c
//my-local-computer/01-zdrive
etc, etc and so forth. this silly editor is trashing my paragraphs--try adding a blank line
There's very little overhead in using the redirector to access local UNC shares instead of local drives. I don't think you'll see a difference in either throughput or latency.
PS: After actually testing this, there is quite a difference. Throughput is very close but latency (open file, dir scan, or recursive traversal) is much slower:
dir /s on 2012R2 file server, 20,000 dirs, 61,000 files:
local drive letter: 3 s
UNC share: 43 s
Related
Is it possible to write to free clusters on disk or read data from them using Windows APIs? I found Defrag API: https://learn.microsoft.com/en-gb/windows/desktop/FileIO/defragmenting-files
FSCTL_GET_VOLUME_BITMAP can be used to obtain allocation state of each cluster, FSCTL_MOVE_FILE can be used to move clusters around. But I couldn't find a way of reading data from free clusters or writing data to them.
Update: one of the workarounds which comes to mind is creating a small new file, writing some data to it, then relocating it to desired position and deleting the file (the data will remain in freed cluster). But that still doesn't solve reading problem.
What I'm trying to do is some sort of transparent cache, so user could still use his NTFS partition as usual and still see these clusters as free space, but I could store some data in them. Data safety is not of concern, it can be overwritten by user actions and will just be regenerated / redownloaded later when clusters become free again.
There is no easy solution in this way.
First of all, you should create own partition of the drive. It prevents from an accidental access to your data from OS or any process. Then call CreateFileA() with name of the partition. You will get raw access to the data. Please bear in mind that the function will fail for any partition accessed by OS.
You can perform the same trick with a physical drive too.
The docs
One way could be to open the volume directly via using CreateFile with the volumes UNC path as filename arguement (e.g.: \\.\C:).
You now can directly read and write to the volume.
So you maybe can achieve your desired goal with:
get the cluster size in bytes with GetDiskFreeSpace
get the map of free clusters with DeviceIoControl and FSCTL_GET_VOLUME_BITMAP
open the volume with CreateFile with its UNC path \\.\F:
(take a careful look into the documentation, especially the Remarks sections part about opening drives and volumes)
seek to the the offset of a free cluster (clusterindex * clusterByteSize) by using SetFilePointer
write/read your data with WriteFile/ReadFile on the handle, retreived by above CreateFile
(Also note that read/write access has to be sector aligned, otherwise the ReadFile/WriteFile calls fail)
Please note:
this is only meant as a starting point for your own research. This is not a bullet proof cooking receipt.
Backup your data before messing with the file system!!!
Also keep in mind that the free cluster bitmap will be outdated as soon as you get it (especially if using the system volume).
So I would strongly advise against use of such techniques in production or customer environments.
my goal is to make a backup program reading a physical disk (with NTFS partitions) while using VSS for data consistency.
i use windows api's functions CreateFile with '\.\PhysicalDriveN'
as described here (basically, it allow me to access a disk as a big file)
https://support.microsoft.com/en-us/help/100027/info-direct-drive-access-under-win32
for tests i create volume shadows with this command
wmic shadowcopy call create Volume='C:\'
this is a temporary solution, i plan on using VSS via the program itself
My question is:
how are stored Volume shadows? does it stores data that have been modified since the volume shadow or does it store modification made since the last volume shadow?
in the first case:
when i read the disk, will i get consistent data (including ntfs metadata files)?
in the other case:
can i access a volume shadow the same way i would access a disk/partition? (in order to read hidden metadata files, etc)
-im am currenctly using windows 7 but planning on using it on differents version of windows server
-i've read a lot of microsoft doc about VSS but how it work seem really unclear for me (if you answer with one please explain a bit it meaning)
-i know that Volume shadows are stored in the folder "System Volume Information" as files with names like {3808876b-c176-4e48-b7ae-04046e6cc752}
"how are stored Volume shadows? does it stores data that have been modified since the volume shadow or does it store modification made since the last volume shadow?"
A hardware or software shadow copy provider uses one of the following methods for creating a shadow copy:(Answer by msdn doc)
Complete copy This method makes a complete copy (called a "full copy"
or "clone") of the original volume at a given point in time. This copy
is read-only.
Copy-on-write This method does not copy the original volume. Instead,
it makes a differential copy by copying all changes (completed write
I/O requests) that are made to the volume after a given point in time.
Redirect-on-write This method does not copy the original volume, and
it does not make any changes to the original volume after a given
point in time. Instead, it makes a differential copy by redirecting
all changes to a different volume.
"when i read the disk, will i get consistent data (including ntfs metadata files)?"
Even if an application does not have its files open in exclusive mode, it is possible—because of the finite time needed to open, back up, and close a file—that files copied to storage media may not all reflect the same application state.
"can i access a volume shadow the same way i would access a disk/partition? (in order to read hidden metadata files, etc)"
Requester Access to Shadow Copied Data
Paths on the shadow copied volume are obtained by replacing the root
of the original path with the device object. For example, given a path
on the original volume of "C:\DATABASE*.mdb" and a VSS_SNAPSHOT_PROP
instance of snapProp, you would obtain the path on the shadow copied
volume by concatenating snapProp.m_pwszSnapshotDeviceObject, "\",
and "\DATABASE*.mdb".
So i did more test and actually Shadow Volume are made at block level not file level. it mean that by using createfile with the path
\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1 it would work in a similar way than using createfile with the path \\.\C:
So yeah you can access a shadow copy file system, it have it own boot sector, mft, etc.
I run the command to find the files named ".*large_files.*"
[root#iz2ze9wve43n2nyuvmsfx5z ~]# find / -iregex ".*large_files.*"
/root/search_large_files.py
It found the file but the cursor is shinning endless even if I leave it alone for over half an hour.
What's the bug in my codes to cause the problem?
Well, it may be that you just have massive file systems :-)
But, if you think it shouldn't be taking that long, you may well have mount points that are slower than normal, such as NFS-mounts where you have to go out over the network to get file information.
You could probably see a slow-down in that case if you just run find / on its own. If it goes out to an external location (like, I don't know, a ZX80 running in Antartica), the output rate may show that, and you'll be able to identify where in the hierarchical structure it happens.
Another possibility is to restrict it to the actual file system you're on to minimise the chance it will go external. That would be by using the xdev flag to prevent it crossing file systems. On my VM with one root file system but mounts for my C and D host drives, I cut the time down from two minutes to seventeen seconds.
Of course, that won't go to other local file systems but you could, if necessary write a script to find (with xdev) the file on all file systems marked ext4 (and whatever other ones you deem to be local).
I am using the Intel Solid-State Drive Toolbox to view an SSD drive. This utility has an option to manually run "TRIM". What I found odd is the utility reports "The selected Intel SSD has no partition letter. This feature requires a partition letter to run."
I have the disk mounted as a junction point. I hope this is a limitation of Intel's utility, or does Windows 7 TRIM require a drive to be assigned a drive letter in order for it to work?
The way trim works, is that it is the hint to the SSD to indicate which address areas is not being used to contain data. This allows the SSD to optimize internally (usually save work being done by "garbage collection").
when there is no partition on the drive, generally it means that everything is "trimmed". This may not be the case if the SSD wasn't made aware of that. So in this case, I think it's the tool, unable to figure out what it could and could not trim and may just want to avoid trimming unintentionally.
Aside from that fact, the trim feature is specific to ATA protocol. Meaning, it's a command sent to the drive at the lower level so it's not tied to Windows 7 or an application. It's open for anything that is will and able to send the command.
Suppose you have a full path to an accessible file or folder on the system. How can I get some kind of unique identifier for the physical device that the file (or folder) actually resides on?
My first attempt was to use System.IO.DriveInfo which depends on having a drive letter. But UNC paths and multiple network drives mapped to the same physical device on a server add some complications. For example these 3 paths all point to the same folder on the same device.
\\myserver\users\brian\public\music\
s:\users\brian\public\music\ (here s:\ is mapped to \\myserver\)
u:\public\users\music\ (here u:\ is mapped to \\myserver\users\brian\)
Ultimately my goal is to take these multiple paths and report the amount of used and free disk space on each device. I want to combine these 3 paths into a single item in the report and not 3 separate items.
Is there any Windows API that can help find this information given any arbitrary full path?
This win API call should get you what you need regarding disk space
GetDiskFreeSpaceEx
http://msdn.microsoft.com/en-us/library/aa364937(VS.85).aspx
Also, to determine if the three mappings all are from the same physical disk, perform a call to
GetVolumeInformation
and compare the returned volume serial numbers
http://msdn.microsoft.com/en-us/library/aa364993(VS.85).aspx