What is the difference between paths prefixed with \??\ and those prefixed with \\?\
At Windows 7 CMD-Line
DIR gives: \??\Volume{00000000-0000-0000-0000-000000000000}\
WMIC VOLUME LIST gives \\?\Volume{00000000-0000-0000-0000-000000000000}\
Thanks
Additional Information:
I created a directory on my system drive where i placed all my mount points into. So I did a MD C:\HDDs first, then a
MD C:\HDD\Drive1, MD C:\HDD\Drive2 ...for all my drives. After mounting the drives on those empty directories, I can see the GUID by switching to that Directory with CD /D C:\HDDs and issueing a DIR command. Maybe I have to issue a DIR /ah to Show hidden stuff, just in case the mount points are hidden directories ...
In NT, "\??\" is a path prefix that stands for the object directory that's reserved for a user's devices, or more specifically, device aliases. A device alias is implemented in the object namespace as a symbolic link that typically resolves to a device object in the "\Device" directory. Sometimes in the documentation these device aliases are referred to as "junctions" in the object namespace, which is not to be confused with "directory junctions" (or mount points) in a file system.
Using the "\??\" prefix instructs the object manager to search in the caller's local device directory, "\Sessions\0\DosDevices\[Logon Authentication ID]", which is coupled to (i.e. shadows) the global device directory, "\Global??". For efficiency, both of these directories are cached by an access token's associated logon-session record and also by each process object. The SYSTEM logon (ID 0x3E7) uses "\Global??" as its local device directory. Note that the local directory has a "Global" link to allow accessing global devices when a local device shadows the global one (e.g. "\\?\Global\Z:"), or to allow a device driver to create a global device when not executing in a SYSTEM thread. NT originally used a single "\DosDevices" directory, regardless of the caller. With the introduction of Terminal Services and fast-user switching in NT 5, they had to generalize it to the present system of local and global devices. Nowadays, for backwards compatibility, "\DosDevices" is a link to "\??".
Translating DOS paths to native NT paths is implemented by NT's user-mode runtime library (i.e. the Rtl prefixed functions that are exported by "ntdll.dll").
The straight-forward case is a path that's prefixed by either "\\.\" or "\\?\". This is a local device path, not a UNC path. (Strictly speaking it's in the form of a UNC path, but "." and "?" are reserved device domains.) For this case, the prefix is simply replaced by NT "\??\". The difference between the two WINAPI device-path prefixes is that a "\\?\" path (all backslashes, no forward slashes) is a so-called "extended" path, which bypasses all normalization, whereas a "\\.\" path gets normalized.
Device path normalization resolves "." and ".." components, replaces forward slashes with backslashes, and strips trailing spaces and dots from the final path component. Because forward slashes are translated to backslashes, the prefix of a normalized device path can be "//./" or "//?/" or any combination of slashes and backslashes, except for exactly "\\?\". Note that if the process doesn't support long paths, normalized paths are limited to less than MAX_PATH (260) characters. (Long-path support can be enabled in Windows 10 through a combination of registry and application manifest settings; consult the relevant documentation.) GetFullPathNameW handles both prefixes equivalently and even normalizes an extended path that begins with "\\?\".
UNC paths are also unsurprising. The runtime library simply replaces the leading "\\" in the normalized path with an explicit reference to the "UNC" device, i.e. "\??\UNC\" (e.g. "\\server\share" -> "\??\UNC\server\share"). Note that "\Global??\UNC" is a symbolic link to "\Device\Mup", the Multiple UNC Provider device, which is responsible for mapping the "server\share" to the correct UNC provider (e.g. to the LanmanWorkstation redirector for an SMB share).
DOS drive paths (i.e. those beginning with an "[A-Z]:" drive) are interesting in a couple of cases. The first is that the runtime library supports per-drive working directories using conventionally 'hidden' environment variables such as "=C:". For example, "C:System32" resolves to "C:\Windows\System32" if the "=C:" environment variable is set to "C:\Windows". Also, if the last component of the path is a reserved DOS device name, including if the name has trailing colons, spaces, dots, and even a file extension, the path gets translated to a device path (e.g. "C:\Windows\nul: .txt" -> "\??\nul"). (DOS devices are also reserved in the final component of relative paths that have no drive.) Otherwise, the runtime library simply prepends "\??\" to the normalized path (e.g. "C:/Windows" -> "\??\C:\Windows").
A DOS drive such as "C:" (i.e. "\Global??\C:") is an alias for an NT volume device (i.e. an object symbolic link). The NT device name is not persistent and is typically enumerated, so the final target depends on the relative order in which volumes are added, and it may even change if a volume is removed and subsequently restored. For example, the final NT path for "E:\Temp" on a removable drive may start out as "\Device\HarddiskVolume8\Temp" and then, after removing and reinserting it, the new final path is "\Device\HarddiskVolume10\Temp". The mount-point manager implements persistence using the unique ID of a volume, which it associates with a volume GUID name (e.g. "Volume{00000000-0000-0000-0000-000000000000}") and optionally (usually) a DOS drive letter. The GUID name is used to implement volume mount points in file systems that support junctions (i.e. IO_REPARSE_TAG_MOUNT_POINT reparse points), such as NTFS and ReFS.
Related
I found a reference to a file in a log that had the following format:
\\?\C:\Path\path\file.log
I cannot find a reference to what the sequence of \?\ means. I believe the part between the backslashes refers to a hostname.
For instance, on my Windows computer, the following works just fine:
dir \\?\C:\
and also, just fine with same result:
dir \\.\C:\
Questions:
Is there a reference to what the question mark means in this particular path format?
What might generate a file path in such a format?
A long read, but worth reading if you are in this domain: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx
Extract:
The Windows API has many functions that also have Unicode versions to
permit an extended-length path for a maximum total path length of
32,767 characters. This type of path is composed of components
separated by backslashes, each up to the value returned in the
lpMaximumComponentLength parameter of the GetVolumeInformation
function (this value is commonly 255 characters). To specify an
extended-length path, use the "\\?\" prefix. For example,
"\\?\D:\very long path".
and:
The "\\?\" prefix can also be used with paths constructed according to
the universal naming convention (UNC). To specify such a path using
UNC, use the "\\?\UNC\" prefix. For example, "\\?\UNC\server\share",
where "server" is the name of the computer and "share" is the name of
the shared folder. These prefixes are not used as part of the path
itself. They indicate that the path should be passed to the system
with minimal modification, which means that you cannot use forward
slashes to represent path separators, or a period to represent the
current directory, or double dots to represent the parent directory.
Because you cannot use the "\\?\" prefix with a relative path,
relative paths are always limited to a total of MAX_PATH characters.
The Windows API parses input strings for file I/O. Among other things, it translates / to \ as part of converting the name to an NT-style name, or interpreting the . and .. pseudo directories. With few exceptions, the Windows API also limits path names to 260 characters.
The documented purpose of the \\?\ prefix is:
For file I/O, the "\\?\" prefix to a path string tells the Windows APIs to disable all string parsing and to send the string that follows it straight to the file system.
Among other things, this allows using otherwise reserved symbols in path names (such as . or ..). Opting out of any translations, the system no longer has to maintain an internal buffer, and the arbitrary limit of 260 characters can also be lifted (as long as the underlying filesystem supports it). Note, that this is not the purpose of the \\?\ prefix, rather than a corollary, even if the prefix is primarily used for its corollary.
I am using date and time to label a new file that I'm creating, but when I view the file, the colon is a forward slash. I am developing on a Mac using 10.7+
Here is the code I'm using:
File.open("#{time.hour} : 00, #{time.month}-#{time.day}-#{time.year}", "a") do |mFile|
mFile.syswrite("#{pKey} - #{tKey}: \n")
mFile.syswrite("Items closed: #{itemsClosed} | Total items: #{totalItems} | Percent closed: % #{pClosed} \n")
mFile.syswrite("\n")
mFile.close
end
Here is the output (assuming the time is 1pm):
13 / 00, 11-8-2012
Why is this happening and how can I fix it? I want the output to be:
13:00, 11-8-2012
Once upon a time, before Mac OS X, : was the directory separator instead of /. Apparently OS X 10.7 is still trying to fix up programs like that. I don't know how you can fix this, if you really need the : to be there. I'd omit it :-).
EDIT: After a bit more searching this USENIX paper describes what is going on. The rule they use apparently is this:
Another obvious problem is the different path separators between HFS+ (colon, ':') and UFS
(slash, '/'). This also means that HFS+ file names may contain the slash character and not
colons, while the opposite is true for UFS file names. This was easy to address, though it
involves transforming strings back and forth. The HFS+ implementation in the kernel's VFS
layer converts colon to slash and vice versa when reading from and writing to the on-disk
format. So on disk the separator is a colon, but at the VFS layer (and therefore anything
above it and the kernel, such as libc) it's a slash. However, the traditional Mac OS
toolkits expect colons, so above the BSD layer, the core Carbon toolkit does yet another
translation. The result is that Carbon applications see colons, and everyone else sees
slashes. This can create a user-visible schizophrenia in the rare cases of file names
containing colon characters, which appear to Carbon applications as slash characters, but
to BSD programs and Cocoa applications as colons.
While OS X "is" a unix operating system, it also derives quite a bit its code, APIs, standards, etc from Mac OS 9. In unix, file paths have "/" separating the elements and ":" is allowed in the names of individual files and directories. In Mac OS 9, it was the other way around: file paths had ":" between elements and "/" was allowed in individual filenames. When Apple developed OS X, they wound up having to support some APIs that used unix-style file paths, and some APIs that used OS 9-style paths, and they had to both be able to work on the same filesystem.
What they did is to swap delimiters and allowed characters depending on context. If you write (/run) a program that uses unix APIs to access the filesystem, you'll see files with colons in their names and slashes separating path elements. If you write (/run) a program that uses the old OS 9 APIs (or their derivatives), you'll see files with slashes in their names and colons separating path elements. See Apple's developer Q&A #1392 and notes on specifying paths in AppleScript for a bit more discussion.
(There are some other differences as well. A unix path is absolute if it starts with the delimiter ("/"), and absolute paths start at the top of the root volume. An OS 9 path is absolute if it doesn't start with a delimiter, and absolute OS 9 paths start with a volume name. Thus, the unix path "/tmp/foo:bar" is equivalent to the OS 9 path "Macintosh HD:tmp:foo/bar".)
So, which character is really in the filename, a slash or a colon? Well, a filename is a rather abstract thing, but if you're asking about the bytes that're actually stored on the disk... if it's on an HFS+ (aka Mac OS Extended) volume, it's being stored in a filesystem that was designed to work with the OS 9 (well, technically Mac OS 8.1) APIs, so it allows slashes but forbids colons, so on an HFS+ volume the file will "really" have a slash in the name. OTOH if you store the file on a unixish volume, it'll be stored using the unix convention, and "really" have a colon in the name. But the difference doesn't really matter unless you're reading raw bytes off the disk or writing a filesystem driver...
Finally, why does the Finder display the controversial filename character as slash rather than colon? I'm pretty sure it's mostly inertia. The Finder isn't even entirely consistent about this, since if you use its Go To Folder option (Command-Shift-G) and type in "/Users/Shared", it treats that as a unix path. If you type in "Macintosh HD:Users:Shared", it has no idea what you're talking about. Furthermore, if you run touch /tmp/foo:bar, then try to get to it with Go To Folder:
Entering "/tmp/foo:bar" works.
Entering "/tmp/fo" then pressing tab autocompletes it to "/tmp/foo/bar/", which works.
Entering "/tmp/foo/bar/" fails, even though it's exactly the same as the autocomplete.
Entering "/tmp/foo" then pressing tab autocompletes to "/tmp/foo/", which cannot be autocompleted any further and doesn't work at all.
Update: as Konrad Rudolph pointed out, the Go To Folder behavior has changed as of El Capitan, and I there's no longer any way to use it to get to folders containing the controversial character.
To avoid as many problems as possible when dealing with File names, paths, and various OSes, you really should take advantage of the built-in File methods, like join, dirname, basename, extname, and split. They try to avoid system dependencies and try to give you a programmatic way to generate valid filenames cross-platform.
This problem was a lot worse back when Apple used the old Macintosh operating system. The move to Mac OS helped, because they dropped using : as a separator, however those people who were manually building filenames found code breaking because it generated the wrong delimiters, whereas taking advantage of the libraries handled the problem.
Because this particular problem isn't a bug, nor is it in Ruby's control but Apple's, I'd say it's not a Ruby problem at all, it's a visualization issue, and if you want the filename to resemble what the Finder displays code accordingly.
I have several level of directories in the folder path. when the path exceeds 256, then I could not create a sub-folder or file from it. Is there any chance to build paths more than this length.
Can anyone help me out.
In fact the limit on path strings is 260 characters. The underlying OS, these days, can support much longer path names, up to 32,767 characters. In order to name a path with a long name you need to use the magic \\?\ prefix, and use the Unicode version of the API.
However, many tools don't support such long names. A classic example of such a tool is Explorer which won't let you create objects with names longer than 260 characters. Because of this I strongly advise you to avoid creating such long names—doing so will save you much heartache in the long run.
This should get you started: http://msdn.microsoft.com/en-us/library/aa365247%28v=vs.85%29.aspx#maxpath
Sadly it's an issue that I don't think will be going away any time soon, so you'd do well to familiarize yourself with that stuff.
As an aside, if you have access to robocopy (comes packaged with Windows Vista and 7, but is also available for XP), which supports long paths, you could create your files/subfolders in a higher-up folder and then use robocopy to move the subfolder to its desired location deeper in the folder tree.
According to the documentation here http://msdn.microsoft.com/en-us/library/Aa365247, the maximum length is actually about 32,000, but most windows APIs still limit you to MAX_PATH which is 260. There are some unicode APIs that let you go beyond the 260 limit.
See here, http://msdn.microsoft.com/en-us/library/aa363856.
In the ANSI version of this function, the name is limited to MAX_PATH characters. To extend this limit to 32,767 wide characters, call the Unicode version of the function and prepend \\?\ to the path. For more information, see Naming a File.
This is an addendum to the answers above. I extracted only a summary to what I think is relevant, from Microsoft's official documentation:
Maximum Path Length Limitation
In the Windows API (with some exceptions), the maximum length for a path is MAX_PATH,
which is defined as 260 characters. A local path is structured in the following order:
drive letter, colon, backslash, name components separated by backslashes, and a terminating null character.
Example: "D:\some 256-character-path-string" -> 256
Using long paths
The Windows API has many functions that also have Unicode versions to permit an extended-length path for a maximum total path length of 32,767 characters.
To specify an extended-length path, use the "\?" prefix. For example, "\?\D:\very long path".
Relative paths
Relative paths are always limited to a total of MAX_PATH characters.
Enable Long Paths in Win10
Starting in Windows 10.1607, MAX_PATH limitations have been removed from common Win32 file and directory functions.
However, you must opt-in to the new behavior.
From Microsoft documentation:
https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file?redirectedfrom=MSDN#maximum-path-length-limitation
Warning for Delphi users:
There is a problem in IOUtils. It cannot be used in conjunction with Max_Path. It uses InternalCheckDirPathParam all over the place!
Details: TDirectory.GetDirectoryRoot does not handle correctly paths of Max_Path characters
After years of programming it's still some of the simple things that keep tripping me up.
Is there a commonly agreed definition of filename ?
Even the wikipedia article confuses the two interpretations.
It starts by defining it as 'a special kind of string used to uniquely identify a file stored on the file system of a computer'. That seems clear enough, and suggests that a filename is a fully qualified filename, specifying the complete path to the file.
However, it then goes on to:
talk about basename and extension (so basename would contain an absolute path ?)
says that the length of a filename in DOS is limited to 8.3
says that a filename without a path part is assumed to be a file in the current working directory (so the filename does not uniquely identify a file)
So, simple questions:
what is a correct definition of 'filename' (include references)
how should I unambiguously name variables for:
a path to a file (which can be absolute/full or relative)
a path to a resource that can be a file/directory/socket
No references, just vernacular from experience. When I'm being specific I tend to use:
path or filespec (or file specification): all of the characters needed to identify a file on a filesystem. The path may be absolute (starting from the root, or topmost, directory) or relative (starting from the currently active directory).
filename: the characters needed to identify a file within the current directory.
extension: characters at the end of the filename that typically identify the type of the file. By convention, the extension usually starts with a dot ("."), and a filename may contain more than one extension.
basename: the filename up to (but not including) the dot that begins the first extension.
Javadoc for File.getName() method
file·name also file name
(fīl'nām') Pronunciation Key n. A
name given to a computer file to
distinguish it from other files, often
containing an extension that
classifies it by type.
# Dictionary.com
It states that a filename is used to name a file, (just like you name a person). And that it's used to distinguish it from other files. This does not tell you it includes a path, or other file-system imposed attributes.
This definition does say that often a filename has an extension. But this definition is very careful... (Which I think is a good thing)
So.. before you start thinking about paths and such, you have to set your scope. Are you in a unix world? Ar you in a dos/windows world?
Again no references, but the file name specification depends on the operating system or to be more accurate the file system. Lets start with early versions of DOS (Disk Operating System). File names were 8 character names containing numbers, letters, dashes, and underscores. They were followed by a three, two, one, or even zero character extension used to identify the file type. A dot separated the name from the extension. The name had to be unique in the directory.
You could extend the name by adding a directory name, or series of directory names. a slash character separated the directory names from each other and from the file name. This was usually referred to as the path name. The path was relative to current directory.
Finally in DOS you could include the drive name. Usually a single letter followed by a : and a slash (some systems two slashes). Adding the drive to the path made it an absolute path instead of relative.
Today most of us use long file names which do not follow the old 8 character dot three character pattern. Still many files systems keep such as name and use the long name simply as a pointer to old style identifier.
We are using a proprietary application for inventory management and have discovered this application is unable to interpret spaces in file paths. For example:
C:\Google Drive\Invoices
Does not work, whereas
C:\Google\Invoices
does work.
Is there a special way to represent a space in Windows much like a URL string can use %20? For example C:\Google%20\Drive\Invoices.
Use 8.3 short name.
Try dir /x c:\
Google Drive should have a short name, probably like GOOGLE~1
Then you can use C:\GOOGLE~1\Invoices
You can use short names if supported. Type dir /x and it's in the middle column.
However it only works if short names aren't turned off. If short names aren't available the only way is making a junction point or a symbolic link1.
Run cmd as admin and type either of the following
mklink /J C:\ggdrive "C:\Google Drive"
mklink /D C:\ggdrive "C:\Google Drive"
This will create a link from ggdrive to the real Google Drive folder and now you can access Google Drive as ggdrive
However it's highly probable that you've used the path incorrectly. In some places you need to quote paths with spaces like this "C:\Google Drive\Invoices". But if an application in the last 15-20 years doesn't support long file names then it is rubbish anyway. Use a better program or report to the developer to fix it.
1 The differences between them is like this
What is the difference between NTFS Junction Points and Symbolic Links?
“directory junction” vs “directory symbolic link”?