I have a bunch of files in a Git repo which works okay on macOS and Linux, but will fail on Windows because the file name contains ?
I was wondering how do I configure Git only on my local Windows machine so that I can check it out and have it auto convert the file to something Windows will allow and push it back preserving the file name?
The best way to do this is to use the Windows Subsystem for Linux. It has a fully POSIX-compatible file system and is fully capable of storing arbitrary byte values (with the normal exceptions).
It is almost certainly not going to be possible to use Git for Windows on a native Windows file system, although if you happened to format an external hard disk as UDF (which, I believe, requires the full disk, not just a partition, to be formatted that way on Windows), then you could probably check it out there. UDF is at least capable of handling these characters on Unix.
The title pretty much says it, on Windows, can I somehow simulate multiple working directories in a multithreaded application by using something similar to openat?
See also this question.
Yes there is, and it's called NtCreateFile() (https://msdn.microsoft.com/en-us/library/bb432380(v=vs.85).aspx) :)
openat() takes an open fd to a base directory from which path operations start. Similarly, you can supply a HANDLE to the ObjectAttributes.RootDirectory of NtCreateFile() to have whatever that directory's path is to be used as a base for relative path operations.
If going direct to the NT kernel API is too much for you and you want to remain within Win32, you can retrieve the current path of the base directory HANDLE easily enough (see https://msdn.microsoft.com/en-us/library/windows/desktop/aa366789(v=vs.85).aspx). If you open the directory without FILE_SHARE_DELETE permissions, nobody will be able to rename it and therefore the path retrieved will never move so long as you keep the HANDLE open.
You then stitch together the retrieved path with the relative path using normal string concatenation.
The NT kernel API approach is the only solution on Windows which allows the base directory HANDLE to be arbitrarily renamed by third party processes and that not cause races in your code. i.e. it's the only equivalent to POSIX openat(). I agree it's very unfortunate that Win32 doesn't make available this facility - in fact, Win32 doesn't make available atomic renames which is a very useful POSIX facility indeed to let you avoid lock files and is something the NT kernel API provides too.
I've got serious troubles with Windows 7 not having a case-sensitive file system.
I'm the only developer who's working on Windows in my team (and we wanna keep it that way for cross-OS checks), and I'm the one producing occasional errors due to Windows not crashing on files with different cases.
Is there any way to make Windows go crazy when I try to include 'aSdF.php' where I meant 'asdf.php'?
From a programming viewpoint, you get case sensitivity on Windows by specifying the FILE_FLAG_POSIX_SEMANTICS flag when you call CreateFile.
You do just about have to call CreateFile directly to do that though -- a typical standard library (e.g., if you use fopen in C or std::ifstream::open in C++) will not pass that flag, so file names will be treated as case insensitive.
Linux operating system identifies files by looking at its magic number at the starting of the header. How does windows do it ? Does it also have some kind of magic number mechanism or does it only rely on the file extension ?
It relies only on the extension, as provided by the filesystem; the contents of the file are not examined. See e.g. this article - it talks about Windows XP, but AFAIK the general behavior is shared by all released versions of Windows so far: http://support.microsoft.com/kb/307859
Linux has a feature called namespaces, which let you give a different "view" of the filesystem to different processes. In Windows terms, this would be useful for example if you had a legacy program "floyd" that always loaded its configuration from C:\floyd\floyd.ini. If Windows had namespaces, you could write a wrapper script which would create a namespace in which to run floyd, making it so when Alice ran the script, floyd would start up in an environment where C:\floyd existed but actually pointed to C:\Users\Alice\Floyd.
Now you may be thinking, "OK, just use soft or hard links and make C:\floyd an alias for C:\Users\Alice." But with namespaces, Bob can also run the startup script, but his instance of floyd (on the same computer, running at the same time) will see C:\floyd with the contents of, say, C:\Users\Bob\Program Settings\Floyd Config (or any other path we like).
You can do this on Linux with namespaces. Is there something similar or analogous on Windows? It's fine if it requires writing a C program, and it's OK if it only works on recent versions of Windows.
NTFS hard links are really a simple case of reparse points. Reparse points are typed, and can include more advanced behavior. For instance, they're also used for "offline storage" (transparent migration of files to and from secondary storage). You can therefore also use reparse points to implement per-user symbolic links, by creating a new reparse type.
The reparse point type even has an explicit "Name surrogate" bit, which (if set) indicates that reparse points of those types are some kind of symbolic link.
You can even have multiple reparse points in a path. Therefore files inside your symbolic namepace can still be migrated to secondary storage - you'd just have two reparse points in the path.
I think Virtual Store does this automatically for legacy programs that try to write to nonstandard directories. So the legacy program writes to a user- and program-specific directory instead to C:\floyd.
This sounds like Windows Vista's file system virtualization. For example, it can silently redirect c:\Program Files\Floyd to c:\Users\<username>\AppData\Local\VirtualStore\Program Files\Floyd. However, file system virtualization isn't nearly as configurable as Linux namespaces. From what I can tell from reading, file system virtualization should apply any time a 32-bit interactive process opens for writing a file, folder, or registry key that's only writable by administrators. (So you typically end up with some read-only files under c:\Program Files and some per-user writable files under c:\Users\<username>\AppData\Local\VirtalStore.)
An application virtualization product can probably also do this, although those are often more complicated and more expensive.
You can use hardlinks for that, but only with NTFS. http://en.wikipedia.org/wiki/Hard_link
I think windows doesn't have virtual FS view per process.
The most relevant thing is probably special environment folders, such as %temp%, %appdata%, %localappdata%. Not that they're equivalent, but they fulfil the same purpose.
You can define your own environment variables then use '%myspecialplace%\myfile.txt' to access them.
As a horrid kludge (and here I book my passage to programming hell) could you use a NamedPipe for C:\Floyd that mapped the IO operations onto a file specific to the current process user?
I know it's not pretty and I don't know enough about NamedPipes (FIFOs in other dialects) on Windows to know how feasible this is.
Dan
There are several things that come to my mind.
First of all you can create a file system filter driver (or use a ready driver, such as our CallbackFilter product) that would redirect all file system calls, coming from the application, to other location. This is close to virtualization that you mentioned but this won't change the list of drive letters though. Such approach is both powerful and non-trivial, so see the other option first of all.
And the other option is:
there exist several products (Thinstall, Molebox if memory serves) that "wrap" the application redirecting it's file I/O to some other location. There was also some SDK to do the same, but I don't remember it's name at all.
e.g. http://www.msigeek.com/4819/file-re-direction-using-correctfilepaths-shim-to-fix-broken-applications
But I think it's not configurable per user, although the target can vary per-user based on environment variable substitution.
Most programs, though, store configuration in the registry, in which case RegOverridePredefKey would do the trick.
There's a shortage of good solutions for this. For simplicity, I can't improve on using NTFS soft links (junctions) for this - as you correctly point out, this creates issues if you want per-user configuration. As MSalters correctly says, all NTFS soft and hard links are just special cases of reparse points, so you could do something more general by impplementing a new reparse type, if you don't mind some work digging into NTFS..
(Junction is a pretty useful tool when experimenting with NTFS soft links: http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx )
You could just take a direct approach - give each user (or your program initialization if you only care about one piece of software) a logon script that sets up the appropriate junction into their user directory (and make sure you clean it up afterwards). But it's clumsy.
In general the right Windows approach is to put things into the folders pointed at by the %localappdata% (from Vista on) and, more generally, %userprofile% system variables.
Win file system virtualization is intended to ensure this in the cases where it applies.