Does Windows offer anything like POSIX stat's st_ctime field, which gives the timestamp of last metadata change?
Background: I have some data files that I'd like to check if they've been modified since a particular timestamp. Checking the modified timestamp (mtime) is easy and takes care of "normal" modifications, but if the user copies over older versions of the data files, then the modified timestamp will show that they're older. On a POSIX system, copying over an older file would result in ctime being newer, even if mtime is older.
As far as I can tell, Windows does provide a "last changed" field, but I haven't found any Win32 API for accessing it, so you have to use the Native API.
Specifically: (I haven't actually tried this.)
Call NtOpenFile to get a handle.
Call NtQueryInformationFile with a FileInformationClass parameter of FileNetworkOpenInformation to get a FILE_NETWORK_OPEN_INFORMATION struct.
The ChangeTime member of the FILE_NETWORK_OPEN_INFORMATION struct is equivalent to the POSIX ctime.
Using the Native API isn't exactly straightforward. This question and answer describe how to do it.
Cygwin takes advantage of this to provide POSIX semantics under Windows, as discussed on their mailing list. I'm getting my information from their implementation.
And although it's only tangentially related, this article has a good description of the "created" timestamp that Windows does present (and unfortunately refers to as ctime). The created timestamp can be newer than the modified timestamp if a file is copied to a new location (since created then refers to when the copy was created, while modified refers to when the original was last modified), but it otherwise is not updated when metadata is changed.
Related
In my Synology NAS, I have an APFS share with files that have been transferred back and forth for decades across different OSes.
original systems: probably ext4 filesystem and Synology-hosted NFS mount, years ago (various systems, Linux/Windows)
current system: EXT4 filesystem, with Synology-hosted AFP mounts (to a macOS 10.15 system, though I doubt that matters)
For files that were copied via NFS originally, and now hosted via AFP, all the file dates seem to be offset by some amount. I can sort of eyeball the datetime offset, but is there a definitive number I can use? (And a simple way to parse/modify the times using something like GetFileInfo?)
For reference, I have a copy of iTerm2-3_2_6.zip, dated "2039-01-22 08:25:17". I would probably map that to 2019-01-21 (release date for 3.2.7), implying a 20-year offset.
The closest thing I can think of is macOS epoch starting on 2001-01-01 instead of UNIX 1970-01-01, but that's a 30-year offset.
There's also the "year 2038 problem", and some software might be doing something clever with 32-bit overflows to support post-2038 datetimes, but I have at least one file dated "2031-08-10", so that seems unlikely.
This seems to be something related to 32-bit and 64-bit overflows, somewhere in this complicated storage stack; the way the datetimes + error offsets add up is consistently close to 2^31, though I wasn't able to find any clear pattern.
Also, I noticed strange behavior from my Synology system's use of eCryptFS, which seems to lag metadata updates when done in batch. (In particular, I suspect that some eCryptFS/Synology metadata is getting translated incorrectly, or just never written to disk.)
Anyway, I basically wrote a Python script that does the following:
check if os.stat() reports an atime/mtime newer than 2030
check that both atime and mtime are newer; error out if they differ
adjust both times back by 632599096 seconds (offset based on comparing copies of the one file I found in common between two systems).
with the following caveats to watch out for:
macOS's GetFileInfo/SetFile utilities only support 32-bit datetimes, so you should generally avoid using them (even just to verify the metadata updates).
something in the Synology/eCryptFS encryption gets very slow, so after a few dozen metadata updates, the updates will no longer be visible (even after calling sync from the shell). But if you give it some time, you'll see the corrected atime/mtime changes.
The OS-specific os.stat field, ctime, does literally track metadata update times. And there doesn't seem to be a way to manually set it (nor a need to, since this isn't visible through any GUI).
The combination of slow metadata updates + GetFileInfo reporting the wrong time made this incredibly frustrating, until I figured both out. In practice, this means you have to test metadata updates on a few files at a time, then your large batch execution can only be verified a few hours later (I waited a day).
This should have been a blog post, good riddance.
How do I check whether a particular File(identified by fileID) in my google drive using API whether its modified or not ??
I would do a file.get which will return the file.resource
File resource contains a lot of info about the file.
version long A monotonically increasing version number for the file.
This reflects every change made to the file on the server, even those
not visible to the user.
modifiedTime datetime The last time the file was modified by anyone
(RFC 3339 date-time). Note that setting modifiedTime will also update
modifiedByMeTime for the user.
There are a bunch more. I guess its going to depend upon how you define been modified. Are you storing last the last time you modified it someplace? Do you just want to see if someone else changed it since you last changed it?
I want to write a program that allows or blocks processes while openning a file depending on a policy.
I could make a control by checking the name of the program. However, it would not be enough because user can change the name to pass the policy. i.e. let's say that policy doesn't allow a.exe to access txt files whereas b.exe is allowed. If user change a.exe with b.exe, i cannot block it.
On the other hand, verifying portable executable signature is not enough for me, because i don't care whether the executable signed or not. I just want to identify the executable that is wanted to execute even its name is changed.
For this type of case, what would you propose? Any solutions are welcome.
Thanks in advance
There are many ways to identify an executable file. Here is a simple list:
Name:
The most simple and straightforward approach is to identify a file by its name. But it is one of the easiest things to change, and you already ruled that out.
Date:
Files have an access, creation, and modification date, and they are managed by the operating system. They are not foolproof, or maybe not even accurate.
Also, they are very simple to change.
Version Information:
Since we are talking about executable files, then most executable files have version information attached to it. For example, original file name, file version, product version, company, description, etc. You can check these fields if you are sure the user cannot modify them by editing your executable. It doesn't require you to keep a database of allowed files. However, it does require you to have something to compare to, like company name, or a product name. Also, If someone made an executable with the same value
they can run instead of the allowed one and bypass your protection.
Location:
If the file is located in a specific place and is protected by file system access rights, and it cannot be changed, then you can use that. You can, for example, put the allowed files in a folder where the user (without admin rights) can only read/execute them, but not rename/move. Then identify the file with its location. If it is run from this location, then allow it, else block it. It is good as it doesn't need a database of
allowed/blocked files, it just compares the location, it it is a valid one, then allow, and you can keep adding and removing files to allowed locations
without affecting your program.
Size:
If the file has a specific file size, you can quickly check its size and compare it. But it is unreliable as files can be changed/patched and without any change in size. You can avoid that by also applying a CRC check to detect if the content of the file changed.
But, both size and CRC can be changed. Also, this requires you to have a list of file names and their sizes/CRC, and keeping it up to date.
Signature:
Deanna mentioned that you can self-sign your executable files. Then check if the signature matches yours and allow/deny based on that. This seems to be a good way if
it is okay for you to sign all the executable files you want to allow. It doesn't require you to keep an updated list of allowed files.
Hash:
arx also pointed out, that you can hash the files. It is one of the slowest methods, as it requires the file to be hashed every time it is executed, then compared to a list of files. But it very reliable as it can uniquely identify each file and hard to break. But, you will need to keep an up to date database of every file hash.
Finally, and depend on your needs and options, you can mix two or more ways together to get the results you want. Like, checking file name + location, etc.
I hope I covered most of things, but I'm sure there are more ways. Anyone can freely edit my post to include anything that I have missed.
I would recommend using the signature, if it has one, or the hash otherwise. Apps such as Office that update frequently are more likely to be signed, whereas smaller apps downloaded off the Internet are unlikely to ever be updated and so should have a consistent hash.
I have a program (jhead) that compiles with very few tweaks for both Windows and generic Unix variants. From time to time, windows users ask if it can be modified to also set the "creation date/time" of the files, but I don't see a way to do this with the POSIX api. What I'm currently doing is:
{
struct utimbuf mtime;
mtime.actime = NewUnixTime;
mtime.modtime = NewUnixTime;
utime(FileName, &mtime);
}
Ideally, struct utimebuf would just have a creation time, but it doesn't. It strikes me it would take a lot of windows specific, non-portable code to change the creation time under Windows. Is there another POSIX way of doing this? Any suggestions?
POSIX only recognizes three different file times:
atime (access time): The last time the file was read
mtime (modification time): The last time the file was written
ctime (attribute change time): The last time the file's metadata was modified
Any other file times that may exist in the underlying OS require OS-specific API calls in order to be modified.
And don't worry about creating non-portable code; only these times really exist under most *nix variants.
The Win32 API for this isn't really all that bad, as Windows APIs go: https://msdn.microsoft.com/en-us/library/windows/desktop/ms724933%28v=vs.85%29.aspx . The trickiest thing is to work out how many seconds Windows thinks there were between 1st January 1601 and 1st January 1970; the rest is straightforward.
I have a program that uses save files. It needs to load the newest save file, but fall back on the next newest if that one is unavailable or corrupted. Can I use the windows file creation timestamp to tell the order of when they were created, or is this unreliable? I am asking because the "changed" timestamps seem unreliable. I can embed the creation time/date in the name if I have to, but it would be easier to use the file system dates if possible.
If you have a directory full of arbitrary and randomly named files and 'time' is the only factor, it may be more pointful to establish a filename that matches the timestamp to eliminate need for using tools to view it.
2008_12_31_24_60_60_1000
Would be my recommendation for a flatfile system.
Sometimes if you have a lot of files, you may want to group them, ie:
2008/
2008/12/
2008/12/31
2008/12/31/00-12/
2008/12/31/13-24/24_60_60_1000
or something larger
2008/
2008/12_31/
etc etc etc.
( Moreover, if you're not embedding the time, what is your other distinguishing characteritics, you cant have a null file name, and creating monotonically increasing sequences is way harder ? need info )
What do you mean by "reliable"? When you create a file, it gets a timestamp, and that works. Now, the resolution of that timestamp is not necessarily high -- on FAT16 it was 2 seconds, I think. On FAT32 and NTFS it probably is 1 second. So if you are saving your files at a rate of less then one per second, you should be good there. Keep in mind, that user can change the timestamp value arbitrarily. If you are concerned about that, you'll have to embed the timestamp into the file itself (although in my opinion that would be ovekill)
Of course if the user of the machine is an administrator, they can set the current time to whatever they want it to be, and the system will happily timestamp files with that time.
So it all depends on what you're trying to do with the information.
Windows timestamps are in UTC. So if your timezone changes (ie. when daylight savings starts or ends) the timestamp will move forward/back an hour. Apart from that, and the accuracy of about 2 seconds, there is no reason to think that the timestamps are invalid, and its certainly ok to use them. But I think its bad practice, when you can simply put the timestamp in the name, or in the file itself even.
What if the system time is changed for some reason? It seems handy, but perhaps some other version number counting up would be better.
Added: A similar question, but with databases, here.
I faced some issues with created time of a file after deletion and recreation under same name.
Something similar to this comment in GetFileInfoEx docs
Problem getting correct Creation Time after file was recreated
I tried to use GetFileAttributesEx and then get ftCreationTime field of
the resulting WIN32_FILE_ATTRIBUTE_DATA structure. It works just fine
at first, but after I delete file and recreate again, it keeps giving
me the original already incorrect value until I restart the process
again. The same problem happens for FindFirstFile API, as well. I use
Window 2003.
this is said to be related to something called tunnelling
try usining this when you want to rename the file
Path.Combine(ArchivedPath, currentDate + " " + fileInfo.Name))