I need to be able to store configuration files on machines that gets disconnected by plugging off the electricity ;), I'm using basic WinApi to store configuration data (WriteFile), this works unless the machine is plugged off ;), sometimes file isn't saved at all.
I was thinking of 2 solutions:
1) Transactional NTFS API (eg. CreateFileTransacted() ), but this thing works only on Vista and NTFS has to be present and I can't use it in most cases
2) To create a backup copy of configuration file in %APPDATA% directory, say 20 backup copies and restore them on application startup when damaged configuration file is detected
If you know of any other solution to my problem (the main problem is turning off the machine by plugging it out), please let me know. Thank you.
You don't really need 20 backup copies. You only need 1 - the last copy. Now, if your client actually asks for a basic versioning system for config files that is another story. But just to have a good config file you only need 1 backup.
Now, here's what I used to do for my embedded projects:
Calculate a hash of the config file and store it in the file. The easiest is to append it to the end of file as a comment. I used to use crc32 for this but these days I would use SHA1. This can even be done automatically by a config-uploader tool just prior to transmitting/storing the config file.
When opening the config file, extract the hash and compare it with the value calculated from the file (obviously calculated after the hash is removed from the file). If the hash is not there then the file is incomplete. If the hash is not the same then the file is corrupted. In either case use the older file.
Now that a valid & correct config file is verified it can replace the older config file. Use the OS's rename operation for this. It is usually atomic on most modern filesystems so a failed rename will not clobber up the old file.
This is the most robust system I've used in my years of experience. It's basically what bittorrent does.
Related
I am looking for a bit of advice on how Windows file system differentiates between files that are copied(copy and pasted from another location) and files that are created (a new file created in a a folder).
A bit of background to this so it makes more sense: I have an application that is used to move files. The application will monitor a directory and when a file is placed in the directory it will move it elsewhere. However, I am having issues where the application will not pick up a file that is created within the monitored directory but will pick up files that have been created else where and are copied into the monitored directory.
Any advice on how Windows differentiates, or if it does at all, would be greatly appreciated.
This is running on Microsoft Windows Server 2008 R2 Standard. I can't dig into the code and see what is going on under the hood unfortunately, so need to get an idea of the difference if any there would be.
The filesystems don't know the operation of "copying" the file. Any copying is a sequence of file open/read/write/close operations. The same applies to moving to the different filesystem. Moving within the same filesystem, though, is an operation native to the filesystems and it can be done with one command to the filesystem.
Now about your problem. Most likely you catch the creation of the file (before the data is written), and when your application reacts, the file is still opened for writing. So you need to wait until the file is closed.
Depending on how you do monitoring, such waiting is done in different ways. In filesystem filters you wait for file close operation. With .NET FileSystemWatcher there's no way to track file close operation, but I saw a couple of tricks here on StackOverflow (don't have a link though, sorry).
A file existing in D: drive, from creation
The same file which was copied to E: drive
As you can see, the file which was copied to E: drive, has a creation time as the latest, when it was copied to and the modification time as the last modification time for that file in previous location.
So I guess this illustrates, how windows differentiates between copied files and created files.
Is it possible to append/remove a ressource file to a binary at execution time?
I have an application written with go, which saves/searches data from a database file, and i would like this database file to be embedded to the binary, and updated by the application itself.
This way the application would be self contained with its database.
Modifying the executable, this is generally a very bad idea.
Several issues pop right into my head, such as:
Does the current user have sufficient permissions?
Is the file locked during execution?
What about multiple running instances of the application?
Even if you manage to do just that, think of what anti-virus and firewall applications will say to it: most when they detect the change will flag the executable and/or contain it, or deny running it, or some may even delete it. Rightfully, as this is what many viruses do: modify existing executables.
Also virus scanner databases maintain reports where files (their contents) are identified based on the hash of their content. Modifying the executable will naturally change the file content hash, thus render the file unknown / suspicious to these databases.
As mentioned, just write / cache data in separate file(s), preferably in user's home folder or in the application folder (next to the executable, optionally in sub-folders). Or make the cache file / folder a changeable option (command line flags).
Technically, this is possible, but this is a bad idea. Your application could be run by users not having write permissions to your binary.
If you're talking about a portable app, your best option might be using a file in the same directory the binary is located, otherwise - use the user's home directory according to the conventions of the OS you're running on. You can use the os/user package to find the home directory.
I'm looking for a editor in windows that constantly saves file.
In linux I do
cat>somefile
and then just start typing. somefile gets filled up as I type.
Is there editor or similar thing in WIndows? Preferably a non-dos tool?
I use WebStorm from JetBrains, which saves constantly and unobtrusively.
I really love it. I use it as a text editor and for my web development.
http://www.jetbrains.com/webstorm/
(and no I don't work there).
Its possible to install some unix features to windows.
Have a look at this CoreUtils
The shareware text editor UltraEdit by default works with using a temporary file which means create a copy of the file to edit in directory %TEMP% and copy this temporary file on save back to original file. The usage of a temporary file makes it possible to use Undo and Redo.
But it is possible at Advanced - Configuration - File Handling - Temporary Files to disable the usage of a temporary file for all files or just for large files depending on a threshold value in KB. All edits made on a file opened without usage of a temporary file are permanent which means immediately done on storage media.
Another feature of UltraEdit is automatic save in regular intervals which can be configured at Advanced - Configuration - File Handling - Save with or without making a backup on every save and even supporting version backups which means backups with an incrementing number on every save.
Last but not least on usage of a temporary file for editing a file as by default UltraEdit can recover last edits if UltraEdit crashes (uedit32.exe process killed with Windows task manager), or Windows crashes, or a sudden power loss occurs. The temporary file is updated quite often in the background by UltraEdit and therefore the restore on next start after an unexpected end of the editing session often restores nearly all edits made last on a file. The recovery feature includes also new files not being saved ever as file with a file name.
It would be interesting to know for offering a perhaps better solution why you want that any edit is immediately written to the file. In general this is the opposite of what users want on editing a text file and is not good on some storage medias like SSD hard disks.
I'm working on an application that generates a set of bitmaps and then loads them into a form for a user to pick from.
The bitmaps are generated from a small vector library which the user can add to. The code now creates the files and then deletes them immediatelyafter use, only to have to generate them again (making the UI take seconds to load) next time the user opens the UI.
So what I'm wondering is, is it okay to leave my bitmaps in the user temp folder "forever", and regenerate them if they are not in the folder? I can't expect to be able to store the images in the application directory, due to possible permission issues, and like I said, I can't prepopulate the files since the user can add more.
Ideally you should generate any temporary data into the RAM rather to the file system.
It is acceptable to depend on temporary files if you can make sure that your application is storing only a limited amount of such files per user. Any temporary files can be left behind on unexpected crashes/power offs no matter what your code does. You therefore need to implement a mechanism that will delete any stale files created by the same application in a previous session - presumably during its next start up.
Assuming such a safety mechanism, intentionally leaving behind temporary files when the application exits sounds like a non-standard but reasonable "cache".
Caveat: the next version of your application may need a slightly different file format, and should detect, delete and regenerate any files in a mismatched format based on some simple versioning scheme to avoid cross-build dependences.
I'm updating some legacy code to more efficiently use .ini files for parameter storage. Basically, it caches all the settings internally once they're read so that they can be read again without opening the file. Before using the cache, I compare the cache creation time to the last modified time of the file, and recreate the cache with the updated file. This works great, except when users rename files, because that action doesn't update the last modified time.
So if I copy my app.ini to app - copy.ini, modify app - copy.ini outside of my program, then delete app.ini and rename the copy to app.ini, my program is now using an outdated cache. Even if my program caught the deletion of app.ini and cleared the cache, it would not rebuild the cache when the copy was renamed. The program is designed to run for a very long time unattended, so I would like to avoid continually monitoring the file - only check it once I need to read a parameter.
What you are needing is a filewatcher for vb6, you could try it by interop with .net or use something like this VB6 Implementation
For that scenario, you could keep a hash of the contents of each .ini file the program is using and periodically check each file. If the hash is different than what it was last time you checked, then it's contents were changed or it's a different file caused by this rename scenario.