Summary: DAO is trying to automatically open a nonexistent file - can this be prevented?
I maintain a VB6 program which uses the DAO36 library for accessing MDB / Jet databases. (Yes, pretty old stuff).
We noticed that during application startup it has been looking for file system.mdb which does not exist. This is not anything our own code was initiating.
Using Process Monitor I was able to determine that the program looks for this file around the time that the DAO library is loaded or initialized. After a bunch of DAO/Jet DLLs are loaded, one of them looks in the registry here:
HKLM\SOFTWARE\WOW6432Node\Microsoft\Jet\4.0\Engines\SystemDB
the value of which is indeed system.mdb. I can see the same in RegEdit:
Having located the desired filename from the registry it tries to open that file in the working folder of the program. i.e., I can see it try to find a file called:
C:\Program Files (x86)\<appname>\system.mdb
which of course does not actually exist.
I looked on Microsoft's website and also through older MSFT KB articles (1, 2), as well as here on SO. It seems that system.mdb is (or used to be) tied into how DHCP and WINS networking was configured in Windows NT. (e.g., refs 1, 2, 3 ... and many other KB articles indicate this).
This is the kind of 'red flag' that sometimes it is good to be proactive about before it leads to some (often rare and unreproducible) problem for an end user. So if there is any way to properly disable or configure this behavior I'd like to find out about it.
Related
Starting from Windows 10 Fall Creators Update (version 16299.15) and OneDrive build 17.3.7064.1005 the On-Demand Files are available for users (https://support.office.com/en-us/article/learn-about-onedrive-files-on-demand-0e6860d3-d9f3-4971-b321-7092438fb38e)
Any OneDrive file now can have one of the following type: online-only, locally available, and always available.
Using WinAPI how can I know that the file (e.g. "C:\Users\Username\OneDrive\Getting started with OneDrive.pdf") is online-only file?
After years, I'm still using FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS attribute described here to determine if a file or a directory is completely present locally or not.
Microsoft docs says the following for FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS:
When this attribute is set, it means that the file or directory is not fully present locally. For a file that means that not all of its data is on local storage (e.g. it may be sparse with some data still in remote storage). For a directory it means that some of the directory contents are being virtualized from another location. Reading the file / enumerating the directory will be more expensive than normal, e.g. it will cause at least some of the file/directory content to be fetched from a remote store. Only kernel-mode callers can set this bit.
There are some advantages of FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS:
It can be used for both files and directories.
It can be set in kernel mode only, so there is no chance for anyone to set the attribute arbitrary.
And as it described in this answer, there are still some interesting undocumented attributes which can provide additional information about cloud files.
Note: I didn't accept Jonathan Potter's answer because I mentioned FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS attribute in comments and started using it a year earlier than he updated his answer.
To check for "online only" all you need is to call GetFileAttributes() and see if the FILE_ATTRIBUTE_OFFLINE attribute is set.
In fact this isn't new for OneDrive, that attribute has existed for a long time.
There are other OneDrive attributes available via the shell (although the property you need is PKEY_StorageProviderState rather than PKEY_FilePlaceholderStatus) but "online only" is easy to check for.
Edit: Another filesystem attribute, FILE_ATTRIBUTE_PINNED is new for Windows 10, and is used by OneDrive to indicate a file that's "always available".
Edit: As of 2019 it appears that OneDrive now uses FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS rather than FILE_ATTRIBUTE_OFFLINE, as suggested below.
Edit: PKEY_StorageProviderState was broken in Windows 10 1903, and still not fixed in 1909. It returns 4 ("uploading") for all files in any apps other than Explorer.
Take a look at the PKEY_FilePlaceholderStatus property for the file (at the shell level, not the file-system level). This blog post has a example program you can test. This question also hints to some undocumented properties you might want to take a look at.
Microsoft has a UWP example on MSDN.
I'm making a simple VB.net application, which basically asks the user for multiple files and later it will need to access the selected files and modify them.
Right now, I'm saving the full paths of the selected files, and in the future, the application will iterate through each path, open the file from such path, and modify it.
The problem with that is that the user could select a file (so the full path is saved) and then they delete or move the file before my application modifies it.
Normally, I'd throw an error saying "File not found", but I'm under the impression that Windows had a feature that would disallow you from deleting/moving/renaming a file because "a program was using it" - which is a feature that would fit way better for my application.
I'm not very advanced with VB.NET, but I suppose that if I "open" a file using my application (with some IO thing), the feature I mentioned earlier would indeed trigger and the user would be unable to modify the file because it is "opened" by my application.
However, since my only desire is to "reserve" files, it seems to be quite wasteful to actually open them when I don't really need to (yet). Is there a way to tell Windows I need a certain file to be intact?
Opening files (with specifying desired sharing mode) is the way to do that.
I don't believe there is anything really wrong with opening multiple files (also you still will not be able to do anything for cases like removing of removable drive). In old times there were restrictions on number of opened files per process, but I it no longer practical limitation - Pushing the Limits of Windows: Handles
There is an easy solution: open each file in exclusive mode.
It should look like this:
Sub test()
Dim FS = System.IO.File.Open("path", IO.FileMode.Open, IO.FileAccess.ReadWrite, IO.FileShare.None)
End Sub
But beware: You have opened a file handle and if you code responsible for closing files fails without terminating the application files will still be locked for very long (till app shuts down).
You can use a using clause or a try/catch/finally clause - I don't know enough about your program to recommend anyone.
Every piece of documentation I've found (references 1 through 5) talks about setting up a symbol server by using a shared UNC path, and then putting the correct settings available to the local debugger instance (whether _NT_SYMBOL_PATH or the Visual Studio IDE Debugging settings).
Microsoft provides a symbol server (reference 6) available via http for their public symbol stores.
I want to create, for my own code, a symbol server accessible over http transport, instead of over UNC file sharing. The Mozilla folks appear to have done so (reference 7), but it is no longer functional.
Are there better references available for performing this task than I have found so far?
References
https://msdn.microsoft.com/en-us/library/b8ttk8zy(v=vs.80).aspx
http://msdn.microsoft.com/en-us/library/ms680693(v=vs.85).aspx
http://stackhash.com/blog/post/Setting-up-a-Symbol-Server.aspx
http://entland.homelinux.com/blog/2006/07/06/…
http://msdn.microsoft.com/en-us/windows/hardware/gg462988
http://support.microsoft.com/kb/311503
http://developer.mozilla.org/en/Using_the_Mozilla_symbol_server
I believe the answer is a very simple, "Just share the directory via some sort of http path." According to Chad Austin's entry on "Creating Your Very Own Symbol Server", this will just work.
In other words, the directory which symstore.exe uses to store the symbols, when served up as http://symbols.example.com/public_symbols/ , will be usable as the symbol server target for the Windows Debugging Tools.
Be careful when having multiple users use Symstore.exe directly against the same symbol store. Microsoft's white papers on this subject make it sound like you simply create a share and have everyone update through the SYMSTORE.EXE program delivered as part of Debugging Tools for Windows. The white papers advised you to have this done by each build.
And it works great with single users or when funneling all updates through a single person who is updating the symbol server for a team.
Unfortunately, the "fine print" at the bottom of some of the white papers says that only one user running symstore.exe can update the shared symbol server at the same time without breaking the content.
(Example: At http://msdn.microsoft.com/en-us/library/ms681417(VS.85).aspx, Microsoft says: "Note SymStore does not support simultaneous transactions from multiple users. It is recommended that one user be designated "administrator" of the symbol store and be responsible for all add and del transactions.")
So there is no inherent mechanism to serialize updates to the symbol store. It appears that multiple, simultaneous attempts to update the symbol store can break the symbol store and/or its index.
We cannot have builds for our entire multi-thousand man, international corporation in all time zones dependent upon coordination thru one man in one location.
Based on those white papers, I raised this issues with Microsoft in March of 2009; who confirmed this was a possible issue. After that discussion, we chose to implement a symbol update service which serializes the updates via direct Windows Debugging Tools SDK DbgEng.DLL SymbolSrvStoreFile() API calls so there is never a possibility of two simultaneous updates against the same area of symbols at the same time. Users have a build action that queues their symbols through the service instead of directly updating the symbol store. The service then serializes the updates to make sure true concurrent update attempts never happen.
The limited documentation available about using SymSrvStoreFile was not very clear at the time. I did get it working. Hopefully it has been improved since then. if not, the most crucial issue was the that the input path must be specified in a format similar to _NT_SYMBOL_PATH. So instead of, for example, using "C:\Data\MyProject\bin" as the input path, you would instead specify "srv*C:\Data\MyProject\bin".
Our service now also logs the updates through a database. The database both serves as a backup to the symbol store (in case it ever gets corrupted and must be rebuilt) and also creates a reporting point so that managers and support people know who is actually saving their symbols and who is not. We generate a weekly "symbol check-in" report which is auto-EMailed to stakeholders.
A symbol server served via HTTP has the same structure as a symbol server served via a UNC file path, so the simplest thing to do would be to use symstore.exe to store the files in a folder somewhere and then use a simple HTTP server which exposes that folder via HTTP (even running python -m SimpleHTTPServer in the symbols dir would work).
A small gotcha is that if a symbol file does not exist, the HTTP server must return a 404 error code (tested under Visual Studio 2013 at least). I ran into an issue where an HTTP server returning 403 for missing files caused Visual Studio to stop making requests after the first failed request.
symstore.exe creates a number of auxilliary files and folders (the 000Admin/ folder, refs.ptr and files.ptr files). None of these are needed for the symbol server to work.
If you want to create a symbol store without using symstore.exe, you can upload the files with this structure:
BinaryName.pdb/$BUILD_ID/BinaryName.pdb
BinaryName.exe/$LINK_ID/BinaryName.exe
Where BUILD_ID is a GUID embedded in the PDB file and executable and LINK_ID is a combination of build timestamp and file size in the executable. These can be obtained by reading the output of the dump_syms.exe tool from the breakpad library. See http://www.chromium.org/developers/decoding-crash-dumps
Our (Mozilla's) symbol server works fine, AFAICT. We're not doing anything particularly complicated, we just put the PDB files into the right directory structure (we have a script for that, but you could use symstore.exe) and serve it up via Apache. I think the only special thing we have are some Rewrite rules to allow accessing the files in a non-case-sensitive manner, because Microsoft's tools are really inconsistent about filename/GUID case.
There is also Electron's variant of this, which sits in front of S3.
It has the additional helpers of converting 403's to 404's (to not upset the debugger), and converting all paths to lowercase, so that incoming requests are case-insensitive.
https://github.com/electron/symbol-server
In the system32 directory I saw an .OCX file with a corresponding .OCA file.
I had thought .OCA files are used only by Visual Basic. Are they therefore unnecessary for program execution and could be removed?
If they are unnecessary, why would there be an .OCA in the system32 folder in the first place?
.OCA files serve as a cache of the extended type library information for its corresponding .OCX file.
If you delete an .OCA file for a control VB recognizes and uses, VB will recreate the .OCA file when you load a project requiring the control. The recreation process takes a little time but comes with no penalty otherwise.
Last reviewed: April 4, 1996
Article ID: Q149429
SUMMARY For every custom control file (.OCX) that Visual Basic uses,
there is an accompanying .OCA file of the same file name. For example,
GRAPH16.OCX has an accompanying .OCA file called GRAPH16.OCA. An .OCA
file is a binary file that functions as both an extended type library
file and a cache for the custom control file.
MORE INFORMATION
A type library is a file or component within another
file that contains OLE Automation standard descriptions of exposed
objects, properties, and methods. The actual working type library for
a custom control used in Visual Basic is a combination of the type
library of the control itself and the additional properties provided
by the framework that wrap the control.
Some of the properties of the control are provided by the framework
and some by the control itself. Programmatically, the properties from
the framework and the control all appear as properties of the control.
In order for these properties to appear, Visual Basic creates an
extended type library when the control is loaded into the toolbox.
Because the process of reading the control's type library and creating
the extended type library is time consuming, Visual Basic caches the
extended type library information into an OCA file.
If you delete the OCA file for a control Visual Basic recognized,
Visual Basic will re-create the .OCA file when you load a project
requiring the control. This re-creation process comes with a time
penalty.
(http://support.microsoft.com/kb/149429)
So, don't worry about including them when you deploy your application.
Sorry to resurrect a zombie thread, but I want to summarize to make sure I understand this.
An OCA file is only needed during compilation of a program and if it is missing, VB will create what it needs so all that is lost is time during compiling. Unless you have really sloppy programming and another homemade dll or ocx needs a particular oca.
If an oca is shipped with a product, it should be able to be safely deleted.
No, they are not necessary for it to execute but they are necessary for the program to run properly (if the program required the file in the first place).
They never need to be deployed with a finished program for it to run.
This could be a dumb attempt to answer, but you could simply rename the file and see if the application quits working. If so, those files are necessairy.
DNA-science is about the same:
Remove a gene and see what stopped "working". De facto, this gene is related to / necessairy for that body-part or whatever.
Re: No, they are not necessary for it to execute but they are necessary for the program to run properly (if the program required the file in the first place).
Since execute and run mean the same thing - what you are saying is that:
it's not required for it to run but it's required for it to run properly??
or alternately
it's not required for it to execute but it's required for it to execute properly??
I am suspecting that the *.OCA files are not required EXCEPT for compiling a program using VB and if they don't exist when you load a project then VB will create any missing OCA files automatically in order to speed up future load or compile operations in VB.
I just deleted EVERY OCA file off my system (I'm a brave man when I'm using a cloned virtual machine to do this).
I then rebooted and ran my software and it worked just fine without ANY OCA files on my entire system. Now it might take a bit longer to load/compile my programs next time as vb will regenerate the OCA files but I'm using an i7 with 16Gb of RAM and an SSD so who cares!
I have a Windows service application on Vista SP1 and I've found that users are renaming its executable file (while it's running) and then rebooting, thus causing it to fail to start on next bootup because the service manager can no longer find the exe file since it's been renamed.
I seem to recall that with older versions of Windows you couldn't do this because the OS placed a lock on the file. Even with Vista SP1 I still cannot copy over the existing file when it's running - Windows reports that the file is in use - makes sense. So why should I be allowed to rename it? What happens if Windows needs to page in a new code page from the exe but the file has been renamed since it was started? I ran Process Monitor while renaming the exe file, etc, but Process Mon didn't report anything strange and just logged changing the filename like any other file.
Does anyone know what's going on here behind the scenes? It's seem counter intuitive that Windows would allow a running process' filename (or its dependent DLLs) to be changed. What am I missing here?
your concept is wrong ... the filename is not the center of the file-io universe ... the handle to the open file is. the file is not moved to a different section of disk when you rename it, it's still in the same place and the part of the disk the internal data structure for the open file is still pointing to the same place. bottom line is that your observations are correct. you can rename a running program without causing problems. you can create a new file with the same name as the running program once you've renamed it. this is actually useful behavior if you want to update software while the software is running.
As long as the file is still there, Windows can still read from it - it's the underlying file that matters, not its name.
I can happily rename running executables on my XP machine.
The OS keeps an open handle to the .exe file,. Renaming the file simply changes some filesystem metadata about the file, without invalidating open handles. So when the OS goes to page in more code, it just uses the file handle it already has open.
Replacing the file (writing over its contents) is another matter entirely, and I'm guessing the OS opens with the FILE_SHARE_WRITE flag unset, so no other processes can write to the .exe file.
Might be a stupid question but, why do users have access to rename the file if they are not suppose to rename the file? But yeah, it's allowed because, as the good answers point out, the open handle to the file isn't lost until the application exits. And there are some uses for it as well, even though I'm not convinced updating an application by renaming its file is a good practice.
You might consider having your service listen to changes to the directory that your service is installed in. If it detects a rename, then it could rename itself back to what it's supposed to be.
There are two aspects to the notion of file here:
The data on the disk - that's the actual file.
The file-name (could be several or none) which you can give that data - called directory entries.
What you are renaming is the directory entry, which still references the same data. Windows doesn't care about your doing so, as it still can access the data when it needs to. The running process is mapped to the data, not the name.