I've got two Win32 programs: "program A", a web interface, and "program B", a worker program, that operate on a collection of files. "Program A" tells "program B" to perform an operation on a set of files, "program B" does it, and signals to "program A" that it's done.
Now, here's the situation: there might be multiple "program A"s and "program B"s on different computers, accessing files stored on a network share. I need to ensure that only one "program B" is accessing a set of files at a time.
I could do it using lock files, but those have issues (such as not going away if the owning process crashes) that I'd need to deal with. I don't think I can use traditional file locking, since my architecture requires "program A" to control the locking while "program B" accesses the files. Is there a solution I'm not seeing, such as a "network-wide mutex"?
Lock files should be all you need. You just need to choose semantics that say "this set of files is locked by someone else if I can't open the associated lock file for exclusive access" rather than "this set of files is locked by someone else if the associated lock file exists".
You could still run into problems requiring admin intervention if one of the worker servers crashes (the entire server, not your program) but that should be rare enough that this is OK; just make sure your logic raises some sort of alarm if any particular set of files is locked for an excessively long time.
If you create the "Lock File" using FILE_FLAG_DELETE_ON_CLOSE this should actually delete the file even if the process crashes. (I'm not completely sure this also happens on network drives however).
Related
the problem is simple, I have a folder with working files, my program stores files/folders within this folder while processing. However, the problem is that I often get an error like "File 'Temp123.temp' cannot be accessed, it is being used by another program." I want to know which program does this.
If I try to use the resource monitor I am not fast enough, as the files are created, used and deleted in a matter of a few seconds and their name is random.
I suspected that the antivirus might be to blame, but even when I deactivate this, the error persists.
Is there a possibility to monitor the whole work-folder and its subfolders to see what programs are using my files?
Thank you so much in advance!
This is a standard function of Microsoft's SysInternals "Process Monitor" aka ProcMon tool
I've been trying to understand the relationship between applications, instances, exes and processes.
When i open two different word files on my windows laptop, 2 different 'Processes' show up in Task manager. I can independently kill each one
When i open Process Explorer - I can only see 1 instance of WINWORD.exe. I can't seem to find the processes related to each of the files i opened. Can someone explain whether each file has an associated process or a single process is used to host multiple files?
There's a mixup of several concepts here; executables, processes, what "opening a file" means and "whatever Task Manager shows in its processes tab", which is mostly a lie.
Double-clicking on a document generally1 boils down to a look up on the file extension in the registry and launching the associated executable, passing the path to the clicked file as command line argument.
Whenever an executable is launched, the kernel spawns a process (which roughly means, allocates some virtual memory for it and creates its main thread of execution) and loads the executable in it ("copies"2 it in memory, loads its dependencies and in general performs a lot of tasks necessary to be able to execute the machine code that is stored in it).
What happens next depends entirely on the code of the executable. Most programs will keep executing in the new process; some instead will look for an instance of the program already executing (which may or may not be the same executable), and communicate to it that a new instance has been launched (possibly forwarding to the existing instance the arguments obtained on the command line). This is generally the case for applications where there's a lot of data to share between different instances (so, to save e.g. memory) or where there's interaction between e.g. the various opened documents that is less cumbersome to implement in a single process (IPC is often a pain).
When instead you open a file e.g. from the File → Open menu of an application, that's generally done in the same process; again, when you actually click open it's up to the application to decide whether to open it inside the same process, or to spawn a different one for the other document.
The important concept here, though, is that if you see different top-level windows (≈ windows that you see in the taskbar) it does not mean they come from different processes, a single process may spawn multiple top-level windows (and indeed applications that deal with multiple files in multiple windows often do that). Also, remember that the fact that the 1 open file = 1 window (or 1 application, for what that matters) is just a GUI concept; you may have files that are handled concurrently by multiple processes (databases are a common instance of this).
As for Task Manager: the processes tab doesn't really list processes as they are seen by the system; some are grouped according to some unspecified rule/heuristics, some other processes are expanded to show sub-components (e.g. the various svchost.exe processes); processes that have multiple top-level windows, in particular, are shown "as if" having sub-processes. Clicking on "End task" on these fake sub-processes will actually send a "polite" quit request to the associated window, which will be handled in an application-specific way.
Ultimately, you shouldn't really think too hard about what is shown here. What are actually the "real" processes as understood by the operating system kernel is what is shown in the "Details" tab, which should match what you see in Process Exporer. If you do kill the process here, you'll see that all windows that are managed by it disappear.
I say "generally" because it's actually a lot more complicated than that; file association can do some really complicated stuff that avoids process creation, generally to directly communicate to an already running program that it has to load another file, essentially skipping the middleman described in the section that follows.
In quotes because it's smarter than that; what is generally done is a copy-on-write file mapping, which instructs the OS virtual memory manager to load the parts of the executable that are needed from disk when they are accessed in memory, and to copy on the fly the sections that are modified.
Is there a way to write to a locked file regardless of what program/process has it open?
Scenario:
A commercial product running as a service, locks a log file.
Service can not be stopped because it will impact customers.
Would like to inject a line at the end of the file as a marker.
Getting error: The process cannot access the file because it is being used by another process.
Is there ANY way to append a line on a locked file?
Also, afraid of breaking the lock status, as it may cause the commercial program to break.
Originally thinking batch file until finding out log file is locked. Will do powershell or whatever language can pull this off.
No, the whole idea of the lock(ed) file is that the process that has the lock does not have to worry about other processes modifying the file, so the lock owner does not have to query the file system all the time and can do it's operations much more efficiently.
And see earlier comments by "David Schwartz" and "Ken White"
I have a VB6 program running on Windows 7. It is copying a large number of files and sometimes FileCopy fails with an access violation (between every 60 and 500 files).
I cannot reproduce it using a single file, only during such mass-copying operations this problem happens.
It makes no difference, if source/target are on hard disks, network shares or CD-ROMs.
What could trigger this problem?
EDIT: My question might be a little bit convoluted, so here's some more data:
Run 1:
Start copying 5.000 files
Access violation on file #983
Access violation on file #1437
Access violation on file #1499
Access violation on file #2132
Access violation on file #3456
Access violation on file #4320
Done
Run 2:
Start copying 5.000 files
Access violation on file #60
Access violation on file #3745
Done
Observations
The affected files are always different
The number of affected files tends to decrease if the same file batch is copied multiple times in succession.
Running as Administrator makes no difference
The application has read/write access to all necessary file system objects
This problem happens on Windows 7 workstations only!
Best guess: Is it possible that another user/application is using the specified file at the time the process is running? (anti-virus scanner, Win7 search indexing tool, windows defender, etc) You might try booting the machine in safe-mood to eliminate any of the background services/apps and try running the process to see.
Is there any consistency in the file types or size of the files causing the issue?
Is the machine low on resources? RAM/Disk Space
You said it occurs on Win7 – is it multiple Win7 machines or just one. (help to rule out system resources vs. software/OS)
Any hints from the event viewer (control panel > admin tools) – doubtful
Does the process take a long time to complete? If you can take the performance hit you might look at destroying and recreating the FSO object after every copy or every X files to make sure there isn’t some odd memory leak issue with Win7/VB6.
Not necessarily a recommended solution but if all else fails you could handle that error and save the files that trigger it in a dictionary/collection and reloop through the process with any those files when done. No guarantee it wouldn’t happen again.
Not enough information (as you probably know). Do you log the activity? If not, it's a good place to start. Knowing whether certain files are the problem, and if the issue is repeatable, can help narrow it down.
In your case I would also trap (and log) all errors and retry N times after waiting N seconds. You could be trying to copy in-use files locked by another process, and a retry may allow time for that lock to go away.
Really, more data is the key, and logging is the way to get it.
Is there any chance your antivirus program or some indexer is getting in the way?
Try creating a procmon trace while reproducing the error and see what is actually failing. With the trace you can see if there is another program causing the issue or if your app is trying to write somewhere it should't (incorrect permissions) or can't (a temp/scratch directory without enough space).
Check out the presentations linked to on the procmon page or Mark Russinovich's blog for some cool examples of using this tool to solve various Windows/application mysteries.
Is there a a hidden/system file in the directory that is potentially blocking it?
Does running the VB6 App with right-click "Run As Administrator" make a difference?
Is the point where it dies at the max # of files in the directory? e.g. Are you sure the upper limit on whatever loop structure you are using in VB6 is correct (Count vs count -1)?
UNIX file-locking is dead-easy: The operating system assumes that you know what you are doing and lets you do what you want:
For example, if you try to delete a file which another process has opened the operating system will usually let you do it. The original process still keeps it's file-handles until it terminates - at which point the the file-system will quietly re-cycle the disk-resources. No fuss, that's the way I like it.
How different things are on Windows: If I try to delete a file which another process is using I get an Operating-System error. The file is untouchable until the original process releases it's lock on the file. That was great back in the single-user days of MS-DOS when any locking process was likely to be on the same computer that contained the files, however on a network it's a nightmare:
Consider what happens when a process hangs while writing to a shared file on a Windows file-server. Before the file can be deleted we have to locate the computer and ID the process on that computer which originally opened the file. Only then can we kill the process and delete our unwanted file.
What a nuisance!
Is there a way to make this better? What I want is for file-locking on Windows to behave a like file-locking in UNIX. I want the operating system to just let me do what I want because I'm in charge and I know what I'm doing...
...so can it be done?
No. Windows is designed for the "average user", that is people who don't understand anything about a computer. Therefore, the OS tries to be smart to avoid PEBKACs. To quote Bill Gates: "There are no issues with Windows that any number of people want to be fixed." Of course, he knows that 99.9999% of all Windows users can't tell whether the program just did something odd because of them or the guy who wrote it.
Unix was designed when the world was more simple and anyone close enough to a computer to touch it, probably knew how to assemble it from dirty sand. Therefore, the OS usually lets you do what you want because it assumes that you know better (and if you didn't, you will next time).
Technical answer: Unix allocates an "i-nodes" if you create a file. I-nodes can be shared between processes. If two processes create the same file (that is, two processes call create() with the same path), then you end up with two i-nodes. This is by design. It allows for a fancy security feature: You can create files which no one can open but yourself:
Open a file
Delete it (but keep the file handle)
Use the file any way you like
Close the file
After step #2, the only process in the universe who can access the file is the one who created it (unless you want to read the hard disk block by block). The OS will keep the data alive until you either close the file or your process dies (at which time Unix will clean up after you).
This design is the foundation of all Unix filesystems. The Windows file system NTFS works much the same way but the high level API is different. Many applications open files in exclusive mode (which prevents anyone, even backup programs) to read the file. This is even true for applications which just display information like PDF viewers.
That means you'll have to fix all the Windows applications to achieve the desired effect. If you have access to the source, you can create a file in a shared mode. That would allow other processes to access it at the same time but then, you will have to check before every read/write if the file still exists, whether someone has made changes, etc.
According to MSDN you can specify to CreateFile() 3rd parameter (dwSharedMode) shared mode flag FILE_SHARE_DELETE which:
Enables subsequent open operations on a file or device to request delete access.
Otherwise, other processes cannot open the file or device if they request delete access.
If this flag is not specified, but the file or device has been opened for delete access, the function fails.
Note Delete access allows both delete and rename operations.
http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx
So if you're can control your applications you can use this flag.
Note that Process Explorer allow for force closing of file handles (for processes local to the box on which you are running it) via Handle -> Close Handle.
Unlocker purports to do a lot more, and provides a helpful list of other tools.
Also deleting on reboot is an option (though this sounds like not what you want)
That doesn't really help if the hung process still has the handle open. It won't release the resources until that hung process releases the handle. But anyway, in Windows it is possible to force close a file out from under a process that's using it. Process Explorer from sysinternals.com will let you look at and close handles that a process has open.