I have a .NET Setup Project to which I've added a custom installer action. During the setup process, the user has to provide a path (which often is a UNC path) to a share on their file server. I attempt to do some validation before proceeding to make sure the directory exists, as such:
if (!Directory.Exists(serverDirectory)) {
throw new InstallException("Specified path does not exist or ...");
}
Pretty vanilla -- and in a console app, the Directory.Exists() code works as expected. However, in the context of the MSI, it does not. Specifically, the call to Directory.Exists always fails when using a network resource. The documentation for Directory.Exists does indicate why:
The Exists method does not perform network authentication. If you query an existing network share without being pre-authenticated, the Exists method will return false.
Searches have led me to other similar scenarios in ASP.NET where impersonation is the solution. That isn't applicable here, but it illustrates the issue.
How can I check for the existence of the network path? To put in the language of the documentation -- how do I pre-authenticate before the call? The user is installing as an administrator, and browsing to that path in Windows Explorer works successfully, so it's not permissions of the user, but rather a lack of checking by the code.
Have I created an unnecessary concern -- should I omit this and throw the exception later when trying to use the network resource... it's largely the same critical failure, right?
It's not just existence: you need to check for permissions as well, and worry about what happens if they change on you in the period between when you check and when you actually use the value.
Therefore the normal mechanism is to just assume everything is okay. Instead, put your development effort into handling the exception when the assumption proves false, because you need to be able to do that gracefully anyway.
For this case, you might be able to improve on that by immediately creating a small placeholder file in directory in question, and holding a lock on that file until you're completely finished with the folder. That would allow you to give some better feedback, since you'll get an immediate error trying to create the file. It also helps guarantee that the folder stays accessible, since under windows at least users will have a hard time deleting or significantly changing folder as long as you hold a lock to that file.
Related
AFAIK, OS X is a BSD derivation, which doesn't have actual mandatory file locking. If so, it seems that I have no way to prevent writing access from other programs even while I am writing a file.
How to guarantee file integrity in such environment? I don't care integrity after my program exited, because that's now user's responsibility. But at least, I think I need some kind of guarantee while my program is running.
How do other programs guarantee file content integrity without mandatory locking? Especially database programs. If there's common technique or recommended practice, please let me know.
Update
I am looking for this for data layer of GUI application for non-engineer users. And currently, my program have this situations.
Data is too big that it cannot be fit to RAM. And even hard to be temporarily copied. So it cannot be read/written atomically, and should be used from disk directly while program is running.
A long running professional GUI content editor application used by humans who are non-engineers. Though users are not engineers, but they still can access the file simultaneously with Finder or another programs. So users can delete or write on currently using file accidentally. Problem is users don't understand what is actually happening, and expect program handles file integrity at least program is running.
I think the only way to guarantee file's integrity in current situation is,
Open file with system-wide exclusive mandatory lock. Now the file is program's responsibility.
Check for integrity.
Use the file as like external memory while program is running.
Write all the modifications.
Unlock. Now the file is user's responsibility.
Because OS X lacks system-wide mandatory lock, so now I don't know what to do for this. But still I believe there's a way to archive this kind of file integrity, which just I don't know. And I want to know how everybody else handles this.
This question is not about my programming error. That's another problem. Current problem is protecting data from another programs which doesn't respect advisory file lockings. And also, users are usually root and the program is running with same user, so trivial Unix file privilege is not useful.
You have to look at the problem that you are trying to actually solve with mandatory locking.
File content integrity is not guaranteed by mandatory locking; unless you keep your file locked 24/7; file integrity will still depend on all processes observing file format/access conventions (and can still fail due to hard drive errors etc.).
What mandatory locking protects you against is programming errors that (by accident, not out of malice) fail to respect the proper locking protocols. At the same time, that protection is only partial, since failure to acquire a lock (mandatory or not) can still lead to file corruption. Mandatory locking can also reduce possible concurrency more than needed. In short, mandatory locking provides more protection than advisory locking against software defects, but the protection is not complete.
One solution to the problem of accidental corruption is to use a library that is aggressively tested for preserving data integrity. One such library (there are others) is SQlite (see also here and here for more information). On OS X, Core Data provides an abstraction layer over SQLite as a data storage. Obviously, such an approach should be complemented by replication/backup so that you have protection against other causes for data corruption where the storage layer cannot help you (media failure, accidental deletion).
Additional protection can be gained by restricting file access to a database and allowing access only through a gateway (such as a socket or messaging library). Then you will just have a single process running that merely acquires a lock (and never releases it). This setup is fairly easy to test; the lock is merely to prevent having more than one instance of the gateway process running.
One simple solution would be to simply hide the file from the user until your program is done using it.
There are various ways to hide files. It depends on whether you're modifying an existing file that was previously visible to the user or creating a new file. Even if modifying an existing file, it might be best to create a hidden working copy and then atomically exchange its contents with the file that's visible to the user.
One approach to hiding a file is to create it in a location which is not normally visible to users. (That is, it's not necessary that the file be totally impossible for the user to reach, just out of the way so that they won't stumble on it.) You can obtain such a location using -[NSFileManager URLForDirectory:inDomain:appropriateForURL:create:error:] and passing NSItemReplacementDirectory and NSUserDomainMask for the first two parameters. See -replaceItemAtURL:withItemAtURL:backupItemName:options:resultingItemURL:error: method for how to atomically move the file into its file place.
You can set a file to be hidden using various APIs. You can use -[NSURL setResourceValue:forKey:error:] with the key NSURLIsHiddenKey. You can use the chflags() system call to set UF_HIDDEN. The old Unix standby is to use a filename starting with a period ('.').
Here's some details about this topic:
https://developer.apple.com/library/ios/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileCoordinators/FileCoordinators.html
Now I think the basic policy on OSX is something like this.
Always allow access by any process.
Always be prepared for shared data file mutation.
Be notified when other processes mutates the file content, and provide proper response on them. For example you can display an error to end users if other process is trying to access the file. And then users will learn that's bad, and will not do it again.
I have a Rails app that reads from a .yml file each time that it performs a search. (This is a full text search app.) The .yml file tells the app which url it should be making search requests to because different version of the search index reside on different servers, and I occasionally switch between indexes.
I have an admin section of the app that allows me to rewrite the aforementioned .yml file so that I can add new search urls or remove unneeded ones. While I could manually edit the file on the server, I would prefer to be able to also edit it in my site admin section so that when I don't have access to the server, I can still make any necessary changes.
What is the best practice for making edits to a file that is actually used by my app? (I guess this could also apply to, say, an app that had the ability to rewrite one of its own helper files, post-deployment.)
Is it a problem that I could be in the process of rewriting this file while another user connecting to my site wants to perform a search? Could I make their search fail if I'm in the middle of a write operation? Should I initially write my new .yml file to a temp file and only later replace the original .yml file? I know that a write operation is pretty fast, but I just wanted to see what others thought.
UPDATE: Thanks for the replies everyone! Although I see that I'd be better off using some sort of caching rather than reading the file on each request, it helped to find out what the best way to actually do the file rewrite is, given that I'm specifically looking to re-read it each time in this specific case.
If you must use a file for this then the safe process looks like this:
Write the new content to a temporary file of some sort.
Use File.rename to atomically replace the old file with the new one.
If you don't use separate files, you can easily end up with a half-written broken file when the inevitable problems occur. The File.rename class method is just a wrapper for the rename(2) system call and that's guaranteed to be atomic (i.e. it either fully succeeds or fully fails, it won't leave you in an inconsistent in-between state).
If you want to replace /some/path/f.yml then you'd do something like this:
begin
# Write your new stuff to /some/path/f.yml.tmp here
File.rename('/some/path/f.yml.tmp', '/some/path/f.yml')
rescue SystemCallError => e
# Log an error, complain loudly, fall over and cry, ...
end
As others have said, a file really isn't the best way to deal with this and if you have multiple servers, using a file will fail when the servers become out of sync. You'd be better off using a database that several servers can access, then you could:
Cache the value in each web server process.
Blindly refresh it every 10 minutes (or whatever works).
Refresh the cached value if connecting to the remote server fails (with extra error checking to avoid refresh/connect/fail loops).
Firstly, let me say that reading that file on every request is a performance killer. Don't do it! If you really really need to keep that data in a .yml file, then you need to cache it and reload only after it changes (based on the file's timestamp.)
But don't check the timestamp every on every request - that's almost as bad. Check it on a request if it's been n minutes since the last check. Probably in a before_filter somewhere. And if you're running in threaded mode (most people aren't), be careful that you're using a Mutex or something.
If you really want to do this via overwriting files, use the filesystem's locking features to block other threads from accessing your configuration file while it's being written. Maybe check out something like this.
I'd strongly recommend not using files for configuration that needs to be changed without re-deploying the app though. First, you're now requiring that a file be read every time someone does a search. Second, for security reasons it's generally a bad idea to allow your web application write access to its own code. I would store these search index URLs in the database or a memcached key.
edit: As #bioneuralnet points out, it's important to decide whether you need real-time configuration updates or just eventual syncing.
I am new to writing Windows Services so this is really strange to me. I have debugged an external library using a WinForm. The external library does some drive mapping, copying a directory structure, deleting the original directory, renaming the copied directory, and then removes mappings.
So, when I write up the service to initialize the external class and start the process, the service doesn't seem to be doing what I think it should be doing. It appears to be doing nothing and completely ignoring what is going on.
Is there anything I am missing? Does the external library need to have any Security attributes?
Update
Found out how to attach a debugger, per #Will's comment: System.Diagnostics.Debugger.Break()
Edit
It also helps when you copy your app.config file to the correct directory!!!
Not a lot to go on here. First, you can debug your service, which is what you should be doing rather than using a winform frontend to test your code.
The issue is either that your service is not created properly, or that you've got a security issue.
Your service will receive a start message, after which it must run its own code, often on a different thread (a Timer is a common way to do this).
If the service is touching a drive, the user account under which it executes must have rights to perform the operations it attempts. Try changing the user account under which it executes to your own and see if it starts working.
I would like to improve the way how an application is checking that another instance is not already running. Right now we are using named mutexes with checking of running processes.
The goal is to prevent security attacks (as this is security software). My idea right now is that "bulletproof" solution is only to write an driver, that will serve this kind of information and will authenticate client via signed binaries.
Does anyone solved such problem?
What are your opinions and recommendations?
First, let me say that there is ultimately no way to protect your process from agents that have administrator or system access. Even if you write a rootkit driver that intercepts all system calls (a difficult and unsafe practice in of itself), there are still ways to use admin access to get in. You have the wrong design if this is a requirement.
If you set up your secure process to run as a service, you can use the Service Control Manager to start it. The SCM will only start one instance, will monitor that it stays up, allow you to define actions to execute if it crashes, and allow you to query the current status. Since this is controlled by the SCM and the service database can only be modified by administrators, an attacking process would not be able to spoof it.
I don't think there's a secure way of doing this. No matter what kind of system-unique, or user-unique named object you use - malicious 3rd party software can still use the exact same name and that would prevent your application from starting at all.
If you use the method of checking the currently executing processes, and checking if no executable with the same name is running - you'd run into problems, if the malicious software has the same executable name. If you also check the path, of that executable - then it would be possible to run two copies of your app from different locations.
If you create/delete a file when starting/finishing - that might be tricked as well.
The only thing that comes to my mind is you may be able to achieve the desired effect by putting all the logic of your app into a COM object, and then have a GUI application interact with it through COM interfaces. This would, only ensure, that there is only one COM object - you would be able to run as many GUI clients as you want. Note, that I'm not suggesting this as a bulletproof method - it may have it's own holes (for example - someone could make your GUI client to connect to a 3rd party COM object, by simply editing the registry).
So, the short answer - there is no truly secure way of doing this.
I use a named pipe¹, where the name is derived from the conditions that must be unique:
Name of the application (this is not the file name of the executable)
Username of the user who launched the application
If the named pipe creation fails because a pipe with that name already exists, then I know an instance is already running. I use a second lock around this check for thread (process) safety. The named pipe is automatically closed when the application terminates (even if the termination was due to an End Process command).
¹ This may not be the best general option, but in my case I end up sending data on it at a later point in the application lifetime.
In pseudo code:
numberofapps = 0
for each process in processes
if path to module file equals path to this module file
increment numberofapps
if number of apps > 1
exit
See msdn.microsoft.com/en-us/library/ms682623(VS.85).aspx for details on how to enumerate processes.
Are you aware of any serious performance problems with checking if a file exists before you open it and send it? The context for this is a web application that serves up files dynamically via an ASP page. I was told that there is no check for whether or not the file exists because the database stores this information and shouldn't be wrong. It is of course sometimes wrong for any number of reasons so we end up sending back nothing leaving the user unhappy.
My instinct is that checking for a file's existence is so cheap that you shouldn't worry about it but I was told otherwise. Our storage solution is a pretty high powered one (not just an IDE drive on the web server) if that helps. My question is basically: Is my instinct correct?
Thanks!
Note: These files are never just deleted. Something has to go wrong for them to be missing but this happens a few times a week. Also, the reason I want to check for a file's existence is because I can return an alternate asset that is on disk so I'd like to be able to have that logic all in one spot rather than deal with catching an exception and dealing with that in that context.
Even if you check it exists just before you try to serve it, it could be deleted between the check and you serving it.
The problem here is that you send back nothing if you can't serve the file. Trying to serve a non-existent file should throw an exception (or whatever the equivalent is on your platform) - you should handle that exception by returning an appropriate "Sorry, we couldn't find your file" error page.
If the file not being there is an exceptional circumstance (as you say the db is always right), you should not check for it.
If its not there you get an exception and handle it accordingly. That seems to be the way you say the system works, and I would handle it as such.
One more note, If your saying that a file not being there when opened is just returning nothing, then that tells me there is a design flaw in your exception hanlding, and your lower level is not bubbling the exception far up enough to be properly handled so you can convey an error message back to the client.
With proper exception handling/bublling you should have no problem returning an alernate asset. Rethink your exception handling rather than redesigning the way the system is supposed to work.
You shouldn't experience any serious performance issues so long as you're not using some sort of really weird storage scheme.
I'm not aware of any major performance problems with this approach, but checking for a file's existence before opening it can have unexpected results (another process can delete the file between the check and opening it).
I really don't know if it is a cheap or expensive task, but, considering that not checking for the existence of the file could result in an unpleasant response to the client I would say it's not expensive.
I do have some web apps where I check for file existence and is working just fine.
Are you redirecting to the file, or reading and serving up the contents of the file through code?
If the first, what happens if the file for some reason doesn't exist? Standard 404 error, or do you have a specialized error page? Is that acceptable for that case?
If the latter, simply open the file and handle the exception appropriately.
File IO is generally costly in comparison to database or in-memory reads, but I have heard the old "the database stores the information so it shouldn't be wrong" too many times when hunting down a system crash or unhandled exception. So I would check for existence and recover elegantly unless your performance requirements are unusually high.
Performance wise there should be essentially no impact. The slow part here is going to disk to read a file header, but if you're going to load the file you'll have to do that anyway and any decent storage system (pretty much any one from the past couple of decades) will cache the read from the check so that part of the read is much faster when you actually open the file. Having said that, I agree with mattlant that it would be generally better to catch the file not there exception and deal with it properly if the file is generally expected to be there.
If you're going to open it immediately after then checking for it's existance is essentially free.
Try opening the file and then streaming it. Is that a possibility? That way if you're unable to open and stream it then you will be unable to send it and can action the appropriate error handling.
If not checking for the file causes confusion for the user and/or doesn't give them what they need, or at least display an error message indicating the problem - then it doesn't matter what the cost is.
The application needs the check.