VSS backups cause custom windows service to fail - windows

I've built custom windows services in vb.net that run on a timer and look for files in a remote directory to process. On a few servers, they fail without any error at the same exact time, around 10:04PM when the VSS service fails. The service errors look like:
'The process cannot access the file filename.txt because it is being used by another process.'
Sometimes there are no errors but the service hangs. I am assuming this is the VSS service. Regardless, there is a huge spike in CPU up to 100% and back down.
I have logs (ie the file above that cannot be written to) that relay all activity to me, and each process is wrapped in a try..catch which will also report exceptions to logs for me. Nothing comes of this. The processes fail in random parts of code, usually in the poll timer and sometimes in the middle of processing without warning. The service hangs but says it's still running in services.msc.
Has anyone experienced this issue or know of a fix? I was thinking to make the services multi-threaded so one can check on another in case one thread hangs but am not sure if this would fix the problem. We have tried giving more resources to the servers and this seems to have helped a bit but did not completely fix the problem. Some days it overloads, others it doesnt. Using windows server 2008 x64.
Thanks in advance for you help!
Public Shared aTimer As New Timers.Timer
Public Shared Sub SetTimer()
aTimer.Interval = 60000
' Hook up the Elapsed event for the timer.
AddHandler aTimer.Elapsed, AddressOf OnTimedEvent
aTimer.Enabled = True
End Sub
' The event handler for the Timer.Elapsed event.
Private Shared Sub OnTimedEvent(source As Object, e As ElapsedEventArgs)
Dim gen = New Service1
aTimer.Enabled = False
gen.ProcessEvents() 'This is the main function of the service
aTimer.Enabled = True
End Sub

Update:
Had never thought of a simple solution like this, but in the Task Manager, I right clicked on the service and set the priority to "Realtime" in hopes that the high CPU load during backups would not affect the service. So far so good.
This can be done programatically so each time the service restarts it is in higher priority. This is in my OnStart():
VB.NET:
System.Diagnostics.Process.GetCurrentProcess().PriorityClass = System.Diagnostics.ProcessPriorityClass.RealTime

Related

Synchronous VB6 apparently behaving asynchronously! Crash

We have a legacy VB6 app which has started, from time to time, hangs. We thought it may be to do with a shift to Citrix, but can now replicate the behaviour on a thick client on Win10. We don't think that we have seen this before on earlier Windows versions, but are still checking logs to confirm that.
We experience the behaviour when tabbing into a text box and then tabbing out. As we pass through it, we are making a simple ado call to lookup/validate some data in a text box. As part of the correct program running we are logging
“Opening Dataset: SELECT ... FROM ... ”
“Opened Dataset”
Between these 2 log statements is simple ado data retrieval code with which we have had no problems previously. It is in an ActiveX dll and is running synchronously. Most importantly is that between these 2 log statements there is no DoEvents or api call which would yield control. As far as we can see, it should be a purely synchronous operation.
When the system crashes, which happens sporadically, we can see other logging statements appear between these 2 which can be either resource status (e.g. how much memory, gdi/user objects - which would usually be found because a timer has triggered in the main form) or focus type events - which aren’t timer driven - at least in our codebase.
“Opening Dataset: SELECT ... FROM ... ”
“Resource Status: ...”
“Opened Dataset”```
or
“Opening Dataset: SELECT ... FROM ... ”
“TextItem.OnLostFocus Item1 ...”
“TextItem.Validate ...”
“TextItem.OnGotFocus Item2 ...
“Opened Dataset”
So my initial question is, in what scenario can what should be a synchronous operation be interrupted and appear to act asynchronously.
For example, and we aren’t doing this, I could imagine writing some unsafe code whereby by using a multimedia timer (on another thread) and supplying an AddressOf parameter to the address of a function on one of our modules, that that timer initiates execution of our code, separate to the correct control flow. Other than something like that, I just can’t see how synchronous vb6 code could be interrupted in this way.
I’d be really grateful of any thoughts, suggestions or advice. I’m really sorry if this is soo vague. It perhaps reflects how I’m struggling to get my head round this problem.
Just to say, we tracked this down to Windows 10 plus an old (out of support) socket component we are using. It looks like it is pumping the message queue "at the wrong time" and hence we are seeing UI events appear in the middle of a synchronous process. We don't see this behaviour on earlier Windows versions.
I don't know what may have changed in Win10 which would result in this, but we obviously need to upgrade.
In our case we had a few long running timers to pull status/changes from the DB which caused this. We are using ADO with SQL Native Client and MARS, which worked great up until Windows 10 where intermittent lock ups occurred. Logging and Windbg confirmed this was happening when 2 requests where hitting the ADO connection at the same time. The error from ADO was "Unable to open a logical session" error number -2147467259, and actually caused SQL Server 2014 (running on another machine) to block all other client queries from multiple different applications and machines until the locked up app was killed. I could not replicate this in the IDE as apparently that forces timers to work the way they always did. The fix was to async our ADO implementation and put a connection manager on top of the SQL connections to force requestors to wait their turn (basically taking the Win10 async'd timer feature back out). My only performance impact was the additional few milliseconds of delaying the timer fired SQL query when it collided with a another query.

How to know if the process is running from service

I made a GUI application that must run in my DELL server to send queries for 24 hours.
In case where the application is stopped by random users, or whatever it is, I created a service program that detects when it stops running, and executes it again.
The problem is, in service, FindWindow() doesn't properly work (always returns nullptr) because Microsoft changed its OS service policies since XP. And my service program has no way to find if the program is on the process list or not.
I found some solutions on the internet, which is to "allow service to interact with desktop on service panel" but since it was a long time ago so doesn't quite fit into the current OS version.
Should I use IPC instead? or any other ways to fix?
I believe that there has got to be a way to do this, because executing a process from service is also possible by using CreateProcessAsUser().
Any advice will be truly appreciated.
Thanks in advance.
So I did what Remy Lebeau suggessted for me, and it properly works in Windows 7, and 2008.
Here's how I went step by step.
Create a named mutex in the global namespace in the GUI application.
::CreateMutex(nullptr, false, L"Global\\MyMutex");
Check periodically if the mutex has disappeared or not
by using CreateMutex(), and do not forget to take care of the reference count to the handle.
HANDLE hDetector = ::CreateMutex(nullptr, false, L"Global\\MyMutex");
if (GetLastError() == ERROR_ALREADY_EXISTS)
{
// The GUI application is still running.
// ...
::CloseHandle(hDetector);
}
else
{
// The GUI application is not running.
// ...
::CloseHandle(hDetector);
}
See it work.

How to determine the association between a VB6 app and an exe instanced with CreateObject()

We need to figure out how a service can peek at a running VB6 app and/or its DCOM spawned exe and figure out which VB6 app goes with which DCOM exe. The VB6 app and the spawned exe are both on the same server.
We have a VB6 app that spawns an instance of Bartender (from Seagull Scientific) by way of a CreateObject() call. On a given server, we may have ten or twenty instances of our app, each represents a handheld RF gun client in a warehouse. 95% or more of these VB6 apps will have their own Bartender.
Due to circumstances beyond our control, randomly, one of our VB6 instances will be killed, just as if you killed it using Task Manager. This leaves it's Bartender still alive and consuming resources. After fifty or so have been killed over the course of a few hours or days, these orphaned Bartenders become enough of a resource hog to bring the server to its knees.
We are trying to develop a watcher service to detect which of the Bartenders are still connected, so this new service can kill the orphaned Bartenders. We are trying to accomplish this without changing our VB6 app, but we will modify our app if we have to.
I think this routine, aptly named Who's Your Daddy, might be of use to you. It figures out who spawned the process. It probably won't solve your entire problem, but it's a start.
This is going to be hard, if not impossible, to do. Out-of-process COM components (i.e. ActiveX EXE's) are always started by the COM Service Control Manager, not by the process that called CreateObject. This is why the parent process for the ActiveX EXE is svchost.exe.
Therefore, there is no direct parent-child relationship between the process that calls CreateObject and the process that gets created. Only the remote procedure call (RPC) layer that actually passes method calls back and forth between the two processes knows the identities of the processes involved, but the RPC mechanism is specifically designed to be transparent to the COM subsystem, and there isn't an easy way to get access to this information that I know of.
However, there is a pretty hackish way to handle the orphaned process problem if you are willing to change the VB6 application:
Have your monitor service periodically terminate all running Bartender EXE's (once a day or however often is necessary to prevent the server from slowing down too much).
Write a wrapper DLL for the Bartender functionality, and have your VB6 class use this wrapper library instead of directly instantiating raw Bartender objects. This library would contain a wrapper class that creates a Bartender object, and that has methods that delegate to this object. Each wrapper method should catch error 462 ("The remote server machine does not exist or is unavailable"), recreate the Bartender object if this occurs, and then retry the method.
For example (I haven't actually looked at the Bartender documentation, so this is just demonstrating the idea):
'BartenderWrapper.cls
Private m_bartender As Object
Private Sub Class_Initialize()
Set m_bartender = CreateObject("Bartender.Application")
End Sub
Public Sub PrintLabel(Byval sLabelData As String)
On Error Goto ErrorHandler
m_bartender.PrintLabel sLabelData
Exit Sub
ErrorHandler:
If IsRpcError(Err) Then
Set m_bartender = CreateObject("Bartender.Application")
Resume
End If
Err.Raise Err.Number, Err.Source, Err.Description
End Sub
Private Function IsRpcError(Byval e As ErrObject) As Boolean
IsRpcError = (e.Number = 462)
End Function
The idea here is that since you can't reliably determine which Bartender processes are still connected to an instance of your VB6 application, you can kill all of the running Bartender processes periodically, and your application will still be able to run properly (in most cases), because if you kill a Bartender EXE that was being used by a running instance of your VB6 application, your application will create a new Bartender instance and continue running normally.
This solution definitely isn't fool-proof, and may be hard to implement if you are using a lot of methods or the Bartender instance you create has important internal state that could be lost when creating a new instance.
When it comes down to it, there isn't a clean way to detect orphaned ActiveX EXE's if you don't control all of the applications that are involved (one common solution when you do control the ActiveX EXE is to have the ActiveX EXE raise an event with a ByRef parameter every second or so, and have it shut itself down if the client doesn't change the value of the parameter).
What we have decided to do is to have the client write a hint file each time the Client creates a Bartender. The client writes a tiny XML file in a common folder that says an XML equivalent of "I am PID number n. Between time x and time y, I created a Bartender." The times x and y are timestamps obtained immediately before and after the CreateObject call. We will have a monitor service that watches for new Clients, new Bartenders and hint files. By watching all these, we think we can create small groups or associations of clients and their associated bartenders. In any given group, when all the clients go away, any remaining Bartenders that were in that group can be KILLED!

how can I make a Windows service beep?

I've got a service that acts as a watchdog for several apps/servers. There are no user sessions on this machine. I'd like the watchdog to be capable of beeping on the internal speaker should something go wrong (that'd be my queue to go fix whatever it's complaining about)
when I try the Beep() API on Windows nothing happens - I suspect the problem is that the services session isnt permitted to make noises?
can I make this work? any other ideas for how to make the service alert me?
-CG
Call CreateFile on \device\beep, then send down IOCTL_BEEP_SET (see http://www.koders.com/c/fidFEC3527B9D951559D62722A9C0C603863106CA9B.aspx for details)
It may work if you allow it to interact with the desktop (an option configurable somewhere, I can't remember where).
But personally, I'd have it email me.
Though maybe you could have it use the task scheduling API to schedule a task for yourself, so next time you log on you can see it.
I don't know; you've got a few options. I'd avoid beeping though.
Try sending beep char "\a" to console. Not sure if it will work.
Beeping doesn't seem like a good idea - it might end up driving everyone mad....
I'd also agree about the "interact with desktop" option and you set this in the services parameters see A Windows Service without a template
I'd recommend creating a simple client application that polls that server to query for any problems and returns a set of status messages. Then an appropriate UI would be raised (e.g. balloon on the tray), an email sent, etc. containing any warning or failure messages.
This way you also know that the watchdog itself is running and has network connectivity - if the watchdog dies and/or machine locks up you wouldn't otherwise know.
It also avoids being thrown out of a window when the machine starts beeping continuously just after you go to lunch. [+1 to #mikej] :-)
The poll period should be around half (see Nyquist sampling rate) your minimum required response time.

Error 1053: the service did not respond to the start or control request in a timely fashion

I have recently inherited a couple of applications that run as windows services, and I am having problems providing a gui (accessible from a context menu in system tray) with both of them.
The reason why we need a gui for a windows service is in order to be able to re-configure the behaviour of the windows service(s) without resorting to stopping/re-starting.
My code works fine in debug mode, and I get the context menu come up, and everything behaves correctly etc.
When I install the service via "installutil" using a named account (i.e., not Local System Account), the service runs fine, but doesn't display the icon in the system tray (I know this is normal behavior because I don't have the "interact with desktop" option).
Here is the problem though - when I choose the "LocalSystemAccount" option, and check the "interact with desktop" option, the service takes AGES to start up for no obvious reason, and I just keep getting
Could not start the ... service on Local Computer.
Error 1053: the service did not respond to the start or control request in a timely fashion.
Incidentally, I increased the windows service timeout from the default 30 seconds to 2 minutes via a registry hack (see http://support.microsoft.com/kb/824344, search for TimeoutPeriod in section 3), however the service start up still times out.
My first question is - why might the "Local System Account" login takes SOOOOO MUCH LONGER than when the service logs in with the non-LocalSystemAccount, causing the windows service time-out? what's could the difference be between these two to cause such different behavior at start up?
Secondly - taking a step back, all I'm trying to achieve, is simply a windows service that provides a gui for configuration - I'd be quite happy to run using the non-Local System Account (with named user/pwd), if I could get the service to interact with the desktop (that is, have a context menu available from the system tray). Is this possible, and if so how?
Any pointers to the above questions would be appreciated!
After fighting this message for days, a friend told me that you MUST use the Release build. When I InstallUtil the Debug build, it gives this message. The Release build Starts fine.
If you continue down the road of trying to make your service interact with the user's desktop directly, you'll lose: even under the best of circumstances (i.e. "before Vista"), this is extremely tricky.
Windows internally manages several window stations, each with their own desktop. The window station assigned to services running under a given account is completely different from the window station of the logged-on interactive user. Cross-window station access has always been frowned upon, as it's a security risk, but whereas previous Windows versions allowed some exceptions, these have been mostly eliminated in Vista and later operating systems.
The most likely reason your service is hanging on startup, is because it's trying to interact with a nonexistent desktop (or assumes Explorer is running inside the system user session, which also isn't the case), or waiting for input from an invisible desktop.
The only reliable fix for these issues is to eliminate all UI code from your service, and move it to a separate executable that runs inside the interactive user session (the executable can be started using the global Startup group, for example).
Communication between your UI code and your service can be implemented using any RPC mechanism: Named Pipes work particularly well for this purpose. If your communications needs are minimal, using application-defined Service Control Manager commands might also do the trick.
It will take some effort to achieve this separation between UI and service code: however, it's the only way to make things work reliably, and will serve you well in the future.
ADDENDUM, April 2010: Since this question remains pretty popular, here's a way to fix another common scenario that causes "service did not respond..." errors, involving .NET services that don't attempt any funny stuff like interacting with the desktop, but do use Authenticode signed assemblies: disable the verification of the Authenticode signature at load time in order to create Publisher evidence, by adding the following elements to your .exe.config file:
<configuration>
<runtime>
<generatePublisherEvidence enabled="false"/>
</runtime>
</configuration>
Publisher evidence is a little-used Code Access Security (CAS) feature: only in the unlikely event that your service actually relies on the PublisherMembershipCondition will disabling it cause issues. In all other cases, it will make the permanent or intermittent startup failures go away, by no longer requiring the runtime to do expensive certificate checks (including revocation list lookups).
I faced this problem because of a missing framework on the box running my service. The box had .NET 4.0 and the service was written on top of .NET 4.5.
I installed the following download on the box, restarted, and the service started up fine:
http://www.microsoft.com/en-us/download/details.aspx?id=30653
To debug the startup of your service, add the following to the top of the OnStart() method of your service:
while(!System.Diagnostics.Debugger.IsAttached) Thread.Sleep(100);
This will stall the service until you manually attach the Visual Studio Debugger using Debug -> Attach to Process...
Note: In general, if you need a user to interact with your service, it is better to split the GUI components into a separate Windows application that runs when the user logs in. You then use something like named pipes or some other form of IPC to establish communication between the GUI app and your service. This is in fact the only way that this is possible in Windows Vista.
In service class within OnStart method don't do huge operation, OS expect short amount of time to run service, run your method using thread start:
protected override void OnStart(string[] args)
{
Thread t = new Thead(new ThreadStart(MethodName)); // e.g.
t.Start();
}
I'm shooting blind here, but I've very often found that long delays in service startups are directly or indirectly caused by network function timeouts, often when attemting to contact a domain controller when looking up account SIDs - which happens very often indirectly via GetMachineAccountSid() whether you realize it or not, since that function is called by the RPC subsystem.
For an example on how to debug in such situations, see The Case of the Process Startup Delays on Mark Russinovich's blog.
If you are using Debug code as below in your service the problem may arise.
#if(!DEBUG)
ServiceBase[] ServicesToRun;
ServicesToRun = new ServiceBase[]
{
new EmailService()
};
ServiceBase.Run(ServicesToRun);
#else
//direct call function what you need to run
#endif
To fix this, while you build your windows service remove #if condition because it didn't work as it is.
Please use argument for debug mode instead as below.
if (args != null && args.Length > 0)
{
_isDebug = args[0].ToLower().Contains("debug");
}
In my case the problem was missing version of .net framework.
My service used
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
</startup>
But .net Framework version of server was 4, so by changing 4.5 to 4 the problem fixed:
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0" />
</startup>
Copy the release DLL or get the dll from release mode rather than Debug mode and paste it to installation folder,,it should work
I was running into a similar problem with a Service I was writing. It worked fine then one day I started getting the timeout on Start errors. It happened in one &/or both Release and Debug depending on what was going on. I had instantiated an EventLogger from System.Diagnostics, but whatever error I was seeing must have been happening before the Logger was able to write...
If you are not aware of where to look up the EventLogs, in VS you can go to your machine under the Server Explorer. I started poking around in some of the other EventLogs besides those for my Service. Under Application - .NETRuntime I found the Error logs pertinent to the error on startup. Basically, there were some exceptions in my service's constructor (one turned out to be an exception in the EventLog instance setup - which explained why I could not see any logs in my Service EventLog). On a previous build apparently there had been other errors (which had caused me to make the changes leading to the error in the EventLog set up).
Long story short - the reason for the timeout may be due to various exceptions/errors, but using the Runtime EventLogs may just help you figure out what is going on (especially in the instances where one build works but another doesn't).
Hope this helps!
Install the debug build of the service and attach the debugger to the service to see what's happening.
I want to echo mdb's comments here. Don't go this path. Your service is not supposed to have a UI... "No user interaction" is like the definining feature of a service.
If you need to configure your service, write another application that edits the same configuration that the service reads on startup. But make it a distinct tool -- when you want to start the service, you start the service. When you want to configure it, you run the configuration tool.
Now, if you need realtime monitoring of the service, then that's a little trickier (and certainly something I've wished for with services). Now you're talking about having to use interprocess communications and other headaches.
Worst of all, if you need user interaction, then you have a real disconnect here, because services don't interact with the user.
In your shoes I would step back and ask why does this need to be a service? And why does it need user interaction?
These two requirements are pretty incompatible, and that should raise alarms.
I had this problem and it drove me nuts for two days…
If your problem similar to mine:
I have settings “User settings” in my windows service, so the service can do self-maintenance, without stopping and starting the service. Well, the problem is with the “user settings”, where the config file for these settings is saved in a folder under the user-profile of the user who is running the windows service under the service-exe file version.
This folder for some reason was corrupted. I deleted the folder and service start working back again happily as usual…
I had this problem, it took about a day to fix. For me the problem was that my code skipped the "main content" and effectively ran a couple of lines then finished. And this caused the error for me. It is a C# console application which installs a Windows Service, as soon as it tried to run it with the ServiceController (sc.Run() ) then it would give this error for me.
After I fixed the code to go to the main content, it would run the intended code:
ServiceBase.Run(new ServiceHost());
Then it stopped showing up.
As lots of people have already said, the error could be anything, and the solutions people provide may or may not solve it. If they don't solve it (like the Release instead of Debug, adding generatePublisherEvidence=false into your config, etc), then chances are that the problem is with your own code.
Try and get your code to run without using sc.Run() (i.e. make the code run that sc.Run() would have executed).
This problem usually occurs when there is some reference missing on your assembly and usually the binding fails at the run time.
to debug put Thread.Sleep(1000) in the main(). and put a break point in the next line of execution.
Then start the process and attach the debugger to the process while it is starting. Press f5 after it hit the break point. It will throw the exception of missing assembly or reference.
Hopefully this will resolve this error.
Once try to run your exe file. I had the same problem, but when I ran it direct by double click on the exe file, I got a message about .Net framework version, because I was released the service project with a framework which it wasn't installed on target machine.
Took me hours, should have seen the event viewer get_AppSettings().
A change in the app config, caused the problem.
Adding 127.0.0.1 crl.microsoft.com to the "Hosts" file solved our issue.
My issue was due to target framework mentioned in windows service config was
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6"/>
</startup>
and my server in which I tried to install windows service was not supported for this .Net version.
Changing which , I could able to resolve the issue.
I had a similar issue, steps I followed:
Put a Debugger.Launch() in the windows service constructor
Followed step by step to see where it got stuck
My issue wasn't due to any error.
I had a BlockingCollection.GetConsumingEnumerable() in the way that caused the windows service to wait.
I had this problem too. I made it to work by changing Log On account to Local System Account. In my project I had it setup to run as Local Service account. So when I installed it, by default it was using Local Service. I'm using .net 2.0 and VS 2005. So installing .net 1.1 SP1 wouldn't have helped.
Both Local System Account and Local Service would not work for me, i then set it to Network Service and this worked fine.
In my case, I had this trouble due to a genuine error. Before the service constructor is called, one static constructor of member variable was failing:
private static OracleCommand cmd;
static SchedTasks()
{
try
{
cmd = new OracleCommand("select * from change_notification");
}
catch (Exception e)
{
Log(e.Message);
// "The provider is not compatible with the version of Oracle client"
}
}
By adding try-catch block I found the exception was occuring because of wrong oracle version. Installing correct database solved the problem.
I also faced similar problem and found that there was issue loading assembly. I was receiving this error immediately when trying to start the service.
To quickly debug the issue, try to run service executable via command prompt using ProcDump http://technet.microsoft.com/en-us/sysinternals/dd996900. It shall provide sufficient hint about exact error.
http://bytes.com/topic/net/answers/637227-1053-error-trying-start-my-net-windows-service helped me quite a bit.
This worked for me. Basically make sure the Log on user is set to the right one. However it depends how the account infrastructure is set. In my example it's using AD account user credentials.
In start up menu search box search for 'Services'
-In Services find the required service
-right click on and select the Log On tab
-Select 'This account' and enter the required content/credentials
-Ok it and start the service as usual
In case you have a windows form used for testing, ensure that the startup object is still the service and not the windows form
We have Log4Net configured to log to a database table. The table had grown so large that the service was timing out trying to log messages.
open the services window as administrator,Then try to start the service.That worked for me.
Build project in Release Mode.
Copy all Release folder files to source path.
Execute Window service using command prompt window in administrative access.
Never delete files from source path.
At lease this works for me.
Release build did not work for me, however, I looked through my event viewer and Application log and saw that the Windows Service was throwing a security exception when it was trying to create an event log. I fixed this by adding the event source manually with administration access.
I followed this guide from Microsoft:
open registry editor, run --> regedit
Locate the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\Application
Right-click the Application subkey, point to New, and then click Key.
Type event source name used in your windows service for the key name.
Close Registry Editor.

Resources