Svchost.exe high CPU usage - EC2 - Windows Server 2008 r2 - amazon-ec2

I am running Windows Server 2008 r2 on AWS. Today I see that svchost.exe takes 100 % cpu. I have right click on svchost.exe and choose go to process in Taskmanager. but there is no process high lighted.
I see that "Beyond Compare" as a description for svchost.exe. Please refer the below screenshots and help me to fix the issue.
[Task Manager][1]

I have checked the Svchost.exe file location and it was C:\ProgramData\Microsoft\National. I could able remove the exe file but it generates automatically. I suspect that it might be some virus, I have used McAfee for scanning the folder C:\ProgramData. But it does not detects anything.
I have checked the startup applications in "msconfig" and I couldn't find any thing different. But I can see that the service National Instruments Domain Service is running in services tab (hiding all Microsoft services). I Google it and I couldn't find any problem with that service. But I could see that the exe file name is differ from what I have it on my server for National Instruments Domain Service. The location and file name for National Instruments Domain Service in my server is C:\ProgramData\National\taskngr.exe.
I Google it and came to know that taskngr.exe might be a virus. So I stopped and disabled National Instruments Domain Service ( file location C:\ProgramData\National\taskngr.exe) after that I can able to kill the Svchost.exe process permanently and CPU usage came normal.
I have removed two folders C:\ProgramData\National and C:\ProgramData\Microsoft\National permanently from the server. Also I have uninstalled the National Instruments Domain Service ( Service name is National).
And one more thing I noticed that in Task Manager, All other svchost process are running as svchost.exe ("s" small) in my case it was Svchost.exe ("S" caps).
Thanks.

Update you Windows 2008 Server to latest

Related

Random 1104 Error Reading File Errors in Multiple VFP Applications

We have multiple applications developed in Visual Foxpro 8.0 running in a data center on Windows 2008 R2 on VMware. We also have a Citrix farm on the same network where users run yet another VFP 8.0 application in Citrix sessions. All applications share the same set of data tables located on a file server (also Windows 2008 R2 VM). Virtual hosts are connected by 10Gb LAN (managed switch).
Since mid-July we started seeing random 1104 "Error reading file..." errors on multiple different applications on multiple servers. All of them reference different files on the file server.
The problem started mid-July and it frequency gradually increased. Earlier it was most frequent in the afternoons by 3 pm, now it happens from early morning till late afternoon. It affects EDI servers (these run batch jobs in unattended mode) and Citrix servers and a variety of applications. It occurs when a VFP application (any of them) tries to open a database container file or individual tables most often with USE command but some times executing a SQL Select statement, or when loading a VFP form that opens tables in DataEnvironment
We caught a moment when the same exact error happened on two different servers running different applications at the same exact moment (up to a second). We also saw two different applications running on the same computer erroring out at the same moment.
We replaced the file server with a new virtual machine with no relief (we since changed it back to the old file server ).
We disabled the antivirus.
We updated VMware on all hosts to the latest version.
Sysinternals Process Monitor displays "INVALID_NETWORK_RESPONSE" event when the error occurs.
We captured traffic on both the server side and client side when the error occurred and had it analyzed by a network analysis specialist. He observed a peculiar pattern, where client OS starts retrieving the file in question from the file server AFTER VFP application had thrown an error. It seems that VFP application requests a file from OS, then it either gets an abnormal response or just times out and only after that the OS sends packets requesting the file. Again, this happens sporadically.
OpLocks and SMB2 have been disabled on all computers both on the server and client side of the equation for many years and everything was running smoothly until now...
Any advice would be greatly appreciated.
My first piece of advice would be to re-enable OpLocks and SMB2. There is no reason to mess with either of those items as things stand today and you are losing a huge amount of performance running at SMB1 level.
In my experience these issues have almost always been caused by one of the following.
Antivirus/antimalware software.
Replication or online backup software like MozyPro.
The Windows Search indexing service.
You should consider installing the Windows 7 / Server 2008 R2 Enterprise Hotfix Rollup if you haven't already.
That problem mostly related by SMB2!
Some Antivirus Software!
Windows updates! If you use VFP apps by DBF/DBC file. Do not update your system/OS. That is my personal suggestion. Windows Server 2012+ or Windows 10+ prorbably would big problems at near future.
And the point high probably is:
What is your I/O request per secs? if your IO request bigger than 1000~2000 per secs for a dbf file that is a bottle neck; and your storage device is HDD -> you need to switch/update your HDD to SSD. I suggest m.2 pro series SSD.

java.net.SocketException: No buffer space available (maximum connections reached?): JVM_Bind

Tomcat is running a webapp under Windows. After a few days (under very low load), the exception mentioned in the title starts to appear in the logs, no new connections can be established from that point on, the only fix is then to reboot the server.
Environment:
Latest Tomcat 6
Windows Server 2008 R2
JDK 6 update 30
SQL Server 2008
Kerberos authentication
Evidence collected so far:
netstat shows no excessive amount of connections
ProcessExplorer shows no excessive amount of open file handles
system main memory usage is average
JVM heap usage is average
restarting Tomcat does not solve the problem
Open questions:
if we were leaking connections, shouldn't they show up in netstat?
shouldn't a restart of the appserver resolve the problem, because the OS should free all process resources?
is there a way to trace the problem to its origin? E.g. installing monitoring software, maybe something similar to lsof etc.?
I'm out of ideas, any hints appreciated!
The reason we got this error is a bug in Windows Server 2008 R2 / Windows 7. The kernel leaks loopback sockets due to a race condition on machines with more than one core, this patch fixes the issue:
http://support.microsoft.com/kb/2577795
I was running Alfresco Community 4.0d on Windows 7 64 bit and had the same symptoms and errors.
The problem was fixed with Microsoft's patch: "Kernel sockets leak on a multiprocessor computer that is running Windows Server 2008 R2 or Windows 7" (http://support.microsoft.com/kb/2577795) (ie. Buddy Casino's answer (see below)).
Another observation I'd like to add is that Windows connections (Internet Explorer, Remote Desktop etc) would work again about 5-10 mins after the Alfresco services were shutdown.
Alfresco is an excellent product and I was afraid I would have to scrap it. Fortunately stackoverflow came to the rescue !
Thanks again to Buddy Casino's answer.
Boo to the person who down-voted the Question.
We are seeing the same thing on a similar setup, W2008R2, Tomcat 6.0.29, Java 1.6.0.25. Restarting tomcat does not help, but restarting the server itself does, at least for a while. After the last time we started shutting down individual services and believe we have it narrowed down to either an instance of Alfresco that is also running on the server or the Backup Exec Agent services. After those services (four in total) were stopped, the applications in Tomcat started working again, although we were still seeing the buffer/connections error in the stdout log which was strange. Will need to wait for the problem to return before confirming which are the culprit, which could be anywhere from a few days to a week or more.
Any chance you are running either Alfresco or BE on your server?

"No printers are installed." problem when using excel 2003 interop sheet.PageSetup through a WCF service

We have a WCF service that generates an Excel file off of a template and feeds it back to the client in a byte array. For some reason, we are getting an error when we try to do this:
sheet.PageSetup.CenterHeader = sheet.PageSetup.CenterHeader.Replace("[customerName]", customerName).Replace("[dateTime]", date.ToShortDateString());
When it hits this line, we get this error message:
No printers are installed. To install a printer:
In Microsoft Windows 2000, click Start, point to Settings, and then click Printers. Double-click Add Printer.
In Microsoft Windows XP, click Start, and then click Printers and Faxes. Under Printer Tasks, click Add a printer.
Follow the instructions in the wizard.
The service runs under the LocalSystem account. When we first deployed this service, it was to an x64 machine and had the same issue. Our dev environment is x86 so we moved the service to another server that was x86 and it worked fine for a while. It recently stopped working and is now giving us this error message again. I am pretty sure it was becuase of a mass windows update that was done on the app server recently.
Funny thing is, I tried changing the service to "allow interaction with the desktop" and it didn't work, but when our sysadmin did that same thing, it worked for one file generation but is not working now.
There are printers installed on the machine, both on domain accounts and local accounts. I've also tried running the service under a different account, but then I wasn't able to connect to the service due to some SSPI error. We tried using the Network Service account but then the app couldn't see the service at all. We've restarted the service after each change also to no avail.
What I do know about the "PageSetup" part of the sheet object is that it needs to have a printer installed to access it at all, even if you aren't printing. I was unable to figure out a way to trick the machine into thinking the LocalSystem account has a printer installed.
We figured out the problem, I had my config file still pointed at the x64 server :(
As to why it won't work on x64 (to my knowledge) - since office 2003 is x86, it needs an x86 print spooler in order to work correctly. I don't know of any way to get an x86 print spooler on an x64 OS so we just stuck it on the x86 server.

Permissions issue when publishing to WMI under network service account

I'm adding WMI publishing to a .net framework 3.5 based windows service that is running under the 'network service' account.
According to a document I came across on MSDN, the 'network service' account should by default have WMI publishing permissions. ("By default, the following users and groups are allowed to publish data and events: ... Network Service, ...")
However, when the service calls Instrumentation.Publish(myStatusClassInstance), it throws a DirectoryNotFoundException;
System.IO.DirectoryNotFoundException was unhandled
Message: Could not find a part of the path 'C:\Windows\system32\WBEM\Framework\root\MyWMINamespace\MyService_SN__Version_1.0.3686.26280.cs'.
..so it looks like System.Management.Instrumentation tries to generate code on the fly, and when running under network service it targets a directory where network service has no permissions.
What is the best fix/workaround for this? Can I override the code-gen target dir in app.config or in code? I don't want to have to fiddle around with file system permissions when deploying the service...
Update: I think this is a 'feature' where older FX code clashes with newer security settings in Win7. Internally the WMI managed classes retrieves the WMI installation directory from registry, and uses that as the output path for generated code. Unfortunately a lot of users are not allowed to (or supposed to) write stuff under %SystemRoot%... ...I filed a connect bug (#530392) to see if MSFT can bring any clarity and/or provide a fix or workaround.
Update 2: I'm guessing that for normal user accounts this is not an issue, because UAC virtualization will kick in and store the files elsewhere. However, apparently the 'network service' account is not covered by UAC virtualization..(?)
Update 3: Added 550pt bounty. Simple constraints: .net framework 3.5 based windows service, running as network service, need to be able to publish data through WMI using System.Management.Instrumentation on Win7 and Win2008[RTM & R2] with default permissions/security settings and without resorting to modifying framework internal/private members using reflection. 'Out-of-the-box' but clean solutions welcome. Will open a second related bounty-Q as a placeholder for another 550pt if SO allows.
Bounty update: I intend to double the bounty for this Q through a second hand-in-hand question that will serve as a bounty placeholder:
https://stackoverflow.com/questions/2208341/bounty-placeholder ( <-- Apparently this was not allowed, so the bounty placeholder question got closed by the SO etiquette police.)
Update 4: This gets better and better. I noticed that installutil was writing the missing files to c:\windows\syswow64...etc..., so I realized that I was using the 32-bit version of installutil to install the service, but the service was running as a 64-bit process. The obvious side effect was that code generated when installutil was running ended up under syswow64 (the 32-bit system directory), while the service was looking for it under the 64-bit system directory (system32). (<-- off topic, but I really like how MSFT managed to switch around the names there... :) ).
So I tried installing the service with the 64-bit version of installutil. That failed miserably with permission errors in the %sysroot%\wbem\framework...etc... path. Next I recompiled the service as x86 and registered it again using the 32-bit version of installutil. That resulted in an entirely new exception:
System.Exception: The code generated for the instrumented assembly failed to compile.
at System.Management.Instrumentation.InstrumentedAssembly..ctor(Assembly assembly, SchemaNaming naming)
at System.Management.Instrumentation.Instrumentation.Initialize(Assembly assembly)
at System.Management.Instrumentation.Instrumentation.GetInstrumentedAssembly(Assembly assembly)
at System.Management.Instrumentation.Instrumentation.GetPublishFunction(Type type)
at System.Management.Instrumentation.Instrumentation.Publish(Object instanceData)
at SomeService.InstanceClass.PublishApp(String name) in e:\work\clientname\SomeService\SomeService\WMIProvider.cs:line 44
at SomeService.SomeServiceService..ctor() in e:\work\clientname\SomeService\SomeService\SomeServiceService.cs:line 26
at SomeService.Program.Main() in e:\work\clientname\SomeService\SomeService\Program.cs:line 17
...getting closer...
I believe the problem is not with publishing data, but with registering that type in WMI for the first time.
If you examine the System.Management.Instrumentation code in reflector, or some other disassembler, you'll see that wen the assembly that is about to publish hasn't been registered, then the code will try to register the assembly and save the assembly info in a specially named sub directory under the WBEM installation folder.
I suspect that if you run code to publish the WMI data as an administrator first, it would register the assembly and then the Network Service account would have the permissions to do the normal publishing.
Have you inspected your assembly with the installutil? That should give you a log of the installation issues. (But since you can't run it as the Network Service account, it might not show the problem you're having.)
Also, are you sure this service must be run under the Network Service account?
Because of the vulnerability risk in running Windows services in privileged accounts, Microsoft has made these special service accounts with some limitations, which were strengthened in Vista and Win7. Since Vista, Microsoft has limited the number of services running under this account in favor of less-privileged ones (see this article). The Network Service account (aka "NT AUTHORITY\NETWORK SERVICE") can access the network (acting as the local machine account PCNAME$), but it has reduced rights on the local machine (unlike the Local System account).
Have you checked the WMI security permissions for the branch your assembly is using? Run wmimgmt.msc and dig in... When I did a quick check of some random branches, I could see that the Network Service account did not have write rights.
Lastly, I would suggest using Sysinternals' ProcMon, which would allow you to filter to just that process and see if there are any Access Denied errors in file or registry settings. This tool has solved many problems for me over the years.
Not sure if you raised it or someone else but please have a look:
http://connect.microsoft.com/VisualStudio/feedback/details/530392/wmi-publishing-fails-on-permission-error-please-provide-a-way-to-override-codepath-in-system-management-instrumentation-schemanaming-in-app-config-web-config
This may help you to understand the root cause of the issue better

How can I permanently bypass Windows XP startup?

I have an application for Windows XP. This application is deployed with the hardware. The application is the only application that ever runs on these machines. These machines are never connected to the internet. I'm interested in instant-on (or quick-on) options that bypass the Windows XP startup for these machines.
This is similar to Windows XP "hibernation", but not exactly. With hibernation, the memory state is only read from disk once - the very next time the system is turned on. I want a memory state permanently stored to disk, so the system always starts from that same spot every time, regardless of how it was shut down. How can I achieve this?
Sounds like you're looking for the Hibernate Once, Resume Many feature of Windows Embedded.
If you like "hibernation", you may use VMWare.
Install a ArchLinux and VMWare on the host machines.
Prepare your Windows XP as the guest OS.
Customize startup process of the host, let it run VMWare and restore Windows to the snapshot.
Hmmm, the short answer is "not easily!", but one way could be to try playing around with replacing the windows shell with your own application / script that launches your own custom interface / state instead of Explorer as the default. Basically it's done using this reg key:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\shell
Info on this actually seems to be reasonably scarce after doing a quick Google search, but this link below provides a little more detail:
http://www.trap17.com/index.php/how-change-windows-xp-shell_t20367.html
I think if you do a Google Groups search on "Windows XP shell replacement" you might get some more informative results.
You could try installing TweakUI on them, and having them autologin. Once you do that, just add your application to the Start Up menu (or in the registry, under:
HKLM\Software\Microsoft\Windows\CurrentVersion\Run
Jeff Atwood has a post on this very thing.

Resources