I have some PLC code which crashes two different PLCs.
On one PLC (CX2040 x64) whenever I activate the solution, the CPU cores where the project is running on, jumps to 99%. At this time it seems to be stuck in some infinite loop. When this happens, it is impossible to get the TwinCAT runtime into config mode. Neither from the visual studio remote manager, neither if I would remote desktop onto the PLC itself and try to set the TwinCAT run time into config mode using the tray icon.
This is very frustrating. But even more frustrating is that occasionally the PLC would become totally unresponsive. No connection was possible from the Visual Studio remote manager or normal windows remote desktop. The PLC didn't even respond to ping requests from another PC. The only solution was to power cycle the PLC and let it reboot.
On a second PLC (CX2040 x86) I get the following TcRTime watchdog exception:
This PLC never jumps to 99% CPU usage and no hardware reset is needed, because it never goes into a non-recoverable state. I can just put it back into config mode, since it always goes into the exception state. How can I enable the TcRTime watchdog on the x64 PLC, to prevent that one from going into the unrecoverable state.
I know there are some functions which enable a watchdog on the PLC, but these functions are not used in this project! So I have no clue how the watchdog is activated on one PLC, but not the other since the code is the same. Does anyone know?
Turns out I completely forgot that I had enabled the watchdog cycles in the project.
However, I do not understand why the watchdog cycles were triggered on the x86 PLC, but not the x64 one. Maybe on the x64 one the 99% CPU usage didn't cause cycle overruns? Although that seems very unlikely to me.
Related
I'm using Visual Studio 2012 Update 3. When I open a project, VS automatically creates a process called <myproject>.vshost.exe, even before starting to debug.
When I start debugging and later close the debug application, most of the times the <myproject>.vshost.exe process closes as well. When this happens, devenv.exe starts taking up 3x more memory than normal and CPU goes up to 25% (on a i7 Quadcore with 8GB ram) for around 1 minute. At the end of 1 minute, a new <myproject>.vshost.exe opens up (even though I'm not debugging), CPU falls back down to 0 and memory drops back down as well.
Trying to start debugging whilst the CPU is at 25% and <myproject>.vshost.exe is not running in the background will results in the solution building but the debug does not start.
If I wait until the CPU falls back down and <myproject>.vshost.exe process is running again then I can start debugging normally.
This happens to me 80% of the times after closing the application I am debugging. The remaining 20% of the times when I stop debugging <myproject>.vshost.exe continues running in the background and I am able to start debugging again immediately after with no delay.
This also happens regardless of code changes in between debugs.
This is a new install of VS2012 U3, I've tried resetting all settings and disabled ReSharper but still no joy.
I don't want to disable vshost debugging because of the features I would lose.
Has anyone else encountered this problem before? Is this a known issue? Are there any solutions/workarounds?
EDIT
I changed the platform from Any CPU to x86 and it appears to work properly, but I still can't understand why I shouldn't be able to debug it as Any CPU. Even though this might be the workaround I'd still be interested in knowing whether this is a known issue and if there are other (better) solutions.
By 'working properly', I mean that when I stop debugging the vshost doesn't close, in fact it never closes, but the CPU of devenv stays at 0% and it allows me to start and stop debugging as many times as I want one right after the other.
EDIT2
Actually it appears that changing the platform to x86 only worked properly for a while, after about 20 rebuilds it is now doing the same as leaving it as Any CPU.
On another note, closing and opening VS makes no difference.
I've ended up formatting the computer again and re-installing everything from scratch. For not it appears to be working fine, let's see how long this lasts for.
I know this is several months old but I'm trying to post this answer in a few places since this is what was causing my ills: I had the Data Sources toolbox open in Visual Studio 2012. Once I closed that, it seemed to eliminate the lengthy delays when switching windows. You may also want to close Server Manager if you don't need it open.
I know this is old Post but, I think I need to share my solution for everyone.
and this is my first post so, please improve the answer if I miss something.
I have the same issue with Visual Studio 2012, When I try to build or debug it will use CPU up to 100%.
So, I try below steps to reduce the CPU usage while debugging:
Please close unnecessary opened file.
Hide unnecessary debugging panels like: breakpoints, auto, local, output, find symbol result, etc.
If still use high CPU then try to hide call stack panel.
Remove unnecessary breakpoints.
I am working on a Windows Embedded Standard 2009 project deploying on an Atom powered tablet. We have some known Windows bug check crashes (BSOD) that I am working through. We also have a bug where the tablet becomes unresponsive with the screen off; requireing a hard power cycle to recover. I am pursuing a theory that the unresponsive tablet is a BSOD crash that happened with the screen off. We have EWF turned on which prevents a memory dump from writing to know if a BSOD occurred. We turn the monitor off after user inactivity using user32.dll SendMessage(Handle, WM_SYSCOMMAND, SC_MONITORPOWER, MONITOR_OFF).
Will a Windows bug check (BSOD) turn on the monitor if it was turned off previously programmatically?
Thanks!
Can't speak for tablets, but the bug check isn't going to jump into a long series of power maangement routines, just to output the fact that it's exploded. The bug check does as little as is physically possible, since once you're in BSOD mode, the system is by definition already crashed and not stable and in a highly unknown state. Starting to call other complicated subsytems is not going to happen, as the BSOD may very well have happened in the very routine(s) it's trying to call.
No, bug checks do not turn on the monitor (doesn't matter if it went to sleep due to inactivity or your message).
Your best bet is to leave a kernel debugger attached.
While bugcheck does not go through any power management code, it does make operations that would usually wake a monitor up. Bug-check changes screen resolution and switches to text-mode. If you have a kernel-debugger attached (or just configured), the system waits for the kernel debugger response and will not display the blue-screen text until you hit "g".
In the default configuration it will also attempt to create a crash-dump and reboot. If you suspect a bugcheck look for memroy.dmp in the windows directory or connect a kernel debugger.
I've got a customer who told me that my program (simple user-land program, not a driver) is crashing his system with a Blue Screen Of Death (BSOD). He says he has never encountered that with other program and that he can reproduce it easily with mine.
The BSOD is of type CRITICAL_OBJECT_TERMINATION (0x000000F4) with object type 0x3 (process): A process or thread crucial to system operation has unexpectedly exited or been terminate.
Can a simple program be responsible for a BSOD (even on Vista...) or should he check the hardware or OS installation?
Just because your program isn't a driver doesn't mean it won't use a driver.
In theory, your code shouldn't be able to BSOD the computer. It's up to the OS to make sure that doesn't happen. By definition, that means there's a problem somewhere either in hardware or in code other than your program. That doesn't preclude there being a bug in your code as well though.
The easiest way to cause a BSOD with a user-space program is (afaik) to kill the Windows subsystem process (csrss.exe). This doesn't need faulty hardware nor a bug in the kernel or a driver, it only needs administrator privileges1.
What is your code exactly doing? The error message ("A process or thread crucial to system operation has unexpectedly exited or been terminate.") sounds like one of the essential system processes terminated. Maybe you are killing a process and unintentionally got the wrong process?
If somehow possible you could try to get a memory dump from that customer. Using the Debugging Tools for Windows you can then further analyze that dump as described here.
1Windows doesn't prevent you from doing so because it "keeps administrators in control of their computer". So this is by design and not a bug. Read Raymond's articles and you will see why.
Short answer is yes. Long answer depends on what is you program is suppose to do and how it does it?
Normally, it shouldn't. If it does, there must be either
A bug in the Windows kernel (possible but very unlikely)
A bug in a device driver (not necessarily in a device your program uses, this could get quite complicated)
A fault in the hardware
I would bet on option number two (device driver) but it would be interesting if you could get us a more detailed dump.
Well, yes it can - but for many different reasons.
That's why we test on different machines, operating systems, hardware etc..
Have you set some requirements for your program and is your user following them?
If you can't duplicate it yourself, and your program doesn't need admin to run, I'd be a bit suspicous about
The stability of that system's hardware
The virus/malware status of that system.
If you can get physical access to the client box, it might be worth running a full virus scan with an up-to-date scanner, and running a full memtest on it.
I had a system once that seemed stable, except that a certian few programs wouldn't run on it (and would sometimes crash the box). Memtest showed my RAM had some bad bits, but they were in higer sims, so they only got accessed if a program tried to use a lot of RAM.
No, and that is pretty much by definition. The worst thing that you can say is that a user-land application may have "triggered" a Windows bug or a driver bug. But a modern desktop Operating System is fully responsible for its own integrity; a BSOD is a failure of that integrity. Therefore the OS is responsible, and only the OS.
(Example of a BSOD bug that your application alone could expose: a virus scanner implemented as a driver, that crashes when executing a file from sector 0xFFFFFFFF, a sector that on this one machine just happens to contain one DLL of your application)
I had problems when exit my app without stopping all the processes and BD connections when the program ends (I crashed the entire IDE). I place the "stopping and disconnecting" code in the "Terminate" of "Form_Closed" event of my main form and the problem wa solved, I don't know it this is your situation.
Another problem can be if the user is trying to access the same resources your app is using (databases, hardware, sockets, etc). Ask him/her about what apps he/she is using when the BSOD happens.
A virus can't be discarded.
happens to me occasionally:
I start my program in visual studio and due to some bug my program goes into 100% cpu usage and basically freezes windows completely.
Only by utter patience requesting the task manager (takes forever to come up and paint itself) I can kill my process.
Do others encounter this too sometimes? Is there a clever trick to get this process down (other than pulling the plug and possible ruining files on the HD)? It now takes 5-10 minutes to kill it properly if the task manager is not accidentally present and I have to request this first
R
p.s. weird that a 'multitasking os' can still allow processes to eat up so much time that nothing else can be done anymore. My program doesn't even bump up it's thread priorities or anything
Check out Process Lasso
"Process Lasso is a unique new technology that will, amongst other things, improve your PC's responsiveness and stability. Windows, by design, allows programs to monopolize your CPU without restraint -- leading to freezes and hangs. Process Lasso's ProBalance (Process Balance) technology intelligently adjusts the priority of running programs so that badly behaved or overly active processes won't interfere with your ability to use the computer!"
http://www.bitsum.com/prolasso.php
I am not affiliated with Bitsum, just a user of their product, and it helps me solve this type of problems.
For what it's worth, I've never see this on either XP 64 or Vista 64, developing C++ apps in Visual Studio. Perhaps an OS upgrade is in order?
Edit: I use Process Explorer as a replacement Task Manager - it wouldn't surprise me if it did a better job of appearing in good time even when there's a rogue process running. And you can use it to boost its own priority.
I usually hit ctrl-alt-delete start the task manager sort by cpu find the offending process and right click and end the process..
task manager usually has enough priority to do this although it may be slow.
I think a shotgun to the head is the only way to be sure.
I generally don't see anything like this happen strictly as a function of an app that's eating 100% CPU. As part of stability / performance testing, I've gotten apps to cause Windows to get very slow, but this is usually done by writing heavily threaded apps (thus causing the O/S scheduler to thrash), or by writing apps that consume all available system memory or resources (much more impactful to the GUI apps than simply one thread that consumes its full share of processor time during its slices).
You say you get this behavior under Visual Studio? VS has a "Pause" button...
A couple of times recently I have noticed that 'something' is causing the Windows System Process to sit at 50+% and it will not quit until the PC is rebooted. Happening on Win2k and Win XP so far.
This is particularly troublesome because it currently appears to be triggered by MSVC 2005/Incredibuild and rebooting the build servers is not a nice thing.
At the same time the 'System Idle Process' process is holding the rest of the CPU and the build steps themselves seem to be starved. ie. a module that normally takes <5 minutes to compile is currently taking 20+.
I'd take a few guesses at maybe being virus checker or tortoise svn but would desperatly like some other suggestions.
Edit:
I've been experiencing this as something that is triggered, and the culprit may not be ongoing. Thats not to say that some other ongoing process hasn't done something 'stupid' and is managing an active lock up of System while appearing to be idle itself.
System (100% of 1 core), and System Idle Process are sharing 98-100% of the total CPU.
Occasionaly mt.exe, link.exe, buildservice would get a look in at 1-2%.
I'm running VNC to view the machine, so it's getting a look in on occasion.
Edit 2:
When left the previous evening the build process seemed to be progressing all be it slowly, but after waiting another 13 hours the 1 hour build process hasn't completed. System is still hogging the 1 core.
My understanding is that the "System" process is the time spent in the kernel (so performing disk I/O, network I/O (you did mention Incredibuild) and the like) -- I'd check for disk fragmentation, virus checkers and possibly look at these on other machines in your Incredibuild cluster.
As the System Idle process runs at "Low" priority, it's a red herring that it'd be "taking up CPU time" -- if anything it's just showing that there is available CPU time available. The fact the processing is stuck to a single processor shows that the process is doing something that is not multi-core aware, or someone has set it's thread affinity to 1.
I've noticed the virus checking software that I use can radically slow down compilation but it does not extend beyond the end of the build. Turning off advanced and heuristic checking improves this to the extent that I do not have to disable the scanner entirely. I have changed my scanning strategy such that I use scheduled full scans now more than advanced on the fly scanning, as it hurts the perfromance of a number of apps. (n.b. I am using the latest cut of Kaspersky). I'm also using an automated backup tool (AJCBackup) that also needs to be restrained when compiling.
You may also want to consider disableing the Windows Indexing service on drives that are be used to create a lot of temporary and object files, as it doesn't provide much value in this context for the amount of performance it draws.
Edit: Have checked which processes are actually hogging the CPU core and traced them back to a given app?
We've encountered issues with Kaspersky and Incredibuild in our offices - compiles and sometimes links will just hang and never finish.
Only seems to affect some machines though which is wierd, and only Windows XP (Vista seems immune from what I've seen).
Only solution I've found so far is to turn Kaspersky off entirely - so if you find a solution then let me know!
RE: smacl, work from the Windows Search/Indexing Service (WSearch) won't be attributed to the System process's CPU time, it should come from the SearchIndexer.exe/SearchFilterHost.exe services (Vista+).
The majority of activity from System you will see will be in disk activity from the lazy writer and other disk accesses. CPU activity from System will be because of kernel activity such as drivers (ISRs/DPCs) and other kernel-level filters (which could include AV file and process filters).
Process Explorer (http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx) can aid in viewing CPU usage across processes, including System. You can use the public Microsoft Symbol Server and this resource to get you started.
If you can take a trace with Xperf (http://msdn.microsoft.com/en-us/performance/cc825801.aspx), I can help you analyze where the CPU time is being spent in the System (kernel) context. Xperf isn't officially supported on XP, but you can take a trace on XP and analyze it on other systems.
Xperf and Process Explorer should be able to shine a spotlight on exactly the module(s) that are causing the runaway CPU usage. Symbols may not even be necessary to diagnose the problem; simply the module name can often point to the component in question that is slowing down your system. For example, high CPU usage from ndis.sys can point to network interrupts, or activity from modules such as aavmker4.sys can point to AV software (Avast! in this case).
And as always, check if there are any updated drivers and AV software for your system.
In my office, a conflict between Incredibuild and Spyware Doctor's Immunize feature caused similar issues. Turning off Immunize solved it for us.
What anti-virus/malware do you use?
I'm having same hangs when compiling using IncrediBuild in VS2003, on clean Windows 7 without any anti-virus. It worked fine on same box in XP and Vista.