I am searching a way to assign failed unit tests to resolvers. The way Sonar raises an issue at a class level whenever one or more unit tests fail does not fit my needs, I would like to assign a specific test to a specific developer.
Since Sonar can raise an issue for unit tests failures and is able to determine which particular test case failed I wonder if there is a way I could assign each failed test case to a different developer rather than the whole test class. And if it is possible, how can I do such a thing ?
This is not possible for the moment. You can indeed assign only the issue that tells how many errors (or failures) you have on a specific file. Most of the time, this will work out of the box as most teams try to avoid having people working on the same class at the same time. But it's true that this can happen.
My email notifier in teamcity works well when the compilation fails. I have it set to notify me of failures and also the first success after the failure.
But when one test fails, I don't get an email. The build is set to fail when at least one test doesn't pass. I have nice red dot next to the build number.
The weird thing is, sometimes I get a success email after the test has been fixed, but not always. Does anyone know of any logs I can use to figure out why I'm not getting my email?
Here is another weird thing: I set the notifications to send me an email when the build fails and when the build succeeds. When the tests fail, I still get no email.
I do get notified properly in the windows try program, however.
There is teamcity-notification.log, especially useful if you enable DEBUG logging:
http://confluence.jetbrains.net/display/TCD65/Reporting+Issues#ReportingIssues-LoggingEvents
Are the rules for email and tray notifiers the same? If yes, most probably that is a transportation failure, the error must be in the log.
BTW there is TeamCity 7.0 RC1 available:
http://confluence.jetbrains.net/display/TW/TeamCity+EAP
I'm dumb, they were in my spam folder all along.
I'm using a VS2010 loadtest against a website, but the site being tested is throwing some errors (eg, SiteUnavailable or other general site-specific errors).
The loadtest continues execution even if an error is returned in the response - so our .NET server logs are showing many errors for a single user session - and the subsequent errors may well be caused because we are trying to continue a web journey that should really have ended.
So is it possible to end the erroring user session as soon as an error is hit in a loadtest without ending the whole loadtest? I would then want the virtual user to continue with another new web journey.
My loadtest is not scripted (it's using the default view) as I read somewhere that loadtests are less efficient when scripted.
However I can't see a setting that would enable me to do what I want, so I'm thinking that scripting would be the way to go.
Any pointers/suggestions gratefully received.
Dave
In case anyone else needs the answer to this, I had it answered via the MS forum. There is a setting in the webtest "StopOnError" - this should be set to True and will end the webtest running, NOT the loadtest, if an error occurs. This setting avoids the chain of potentially unrelated errors that may occur as a result of a single error.
In Visual Studio 2010, I have a number of unit tests. When I run multiple tests at one time using test lists, I sometimes reveive the following error for one or more of the tests:
The agent process was stopped while
the test was running.
It is never the same test failing, and if I try to run the test again, it succeeds.
I found this bug report on Connect, which seems to be the same problem, but it does not offer a solution.
Has anyone else seen this behaviour ? How I can avoid it ?
Edit
I am still experiencing this bug, and so is many of my colleagues on the same software/hardware setup. I have evaluated the answers so far, but they don't resolve the problem. I am starting a bounty for a solution to this problem.
This message is caused by an exception on a thread different from the executing test thread. All answers so far boil down to this simple explanation. It is a known bug in Visual Studio not to display any sensible information in that case.
Visual Studio’s test runner totally chokes if a thread other than the executing test thread throws an exception: It gets swallowed and there’s no output, no chance to intercept and debug and no nothing except a burned-down smoldering mess that was supposed to be your unit test.
I've just experienced the similar problem: some tests fail and they are different in different test runs. I don't know exactly the reason why it happens, but it began to occur when I added a finalizer to one of my classes. When I disable the finalizer - the problem disappears. When I turn the finalizer on - the problem comes back.
Right now I don't know how to overcome this.
I was having this problem, and it turned out to be a problem in my code which the Test Framework wasn't catching properly. A little accidental refactoring had left me with this code:
public void GetThingy()
{
this.GetThingy();
}
This is of course an infinite recursion, and caused a StackOverflowException (I guess). What this caused was the dreaded: "The agent process was stopped while the test was running."
A quick code inspection showed me the problem, and my tests are now running fine. Hope this helps - might be worth inspecting the code looking for issues, or maybe extracting a bit into a console app and checking it works properly there.
I was able to find the source of my problem by looking in the test result file (/TestResults/*.trx) It provided the full details of the exception that occurred in the background thread, and once I resolved that exception the "agent processed stopped..." error went away.
In my case I was unintentionally launching the GUI in my unit test, which eventually caused a System.ComponentModel.InvalidAsynchronousStateException to be thrown.
So my .trx file contained:
<RunInfo computerName="DT-1202" outcome="Error" timestamp="2013-07-29T13:52:11.2647907-04:00">
<Text>One of the background threads threw exception:
System.ComponentModel.InvalidAsynchronousStateException: An error occurred invoking the method. The destination thread no longer exists.
at System.Windows.Forms.Control.WaitForWaitHandle(WaitHandle waitHandle)
at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
at System.Windows.Forms.Control.Invoke(Delegate method)
...
</Text>
</RunInfo>
This didn't provide any information on what test caused the error, but it did show me where the exception was, which was very useful.
This message is typically generated when the test process crashes and can happen when there is an unhandled exception on a background thread, a stack overflow occurs, or an explicit call to Process.GetCurrentProcess().Kill() or Environment.Exit. Another possible cause is an access violation in unmanaged code.
Something no one has mentioned is that there may be additional information in the event log. Usually you will not get much information on why the test crashed in the results, however in the event of an unhandled exception on a background thread, then the test framework writes details to the Application event log with source VSTTExecution. If there is no information written to the event log then it is likely one of the other causes listed above.
I encountered the same Problem and solved it while Removing
Environment.Exit(0);
So i am pretty sure, that this error occurs while your test or method under test, is causing the executing process to terminate.
In my case the solution was resolved by checking the Output Window.
'QTAgent32.exe' (Managed
(v4.0.30319)): Loaded
'C:\TestResults\bdewey_XXXXXX072
2011-01-11
17_00_40\Out\MyCode.dll',
Symbols loaded. E, 9024, 9,
2011/01/11, 17:00:46.827,
XXXXX072\QTAgent32.exe, Unhandled
Exception Caught, reporting through
Watson: [Exception message]
In my case I had a FileSystemWatcher that was throwing an error on a seperate thread.
Thanks for posting the question. I just ran into this problem and figured out a cause that you may be running into.
An asynchronous exception may have
occurred
During my test setup, I create an object that queues a worker thread in the thread pool. If I run through debugging fast enough my code passes.
If the worker thread kicks off and has an error BEFORE the test setup completes, then I get a result of Aborted with no reasoning.
If the worker thread kicks off and has an error AFTER the test has begun, then I get a result of : Error - The agent process was stopped while the test was running.
Important to note: this is a component that I use throughout several of my tests. If the test framework encounters too many of these errors it aborts the rest of the tests.
Hope this helps
I added try/catch blocks to the descructor ~ClassName(){} that were defined in any class involved in my tests. This fixed the problem for me.
~MyClass()
{
try
{
// Some Code
}
catch (Exception e)
{
// Log the exception so it's not totally hidden
// Console.WriteLine(e.ToString());
}
}
For finding out where the exception was thrown click on the hyperlink "Test Run Error" next to the exclamation icon in the Test Results window. A window with the stack trace is opened.
This helps a lot to track down the error!
I had the same problem and it was caused by a finalizer for an unmanaged resource (a file writer that was not getting disposed properly for some reason).
After wrapping the finalizer code in a try-catch that swallows the exception, the problem disappeared. I don't recommend swallowing exceptions like that, so it would obviously be wise to find out why the exception is occurring in the first place.
I have had this happening on the odd occasion, and the culprit almost always turns out to be threading.
Strangely enough all the tests would work fine on the development machines, then randomly fail on the build servers.
On closer inspection it turned out that although the tests were being listed as passed on the dev boxes, there were exceptions being thrown. The exceptions were being thrown on a seperate thread which didn't get picked up as an error.
The exception details were being logged against the test trace, so we were able to identify which code/tests needed to be modified.
Hope this helps someone.
In my case I had some unit-tests for a WCF-service. This WCF service was starting up 2 timers.
Those timers caused side effects.
--> I disable these timers by default and everything is fine!
BTW: I use WCFMock to fake the WCF service, so I have "real" unit tests around my WCF service
This error was caused by a Finalizer for me as well.
The Finalizer was actaully calling some DB code which wasn't mocked out. Took me a while to find it as it wasn't a class I wrote and the reference to it was burred quite a few classes deep.
I have run into a similar problem where a test is failing in TestInitialize and is also running code from a ddl from another of my projects. I get the error message as described above and if I try to debug the test, the test is just aborted without any exception details.
I suspect that the problem may be that the dlls from my other project are from a Visual Studio 2012 project and I am running my tests in a VS2010 project, and/or possibly that the UnitTestFramwork dll versions from the 2 projects are mismatched.
The problem can also be triggered by an Exception or Stackoverflow in Constructor of a TestClass.
As this error can have many different causes, I'd like to add another one for completeness of this thread.
If all your tests are aborting as described by the OP, the cause might be a wrong project configuration. In my case the target framework was set to .NET Framework 3.5. Setting it to a higher version through the project properties page (tab Application) resolved the issue.
I was able to determine what was causing my issue by looking in the Windows Logs > Application log entries within the Windows Event Viewer. Look for entries at the time the test bombed-out. I had an Error entry similar to below:
QTAgent32_40.exe, PID 10432, Thread 2) AgentProcess:CurrentDomain_UnhandledException: IsTerminating : System.NullReferenceException: Object reference not set to an instance of an object.
at XXX.YYY.ZZZ.cs:line 660
at XXX.YYY.AAA.Finalize() in C:\JenkinsSlave\workspace\XXX.YYY.AAA.cs:line 180
It was indeed a null reference exception within a method called from a class finalizer.
For anyone happening upon this old question and wondering what's being thrown from their thread(s), here's a tip. Using Task.Run (as opposed to, say, Thread.Start) will report child thread exceptions much more reliably. In short, instead of this:
Thread t = new Thread(FunctionThatThrows);
t.Start();
t.Join();
Do this:
Task t = Task.Run(() => FunctionThatThrows());
t.Wait();
And your error logs should be much more useful.
I am going to create a typical business application that will be used by a few hundred consultants. Normally, the consultants would be presented with an error message with a standard text. As the application will be a complicated one with lots of changes being made to it constantly I would like the following:
When an error message is presented, the user has the option to "send" the error message to the developers. The developers should be able to open the incoming file in i.e. Eclipse and debug the steps of the last 10 minutes of work step by step (one line at a time if they want to). Everything should be transparent, meaning that they for example should be able to see the return values of calls to the database.
Are there any solutions that offer such functionality today, my preferred language is Python or also Java. I know that there will be a huge performance hit because of such functionality, but that is acceptable as this kind of software is not performance sensitive.
It would be VERY nice if the database also had a cronology so that one could query the database for values that existed at the exact time that a specific line of code was run in the application, leading up to the bug.
You should try to use logging, e.g. commit logs from the DB and logging the user interactions with the application, if it is a web application you can start with the log files from the webserver. Make sure that the logfiles include all submitted data such as the complete GET url with parameters and POST with entity body. You can configure the web server to generate such logs when necesary.
Then you build a test client that can parse the log files and re-create all the user interaction that caused the problem to appear. If you suspect race conditions you should log with high precision (ms resolution) and make sure that the test client can run through the same sequences over and over again to stress those critical parts.
Replay (as your title suggests) is the best way to reproduce an error, just collect all the data needed to recreate the input that generated a specific state/situation. Do not focus on internal structures and return values, when it comes to hunting down an error or a bug you should not work in forensic mode, e.g. trying to analyse the cause of the crash by analyzing the wreck, you should crash the plane over and over again and add more and more logging/or use a debugger until you know that goes wrong.