I have a test hub .Net application which can fire off different selenium tests.
I'm looking for a good way to record the results of that specific test and tell the user if there has been an error.
I've added the "-log selenium.log -browserSideLog line" but, can I access the log programatically?
The idea being that at the tear down stage I can look at the entries and search for "Error" and report this to the user.
I know there is the "RetrieveLastRemoteControlLogs" command but that only gets part of log.
if you know PowerShell, then you can to use cmdlet get-content.
In .Net you can to use methods from the class File MSDN, but i dont work with it.
you can run "get-content" after each test and check whether there are errors
if i have not answered your question, then you can add tags, because it is not only selenium-rc`s question.
Related
What is the best elegant approach for debugging a large E2E test?
I am using TestCafe automation framework and currently. I'm facing multiple tests that are flaky and require a fix.
The problem is that every time I modify something within the test code I need to run the entire test from the start in order to see if that new update succeeds or not.
I would like to hear ideas about strategies regard to how to debug an E2E test without losing your mind.
Current debug methods:
Using the builtin TestCafe mechanism for debugging in the problematic area in the code and try to comment out everything before that line.
But that really doesn't feel like the best approach.
When there are prerequisite data such as user credentials,url,etc.. I manually Declare those again just before the debug().
PS: I know that tests should be focused as much as possible and relatively small but this is what we have now.
Thanks in advance
You can try using the flag
--debug-on-fail
This pauses the test when it fails and allows you to view the tested page and determine the cause of the fail.
Also use test.only to to specify that only a particular test or fixture should run while all others should be skipped
https://devexpress.github.io/testcafe/documentation/using-testcafe/command-line-interface.html#--debug-on-fail
You can use the takeScreenshot action to capture the existing state of the application during the test. Test Café stores the screenshots inside the screenshots sub-directory and names the file with a timestamp. Alternatively, you can add the takeOnFails command line flag to automatically capture the screen whenever a test fails, so at the point of failure.
Another option is to slow down the test so it’s easier to observe when it is running. You can adjust the speed using the - speed command line flag. 1 is the fastest speed and 0.01 the slowest. Then you can record the test run using the - video command line flag, but you need to set up FFmpeg for this.
I'm running tests in Visual Studio using "Test Explorer" with NUnit and a .runsettings file (specified by choosing the option in the GUI "Select Settings File")
My settings file (called mytests.runsettings) is:
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<RunConfiguration>
<DisableAppDomain>True</DisableAppDomain>
</RunConfiguration>
<ForceListContent>true</ForceListContent>
<NUnit>
<DomainUsage>None</DomainUsage>
</NUnit>
</RunSettings>
I have verified that it is loading this file (verified by adding the framework node and setting it to a fake version, which then results in an error).
But no matter what I do, it doesn't run without an AppDomain!
Running from the command line does work:
nunit3-console.exe --domain=None --inprocess MyTests.dll
What do i need to do to get it to use that setting in NUnit?
I don't believe you can do that.
There is no recognized DomainUsage element under NUnit in the runsettings file. DomainUsage is an internally used property, which will be honored if set. But you can't set it that way. Your DomainUsage element is simply being ignored.
If the adapter received your DisableAppDomain setting, then it would set DomainUsage to None. However, I don't believe it is actually receiving it.
Point 2 requires some explanation. Note that I haven't worked on the adapter for a few years and I'm going from memory but here it is...
The DisableAppDomain setting was added to allow Visual Studio to force NUnit to try to run without using an AppDomain. The Test Explorer is supposed to set things up so that it's possible to run that way, i.e. by making sure everything is already available in the current domain.
In order to prevent misuse of the feature, I believe that Test Explorer always overrides any user-provided setting. Again, this is from memory of work that was done a few years ago, but it seems as if the results you are seeing validate it.
The rationale for this past decision was that Test Explorer is completely responsible for setting up the Process and AppDomain used to run the tests. The user has no way to impact that and neither does NUnit. Of course, when using the console runner, that's not the case - control is in the hands of the user.
Something else to investigate is why you feel the need to run without a test AppDomain being created. But that's probably another question. :-)
I'll ask some other folks who may have a better memory than I to look at this as well.
UPDATE:
#Terje, who maintains the adapter now, replied and confirmed that there is no way to set DomainUsage in the runsettings file or any other way we know of when running under the test adapter. The docs have been corrected to avoid the implicit suggestion that it's possible.
We believe, but have not confirmed experimentally, that TestExplorer creates it's own AppDomain whenever it uses this setting to suppress its creation by the test adapter.
This answer is a follow up to the answer from #charlie above, just need some more space here.
I've checked on the domains that are being created, based on whether the DisableAppDomain is set or not.
When the disable app domain is not set, the appdomain is created by NUnit with applicationbase and friendly name as shown below:
The NUnitCheckDomain is the test dll.
When the disable app domain is set, NUnit no longer sets its own domain, and you then see it runs under the testhost appdomain, which is the process that runs all tests:
So this seems to work the way it should and can.
Do you need it to run under some other app domain than one of these?
BTW: If I run the NUnitCheckDomain test assembly using the NUnit3 console, then it a) works when run directly b) crashes when used with the parameters you give above (domain=None and --inprocess) with not able to load NUnit.Framework. #charlie - Any reason this should not work the same using NUnit3-Console ?
We are having to use the command configuration (to be able to specify the runsettings file that I want to use at the time of running the job) for vstest.console. But now it does not create the results within bamboo. like the vstest.console bamboo task does.
First question, does the mstest parser create that kind of results? Is there a way to do this after running the command prompt. Also, since they both create trx files, can I use mstest parser for the trx created by vstest.console?
Second question, I don't see in my log that it kicked off that step. Is there anything special I need to do to have it kicked off? Especially if the previous step fails?
Also, the .trx logger setting is not renaming the trx to my TestResult.trx file name.
I looked in the list of apps to see if there is a vstest.console version of the parser, there is not. We are using version 6.6.3 ..so we are a bit behind but not sure if this means anything with the parser.
Must have had to do a reboot. I don't know why, but I came back in after the weekend, and the servers have been rebooted, and with no change to what I said above, the results are showing up as expected.
When I run tests in test explorer individually they all run without errors.
When I click on "Run All" I get varying test that fail.
My question:
Does anybody know what the minimum requirements regarding tests to be able to run them with "Run All"? Like maybe they all have to in a certain way with a certain command?
Running test manually one by one or run all has nothing to do with your tests being failed. You need to check the failure reason for each test and solve it.
Maybe you can post the error messages here so we can help you better.
I discovered that I had a static variable still containing a value other than that expected. Now the outcome is always the same regardless of how I run it.
Sorry for troubling you all
I would like to be able to email my automation test results to my team. I am not sure how to get this done so a point in the right direction would be really helpful. I have found code that will be able to send out an email, however to fill the body with the results I would need to know the file that xCode is storing them in, which I am not able to find.
This is just ideas that I am throwing out there, if there is a better way to go about this I am all ears.
Thank you for your help
ui automation test results are in plist format (TestSummaries.plist) so you need to parse this plist file to extract the test result.
We are using trainer gem to convert results to junit xml format. I think it would be good starting point for you to use above gem and attach xml file in your email.
You could look into running your suite on Jenkins server. There is an option you can set up to send result to an email address. However be warned that since xcode 8 has been release Jenkins and Xcode are not the best of friends.