I work on automated tests for DirectX based applications that run on various automated servers. The servers are setup to run as a service, which means when the tests execute they are executed under session 0.
However, the tests started failing after installing the windows platform update kb2670838. When initializing the application, a DXGI_ERROR_NOT_CURRENTLY_AVAILABLE error occurs after a call to CreateSwapChain.
I have found that some API calls do not work under session 0. (See http://msdn.microsoft.com/en-us/library/windows/desktop/hh404562(v=vs.85).aspx).
But last week this worked. I posted this in various Microsoft forums, but it seems no one else is having this problem. So my question is has anyone else experienced this? And how did you work around it?
Related
I have a cross-platform desktop application written in Xamarin.Forms that runs both in Windows and MacOS. I want to do some UI automation on top of that application.
After some research it seems like the most cross-platform friendly option is to use Something like Sikuli. As the default stack on our team is centered on the .NET stack we want to use SikuliSharp or Sikuli4Net to perform the automated UI tests.
However, despite of the fact that we've been able to run Sikuli4Net successfully on Windows, automating several flows so far. We have a dire situation on MacOS. Our team (myself included) doesn't have a lot (or maybe any) knowledge of Java applications.
I've installed the JDK 8, but was unable to run the tests in the same way we did on Windows. The code builds, but it seems like something in the environment is lacking.
With Sikuli4Sharp when starting the APILauncher like this:
launch = new APILauncher(true);
launch.Start();
I get the following error:
With SikuliSharp when trying to run a simple demo application on our software I have this error:
I have tried to set up the SIKULI_HOME environment variable using this answer as a reference, but still the same problem (and I did restart the console and IDE, even my machine).
When I run echo $SIKULI_HOME on the terminal I do get the directory that cointains the .jar files:
So I'm kind of lost about were to go from here. These problems made me unsure about being possible to run Sikuli4Net or SikuliSharp on MacOS environments. Is this the case? If not, what am I doing wrong?
as mentioned in the error message: sikuli-script.jar is missing.
You have to check on what version of Sikuli/SikuliX your SikuliSharp or Sikuli4Net are depending on.
In doubt you have to dive into the sources of those Net packages.
We have a testing framework made in AutoIt for our Windows apps (older legacy apps that we will continue supporting). These have never been run on a schedule or part of CI (always been run manually). I tried to get some kind of auto run (or even just status reporting) out of the tests, with minimal luck.
I have a VM where the tests can run. I experimented with my own web app, which works okay locally for running and status reporting. But when set up on the server, AutoIt reports it cannot open the application. Same thing happens if I try to run the tests from a .bat file.
My current solution is to have AutoIt call my web app to report status (working okay locally, untested on the server), or to see if I can get AutoIt to report results back to TeamCity. I have the agent installed but when I run the build from TeamCity, AutoIt reports it can't launch the application. I tried this while logged into the VM, logged out, with RDS open; no luck.
Is it possible to run the tests manually from the VM and send results back to TeamCity? When I run them from TeamCity it reads the AutoIt output (which is in the expected format), but I need to let TeamCity know to update the results (so we can use TeamCity rather than my web app to show the results).
I may need to find a way to let TeamCity know a build has been started, which might then let it know to keep an eye on the process' output, but I'm not positive. Any ideas?
I solved this so it could be done more traditionally.
If anyone is confused by what "running the agent from the console" meant, it just means installing the agent without selecting the "as a service" checkbox, and then manually starting the agent by cd'ing to the BuildAgent/bin directory and running the command agent start. I also created a batch file that will do this automatically (but you must run it as admin).
Further, I found AutoIt couldn't do anything if the test doesn't run from the right directory, so I had to devise a solution to this.
The only issue now is that I have to have an old laptop always connected to the virtual machine the tests and app run on (since the AutoIt tests won't work without the VM desktop being interactive).
My team is developing a desktop application (mixed C++/Tcl) that is used in a client-server setup. Currently it is Windows-only, but soon we will need to port it to Linux. CruiseControl.NET builds it every night from the source code in SVN and packages it into NSIS installer, but we have no automated tests to run.
It is nearly impossible to add any unit tests, but integration testing of the application is easy, because it is already heavily script-based.
The main task is to install the app into 3 PCs, configure it (that involves copying some files around), run it, monitor a possible crash, wait till integration testing is done, collect a summary, send emails. It could be done with a bunch of custom PowerShell scripts, but
In future we will want to add more features and more testing, and
what used to be a simple script soon blows up (as usual), so I want
to minimize custom scripting, and if I need to script something, I
prefer bash/cygwin (I am not familiar with Python or Ruby).
I want a web dashboard that will report current progress, and if
something failed - show logs
I need some supervisor that will monitor the app under test and
report if it hangs or crashes
we will need to test it also on Linux
ideally I would like to orchestrate some test steps between the PCs
(e.g. run test X on PC1 and test Y on PC2 in parallel, wait till they
both finish, then run test Z on PC1, while monitoring that nothing
crashes on PC2 etc)
So, I am looking for a COTS tool/set of tools that will help me to do it and don't have a steep learning curve. Ideally, for free, but if it is really good and has fair pricing, my company may purchase a license.
The process should be triggered from CruiseControl.NET when the NSIS installer is ready, and then perform everything described above. Basically, it should allow at least remote installation of software, running custom scripts and have a web dashboard.
Apparently, SCCM tools like Chef could be used, but so far neither of them supports a Windows server, only nodes. I would like to avoid setting up a Linux VM just for that, although I can do it, if I have no other choice. Also, Chef seems to be a bit overkill - good for 10k machines, but I have only 3... maybe 5 in future. And I am particularly curious about chances to orchestrate a distributed test.
Most of the similar questions here on StackOverflow and in internets are about web apps, Java containers, Maven etc, and there are just so many tools and plugins for these tools to evaluate.
Thanks in advance.
Install ccnet on your test machines. Have those ccnet projects listen to a file that gets edited when a new installer is ready. Have the test machines install that new installer and run tests. There you go. ccnet sends emails so there's your basic reporting.
Have the test results get reported into a database via web services using gSOAP(that's what we did). For linux you can run java cruisecontrol if you must. Write a gSOAP enabled test controller program to report the test results from the test machines. A little c++ app will do. Then write a website(we use ASP.NET) to query the database(Postgresql) and show results. Have the test machines auto update themselves via SVN to get the latest changes to the configuration. Use Nant. Nant is far superior to just using ccnet to run tasks. Nant works through ccnet. Use XML, XSL and CSS with ccnet to make test emails have the information you want(new passes, new failures, SVN differences to code bases, etc...)
Our latest development is putting a big TV in the kitchen with a summary of test results so people can know more readily what they broke!
The first thing I'd get working is a test machine listening for the new installer, installing it, running some basic tests and emailing the results back. Put the ccnet and nant configuration in version control and get that auto updating on the test machine so you don't have to log into every test machine and do an update every time you make a change.
This is hugely broad and pretty close to opinion based. Chef can handle steps like deploying the application to the test machines but it isn't a GUI test framework so you would need something else to handle that. Jenkins supports distributing tests to windows hosts so that seems like a good choice on that side of things but it isn't that great at multi-node tests or orchestration between them. I suspect you'll need to write most of this yourself given the requirements.
I am planning to use the Prism for Windows Runtime in my Windows 8.1 Store App. Run the Windows App Certification Kei 3.1 over Prism for Windows Runtime, "EventAggregatorQuickStart". The test failed at following:
Failed at Crashes and Hangs
◦Executable D:\Code Samples\WindowStoreApp81\Prism for Windows Runtime QuickStart\EventAggregatorQuickstart\EventAggregatorQuickstart\bin\Debug\AppX\EventAggregatorQuickstart.exe was detected by Windows Error Reporting and experienced a crash or hang.
◦Application eventaggregatorqs_1.1.0.0_neutral__8wekyb3d8bbwe was detected by Windows Error Reporting and experienced a crash or hang.
◦Crash dump file C:\Users\Silong\AppData\Local\Microsoft\AppCertKit\EventAggregatorQuickstart.exe.3896.dmp was created by Windows Error Reporting and provides additional information.
Failed at Performance Suspended
◦Application Error: Application Launch was not detected for application App. This could be because your application failed to launch correctly. Please consider re-running the test and avoid interacting with the application while tests are running.
Any comment on this matter is much appreciated. Thanks.
I answered a similar question posted in the Q&A of Windows Store Apps MSDN site. I wonder if you posted it:
EventAggregatorQuickStart failed the Windows App Certification Kit 3.1 Test
A QuickStart solution is intended to show how a particular service or pattern work in a simple and intuitive way. Therefore, some other requirements that may be needed for a complete certified Store App might be excluded from the solution as that would not be the QuickStart purpose.
Anyway, you may consider running the certification tests on the AdventuresWorkShopper RI solution, which implements EventAggregation service among others, and it may be considered as a complete Windows Store App. This solution passes the certification tests.
Note: I was able to run all certification tests with Windows App Certification Kit 3.1 over the EventAggregator QuickStart and "Performance" and "Crashes and hangs" tests PASSED successfully. I would recommend you to re-run the test avoiding application and system interaction as much as possible.
I hope this helped, regards.
I am trying to run unit tests for Mac application in xcode 5.0 .
The unit tests run fine if I have the sandbox disabled but as soon as I enabled the sandbox the tests are unable to run, failing with error loading bundle (IDEBundleInjection.c: Error loading bundle).
I've seen a few other reports about this error and looks like the general assumption is that it won't work. The suggested workaround is to create a new configuration in xcode with sandboxing disabled and running the tests with that configuration.
This works, of course, but it seems like a poor workaround since it will be running the tests in a different environment than the real application normally would and potentially missing on a whole category of bugs resulting from different permissions, directories etc.
I can hardly believe that Apple intended developers to disable sandboxing for testing, seems very short sighted. Has anyone figured out a better solution for running unit tests for an application that is intended to run in sandbox mode?