I am developing automated tests in Visual Studio 2010 (Ultimate). I can execute my test locally trough VS and remotely on a test agent through MS Testing Center. Everything works well.
The problem for me now is that I want to deploy test data with my tests. This test data is ordinary image files that is to be processes by the application I am testing. Deploying these files works fine as long as I'm running the test locally. I can either use the "Deployment" section in my Local.testsettings file, or I can use the DeploymentItems attribute in my code. Of course - the files that I want to deploy are included in my project file.
However, when I execute my test through the test agent - it will not deploy any files. And as I said, the test itself works fine both locally and remote.
I have been facing the same problem recently.
It sounds like you have the DeploymentTest attribute like this:
<DeploymentItem #"MyProject.Tests\TestDataSubfolder>\", "OutputDir">
Just change it to:
<DeploymentItem #"TestDataSubfolder>\", "OutputDir">
(That means remove the project folder, since the project folder does not exit on the remote machine).
Related
I am trying to use Specflow with Playwright in order to do BDD on a portal app developed but I am facing a small problem.
The Specflow project is a separate project with the ASP.Net core server that has the Api of the portal app (it is in Vue). Since the tests are pointing to a specific URL (currently localhost), before running the tests, I need to run the ASP.Net core & Vue project locally. Otherwise, Specflow & Playwright will not be able to do the test (as it will not find the localhost).
Is it any way I can force the run of the Web Server project? I tried to run it from outside Visual Studio with dotnet build and then dotnet run commands but somehow they are missing parameters (that exist while running it from inside VS) and apart from that, these commands must somehow be triggered while trying to run the tests.
I have seen solutions like creating a Docker image from a Docker Compose file in order to pack a .Net project & server in it before running the Specflow tests. Then in the BeforeTestRun hook using the FluentDocker to spin-up the server but I am not quite sure it is the easier (or best) solution.
Does anyone know how I can trigger running the .net core project (with the Vue pages)?
This is actually a pretty big question, with a pretty big answer, however this is well-trodden ground. The issue isn't so much a "specflow" issue as a general automated testing issue. Development practices like continuous integration and continuous delivery can help. Each one is too big for a single question, however I can answer this in more general terms.
In its simplest form, running automated tests locally involves these steps:
Build the application
Deploy the application to a real web server
Run tests
I'm going to assume you are developing in a Windows environment, however every operating system has some sort of command line scripting solution available. The scripting language might change, but the overall idea will not.
Configure a web server. In Windows, this would be Internet Information Services (IIS).
Add a new "application" (or "IIS app" as some people call it) to your localhost web server. Point the physical directory to the root directory for the web project. Repeat this for each web site or web app your system requires.
Write a PowerShell script that gives you an easy way to build and deploy the applications to your local web server.
This script should use publish profiles set up in Visual Studio, which allows you to publish directly from Visual Studio before invoking tests manually through Test Explorer.
Write a PowerShell script used has a "harness" script to coordinate building, deploying locally, and then invoking dotnet test.
Running tests locally just requires a single line of PowerShell to invoke your test harness script:
.\Scripts\Run-Tests.ps1 -solutionDir . -tags BlogPosts,Create
# Skip deploying in case web apps haven't changed:
.\Scripts\Run-Tests.ps1 -solutionDir . -tags BlogPosts,Create -deploy:False
I'm trying to do something incredibly trivial I thought but apparently this needs to be hard. And yes there are bits and pieces throughout stack overflow but they're either out of date or don't actually work.
I've got an asp.net core site that I've dockerized with the add/docker/linux command.
In VSTS I can build the image and publish it with 2 docker-compose items.
And then I can release the image with the release management.
What I can't figure out how to do:
run dotnet test on my image and report the results to VSTS
Setup environment variables on Azure App Service Container that get properly passed into the image when its run.
On #1, I cannot find any up-to-date documentation on how to set it up so that while developing unit tests don't run unless specifically specified (and if I tell it to run tests in visual studio they should run in the docker image! I can get them to run always, but that's a waste of time while developing if they run every time you start debugging!).
And I cannot figure out how to use either docker-compose or the new VS.net 2017 15.8 way with just docker run commands to run the tests. It seems to me that I would need a new dockerfile just for the tests to run and have it generate and then discard the image that was created. But I can't figure out how to do this or even if this is the right way.
How should this be setup to do unit tests? (I've gone through 5 pages of google search results and none of them work right.)
On #2, setting and application setting in the App Service does not pass the values in docker run. I've tried everything and they never get passed. How do you pass environment variables on Azure so that the run command gets the right -e parameters?
For#1 you could use dotnet test command. This will generate a .trx file that VSTS can pick up and render a nice test report. You just need to setup the “Publish Test Results” task.
dotnet test --logger trx --results-directory /var/temp
More details please take a look at this blog: Running your unit tests with Visual Studio Team Services and Docker Compose
For#2 not totally get your point, if you want to override environment variable values on VSTS and use the value on Azure App Service Container. Please try this solution through powershell script: How to override values of environment variables on VSTS tasks
Beside suggest you also go through this blog shows how Docker Deployment to Azure App Service (Linux) using VSTS including both CI and CD. Which maybe helpful to you.
I am looking to be able to run my postman scripts using newman during a TeamCity build.
Instead of deploying the build to a test environment, I'd like to run the postman scripts on that particular build, so it isn't deployed to an environment used by other developers which could potentially break it.
My current build chain in TeamCity is:
Build main project (contains the REST Api and all required code)
Run Postman scripts using Newman on that project
I have the collection and environment file, along with the CLI command to call it. When I try and point the environment for a local build, it does not work.
I am thinking of running an IIS Express server on the agent and then with that active port, run the tests but I have been unsuccessful.
Any ideas on how to approach this would be appreciated!
I have looked at How do I integrate my Postman Integration Tests with TeamCity and this uses a test environment, which is not what I am after.
I looked at https://ie.com.au/a-how-set-up-automated-api-testing and this was helpful, but I think this is still reliant on setting up a test envrionment.
TeamCity isn't really equipped to handle what you are trying to do. You are trying to run API tests against a build, in order to do that, you'll need an environment. You need something to run your project in order to query against it.
The only potential path you might try looking at is containerizing your project, in docker or something similar, then running your image after it's built and querying against that. However this isn't a great practice and bloats the build time.
A good practice would be to build your project > deploy it to a test environment, you should set up a separate 'test' or 'dev' environment that is ok being broken > after deploy trigger a service to run your tests against the 'dev'
I work with Coded UI Tests and Visual Studio 2013. Now I have to test if images from a folder are shown correctly inside the application. That’s why I have created a folder containing the images and set build action to none and deployment to always.
Unfortunately all tests are executed in an own test results folder and my images are not deployed correctly. I know I could do this by using the DeploymentItem attribute or a testsettings file but I don't want to do this. I want to avoid the "test results" folder and run the tests from within the output folder of my test project.
I do this with the unit tests for instance. They are written with XUnit.Net which works just fine. I thought it would also work with MS Test but this only seems to work for Unit Tests but not for Coded UI Tests.
So, to sum it up again: How can I get rid of the “Test Results” folder when using Coded UI Tests with Visual Studio 2013 and run my tests simply from within the output directory of the project?
You can do this using a custom entry in the .runsettings file I think.
DeploymentItemAttribute Class
Consider running your unit tests directly in the build output
directory, so that testing runs more rapidly. This is especially
useful on the build server after you have checked in your tests. To do
this, add a .runsettings file to your solution, include
False, and select the file in
the Test, Test Settings menu. The same effect occurs in any test run
in which DeploymentItemAttribute is not used at all.
However, you
might prefer not to do this if you want to be able to inspect the data
files after a failed run.
You cannot avoid using a deployment folder
if you are using a .testsettings file, which is required for web and
load tests, coded UI tests, and any test in which you deploy an
application to remote machines.
As for DeploymentItemAttribue itself I would steer clear of it unless the following issues have been fixed, Gotchas: MSTest’s [DeploymentItem] attribute. +1 for xUnit in TestDriven.NET.
You can try deleting the LocalTestRun.testrunconfig file from the Solution Items folder (directly under the solution). When we did that it started using the \bin\debug\ folder instead of the TestResults folder when running our unit tests using MSTest.
I have a solution with a web application and test project. The test application has coded ui tests of the web application. How can I get the web application to start up when the tests execute?
I've tried
adding a TestSetup method to start the web server via System.Diagnostics.Process. I don't like this because it seems clunky and the coded ui tests don't lend themselves to inheritance real well
adding a setup script in the testsettings to start the web server (same command line) via a batch file. However the batch file never exits and eventually times out. I've tried starting the web server via the start command, but that didn't work -- it works great when you run it from the command line, but not from the testsettings setup script.
I've thought about maybe just running IIS in the background and pointing it to a specific folder. Then deploy the web application to that folder. And let the test project test against IIS.
This seems like a lot of work though. There's got to be an easier way. I want to do this on several developer's machine and also on the TFS build machine via scheduled or continuous integration.
Any thoughts?
Thanks
I configure the web project (in the project properties) to be hosted on IIS. Its just 3 or 4 clicks away. It will prompt to create a virtual application. After that, build and run your project and it will be permanently hosted on IIS. So you can start Coded UI tests without manually starting the web server.
You could do what Gerardo Grignoli suggested and just host the site in IIS so that it's always on.
You could also just start the web project (Without Debugging if you want to debug your tests) and then run/debug the tests. In your tests, just use BrowserWindow.Launch("localhost:appPort/whateverPageToStartOn"); just fill in the port instead of appPort based on your IIS express settings.
In regards to running it on the TFS build machine, that seems like probably a questionable thing to do. You can test the deployed site (dev, test, prod, whatever) from a testing lab. Microsoft Test Manager (MTM) is one of the products that can do this.