I am load testing a server that requires that a user not be connected more than once at a time.
If I bind the VUsers to real users will this ever occur, or can I be sure that that VUser will not be reused until the previous iteration is complete?
I've created a load test to test this.
Roughly:
Method1 {
Trace.WriteLine(userId);
}
Method2 {
Trace.WriteLine(userId + "locked");
Thread.Sleep(5 min);
}
Mix these two up and you'll see that as soon as a UserId is locked up in Method2 you won't see it hit either method again for 5 min. And when all users are locked up the test just sits until one is released.
You can set the user that the load test connects as when you create a coded web test. Adding some code to get the user from a pool could work, but it would be challenging as the code to get a new user could easily become a bottle neck and will open to concurrency/multi threading bugs.
Related
I have a service that authenticates a user and I put delayElement for 2 seconds when it is about to return a 401.
When I run through artillery, it seems to start timing out some of the requests and I was wondering what kind of resources are used for delayElement as I was sort of hoping it just switches context away from the current request and goes to the next one that needs processing and just alarm itself back later to return the response.
private Mono<R> addDelaySpecifiedInServiceResponse(AuthServiceResponse<R> serviceResponse) {
return fromCallable(serviceResponse::getOperationResponse)
.delayElement(serviceResponse.getDelay(), Schedulers.newParallel("penalty"));
}
I tried different schedulers and seem to get a similar result.
delayElement in essence schedules a task on the provided Scheduler (or the common parallel scheduler by default), so yes it "alarms itself back later" if I understand your comment correctly.
I want to do load testing with Visual Studio but i don't get the idea how to setup load testing with authenticated users.
Imagine my scenario. This should be a quite common problem:
a website where you need to authenticate with username and password.
Perform a action that is only allowed for a authenticated user
What i have done so far:
i have already written UI tests with Selenium:
(this is working quite nice)
UPDATE: My Selenium test class: I want to use this code with the load test.This is a data-driven unit test project as you can see on method TestCase4529
[TestClass]
public class Scenario2
{
private IWebDriver driver;
public TestContext TestContext { get; set; }
[TestInitialize]
public void SetupTest()
{
this.driver = new ChromeDriver();
this.driver.Manage().Timeouts().ImplicitWait = new TimeSpan(0, 0, 30);
}
[TestCleanup]
public void TeardownTest()
{
this.driver.Quit();
}
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "|DataDirectory|\\4529.csv", "4529#csv", DataAccessMethod.Sequential), DeploymentItem("4529.csv"), TestMethod]
public void TestCase4529()
{
var userName = TestContext.DataRow["UserName"].ToString();
var password = TestContext.DataRow["Password"].ToString();
// UI Test logic
var loginPage = new LoginPage(this.driver);
loginPage.FillForm(userName, password);
loginPage.LoginButton.Click();
// Some assertions
}
}
Now when i setup the load test in Visual Studio i am asked how many users should do something:
I am not getting what this number means:
Does it only mean number of simultaneous threads?
How can i get a connection between a user (defined in the load test) an a authenticated user in my Selenium test?
What i would like to achieve:
Each user defined in the load test should be a authenthenticated user in my selenium UI test.
May somebody give me an idea how to do that or what i am thinking wrong...
A Visual Studio load test provides a way of running other tests repeatedly and simultaneously. It works best with Web Performance Tests but it can run unit tests and Coded UI tests.
When a "constant load" of (say) 25 users is selected then 25 test cases chosen from the "test mix" of the load test will be started. Whenever one of those test cases finishes another test will be chosen and started so that there are always 25 test cases being executed. That will continue until the end of the test run, which is normally either a test duration or a number of iterations. (Here "iterations" means number of test cases executed.)
Assuming "Web Performance Tests" are being used then those tests are responsible for providing the user authentication. A common way of doing that is to data drive the test and provide user names and corresponding passwords in that data. See here for more detail.
You question asks whether the "constant load" of 25 users means 25 threads. It means that 25 tests cases will be running at the same time, but it does not use Windows threads.
In response to comments:
I think you are misusing or misunderstanding the terminology of Microsoft's test environments. You may be able to have a Selenium test within the test mix of a load test although I have never done it. The user count and the data source are independent items. The user count is about how many simulated users are running tests at the same time. The data source is used by the test cases. If you have 25 users and one data driven test then that test should be started 25 times and those 25 executions should use the first 25 lines of the data source (assuming a Sequential or Unique access method).
To provide user and password you have to check for QueryString Parameters in your recorded webtest and pass data through Data Source see the following images for more detail:
Then pass the recorded webtest into load test as follow:
I have a very simple test cases(scalatest, but doesn't matter) and I provide two implementation of accessing some resources, this method returns either Try or some case class instance.
Test cases:
"ResourceLoader" must
"successfully initialize resource" in {
/async code test
noException should be thrownBy Await.result(ResourceLoader.initializeRemoteResourceAsync(credentials, networkConfig), Duration.Inf)
}
"ResourceLoader" must
"successfully sync initialize remote resources" in {
noException should be thrownBy ResourceLoader.initializeRemoteResource(credentials, networkConfig)
}
This tests testing different code which access some remote resources
Sync version
def initializeRemoteResource(credentials: Credentials, absolutePathToNetworkConfig: String): Resource = {
//some code accessing remote server
}
Async version
def initializeRemoteResourceAsync(credentials: Credentials, absolutePathToNetworkConfig: String): Future[Try[Resource]] = {
Future {
//the same code as in sync version
}
}
In IDEA test tab I see that future based version is twice slower then sync version, my question is there overhead for calling Await.result explicitly? If not, why it slows down the execution? Appreciate any help, Thanks.
Note: I know it is not the best way to measure performance of production system. But it at list says how much time was spend on each test case.
Yes, there will be a small overhead for Await.result, but in practice it probably doesn't amount to much. Future {} requires an ExecutionContext (thread pool or thread creator) in implicit scope so you won't be able to successfully use it without importing the default execution context (which will simply spawn a thread) or some other context. If you're using the default execution context, for example, you will have two threads instead of one which will involve some overhead for context switching. It shouldn't be much though. If 'twice as slow' means 40ms instead of 20 then perhaps it's not worth worrying about.
I'm currently using Calabash framework to automate functional testing for a native Android and IOS application. During my time studying it, I stumbled upon this example project from Xamarin that uses page objects design pattern which I find to be much better to organize the code in a Selenium fashion.
I have made a few adjustments to the original project, adding a file called page_utils.rb in the support directory of the calabash project structure. This file has this method:
def change_page(next_page)
sleep 2
puts "current page is #{current_page_name} changing to #{next_page}"
#current_page = page(next_page).await(PAGE_TRANSITION_PARAMETERS)
sleep 1
capture_screenshot
#current_page.assert_info_present
end
So in my custom steps implementation, when I want to change the page, I trigger the event that changes the page in the UI and update the reference for Calabash calling this method, in example:
#current_page.click_to_home_page
change_page(HomePage)
PAGE_TRANSITION_PARAMETERS is a hash with parameters such as timeout:
PAGE_TRANSITION_PARAMETERS = {
timeout: 10,
screenshot_on_error: true
}
Just so happens to be that whenever I have a timeout waiting for any element in any screen during a test run, I get a generic error message such as:
Timeout waiting for elements: * id:'btn_ok' (Calabash::Android::WaitHelpers::WaitError)
./features/support/utils/page_utils.rb:14:in `change_page'
./features/step_definitions/login_steps.rb:49:in `/^I enter my valid credentials$/'
features/04_support_and_settings.feature:9:in `And I enter my valid credentials'
btn_ok is the id defined for the trait of the first screen in my application, I don't understand why this keeps popping up even in steps ahead of that screen, masking the real problem.
Can anyone help getting rid of this annoyance? Makes really hard debugging test failures, specially on the test cloud.
welcome to Calabash!
As you might be aware, you'll get a Timeout waiting for elements: exception when you attempt to query/wait for an element which can't be found on the screen. When you call page.await(opts), it is actually calling wait_for_elements_exist([trait], opts), which means in your case that after 10 seconds of waiting, the view with id btn_ok can't be found on the screen.
What is assert_info_present ? Does it call wait_for_element_exists or something similar? More importantly, what method is actually being called in page_utils.rb:14 ?
And does your app actually return to the home screen when you invoke click_to_home_page ?
Unfortunately it's difficult to diagnose the issue without some more info, but I'll throw out a few suggestions:
My first guess without seeing your application or your step definitions is that #current_page.click_to_home_page is taking longer than 10 seconds to actually bring the home page back. If that's the case, simply try increasing the timeout (or remove it altogether, since the default is 30 seconds. See source).
My second guess is that the element with id btn_ok is not actually visible on screen when your app returns to the home screen. If that's the case, you could try changing the trait definition from * id:'btn_ok' to all * id:'btn_ok' (the all operator will include views that aren't actually visible on screen). Again, I have no idea what your app looks like so it's hard to say.
My third guess is it's something related to assert_info_present, but it's hard to say without seeing the step defs.
On an unrelated note, I apologize if our sample code is a bit outdated, but at the time of writing we generally don't encourage the use of #current_page to keep track of a page. Calabash was written in a more or less stateless manner and we generally encourage step definitions to avoid using state wherever possible.
Hope this helps! Best of luck.
My User model has a nasty method that should not be called simultaneously for two instances of the same record. I need to execute two http requests in a row and at the same time make sure that any other thread does not execute the same method for the same record at the same time.
class User
...
def nasty_long_running_method
// something nasty will happen if this method is called simultaneously
// for two instances of the same record and the later one finishes http_request_1
// before the first one finishes http_request_2.
http_request_1 // Takes 1-3 seconds.
http_request_2 // Takes 1-3 seconds.
update_model
end
end
For example this would break everything:
user = User.first
Thread.new { user.nasty_long_running_method }
Thread.new { user.nasty_long_running_method }
But this would be ok and it should be allowed:
user1 = User.find(1)
user2 = User.find(2)
Thread.new { user1.nasty_long_running_method }
Thread.new { user2.nasty_long_running_method }
What would be the best way to make sure the method is not called simultaneously for two instances of the same record?
I found a gem Remote lock when searching for a solution for my problem. It is a mutex solution that uses Redis in the backend.
It:
is accessible for all processes
does not lock the database
is in memory -> fast and no IO
The method looks like this now
def nasty
$lock = RemoteLock.new(RemoteLock::Adapters::Redis.new(REDIS))
$lock.synchronize("capi_lock_#{user_id}") do
http_request_1
http_request_2
update_user
end
end
I would start with adding a mutex or semaphore. Read about mutex: http://www.ruby-doc.org/core-2.1.2/Mutex.html
class User
...
def nasty
#semaphore ||= Mutex.new
#semaphore.synchronize {
# only one thread at a time can enter this block...
}
end
end
If your class is an ActiveRecord object you might want to use Rails' locking and database transactions. See: http://api.rubyonrails.org/classes/ActiveRecord/Locking/Pessimistic.html
def nasty
User.transaction do
lock!
...
save!
end
end
Update: You updated your question with more details. And it seems like my solutions do not really fit anymore. The first solutions does not work if you have multiple instances running. The second locks only the database row, it does not prevent multiple thread from entering the code block at the same time.
Therefore if would think about building a database based semaphore.
class Semaphore < ActiveRecord::Base
belongs_to :item, :polymorphic => true
def self.get_lock(item, identifier)
# may raise invalid key exception from unique key contraints in db
create(:item => item) rescue false
end
def release
destroy
end
end
The database should have an unique index covering the rows for the polymorphic association to item. That should protect multiple thread from getting a lock for the same item at the same time. Your method would look like this:
def nasty
until semaphore
semaphore = Semaphore.get_lock(user)
end
...
semaphore.release
end
There are a couple of problems to solve around this: How long do you want to wait to get the semaphore? What happens if the external http requests take ages? Do you need to store additional pieces of information (hostname, pid) to identifier what thread lock an item? You will need some kind of cleanup task the removes locks that still exist after a certain period of time or after restarting the server.
Furthermore I think it is a terrible idea to have something like this in a web server. At least you should move all that stuff into background jobs. What might solve your problem, if your app is small and needs just one background job to get everything done.
You state that this is an ActiveRecord model, in which case the usual approach would be to use a database lock on that record. No need for additional locking mechanisms as far as I can see.
Take a look at the short (one page) Rails Guides section on pessimistic locking - http://guides.rubyonrails.org/active_record_querying.html#pessimistic-locking
Basically you can get a lock on a single record or a whole table (if you were updating a lot of things)
In your case something like this should do the trick...
class User < ActiveRecord::Base
...
def nasty_long_running_method
with_lock do
// something nasty will happen if this method is called simultaneously
// for two instances of the same record and the later one finishes http_request_1
// before the first one finishes http_request_2.
http_request_1 // Takes 1-3 seconds.
http_request_2 // Takes 1-3 seconds.
update_model
end
end
end
I recently created a gem called szymanskis_mutex. It is a module that you can include in the class User and provides the method mutual_exclusion(concern) to provide the functionality you want.
It doesnt rely on databases and doesn't depend on how many processes want to enter the critical section at any given moment.
Note that if the class is initialized in different servers it will not work.
I may suite your needs if your app is small enough. Your code would look like this:
class User
include SzymanskisMutex
...
def nasty_long_running_method
mutual_exclusion(:nasty_long) do
http_request_1 // Takes 1-3 seconds.
http_request_2 // Takes 1-3 seconds.
end
update_model
end
end
I suggest rethinking your architecture as this is not going to be scalable - imagine having multiple ruby processes, failing processes, timeouts etc. Also in-process locking and spawning threads is quite dangerous for application servers.
If you want to sleep well with production then try some async background processing framework for long running tasks with serial queue which will ensure order of running tasks. Just simple RabbitMQ or check this QA Best practice for Rails App to run a long task in the background? , eventually try DB but Optimistic Locking.