I'm using JMeter to test a Java application written by a 3rd party vendor using Versata Logic Studio.
I've got some steps in my test plan that submit a request using some post data and then receive a response back:
Response too large to be displayed. Size: 445817 > Max: 204800, Start of message:
{"header":{"action":"300","arguments":{"tabid":"Header","divid":"ActgDisb,Vendor,BusinessType...ETC
This seems fine (I'm guessing that's 400K?), except that the step is taking far longer than it does to click through the pages in a browser. In the browser it takes 5 seconds at most. In JMeter it's taking 2 minutes. The CPU is also at 60% for just one thread during these steps.
Any ideas on speeding this up? We're struggling to get enough slaves going and this certainly isn't helping.
The message that is displayed tells that you are using View Result Tree during your load test. Jmeter sets a limit that can be changed on the size of pages displayed in this component by adding to user.properties file:
view.results.tree.max_size which defaults to 200 ko
BUT during a load test never ever use this component as it requires a lot of resources (memory and cpu). This component must only be used during scripting phase.
You can read this article that gives tips on JMeter configuration and tuning:
http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/
Disclaimer : I wrote it and it's my company but IMHO I think it's worth reading :-)
Also read this:
http://jmeter.apache.org/usermanual/component_reference.html
If your JMeter script is using a lot of file I/O, then putting those files in RAM will significantly improve the speed. You can use any app such as IMDisk (freeware) to create a virtual disk in RAM. Make sure that you have more than 4GB RAM.
In our case, we are sending around 8000 small files per user. With 200 users on each system, Jmeter is reading 16 Lack files. This was the bottleneck. With the RAM Disk, the file read speed was increased by 20 times and it helped Jmeter to run at full speed.
If you're ready to move to TCP level, there is HTTP Raw Request that allows memory-efficient operation for huge uploads/downloads. Read its manual carefully, there is some JMeter properties for tuning its performance.
However, my experience is that you possibly have a situation where Java itself is a bad technology to perform load tests. I suggest you to take a pair of tries for Raw Request and in case of failure to seek for some C/C++ tool for performance tests.
Related
I am testing a web application login page loading time with 300 thread users and ramp up period of 300 secs.Most of my samples return response code 200.But few of them return response code 400,503.
My goal is to just check the performance of the web application if 300 users start using it.
I am new to Jmeter and have basic knowledge of programming.
My Question :-
1.Can i ignore these errors and focus just on timings from the summary report ?
2.If i really need to fix these errors, how to fix it ?
There are 2 different problems indicated by these errors:
HTTP Status 400 stands for Bad Request - it means that you're sending malformed requests which cannot be understood by the server. You should inspect request details and amend JMeter configuration as it is the problem in your script.
HTTP Status 503 stands for Service Unavailable - it indicates the problem on server side, i.e. server is not capable of handling the load you're generating. This is something you can already report as the application issue. You can try to identify the underlying cause by:
looking into your application log files
checking whether your application has enough headroom to operate in terms of CPU, RAM, Network, Disk, etc. It can be done using APM tool or JMeter PerfMon Plugin
re-running your test with profiler tool telemetry to deep dive into what's under the hood of the longest response times
So first of all you should ensure that your test is doing what it is supposed to be doing by running it with 1-2 users/loops and inspecting requests/response details. At this stage you should not be having any errors.
Going forward you should increase the load gradually and correlate the increasing number of virtual users with the increasing response time/number of errors
`
Performance testing is different from load testing. What you are doing is load testing.
Performance testing is more about how quickly an action takes. I typically capture performance on a system not under load for a given action.
This gives a baseline that I can then refer to during load tests.
Hopefully, you’ve been given some performance figures to test. E.g. must be able to handle 300 requests in two minutes.
When moving onto load, I run a series of load tests with increasing number of users/threads and capture the results from each test.
Armed with this, I can see how load degrades performance to the point where errors start to show up. This gives you an idea of how much typical load the system can handle.
I’d also look to run soak tests too. This where I’d run JMeter for a long period with typical (not peak) load to make sure the system can handle sustained load.
In terms of the errors you’re seeing, no I would not ignore them. Assuming your test is calling the same endpoint, it seems safe to say the code is fine, its the infrastructure struggling with the load you’re throwing at it.
I have a 32GB, i7 core processor running on windows 10 and I am trying to generate 10kVU concurrent load via jmeter. For some reason I am unable to go beyond 1k concurrent and I start getting BindException error or Socket connection error. Can someone help me with the settings to achieve that kind of load? Also if someone is up for freelancing I am happy to consider that as well. Any help would be great as I am nearing production and am unable to load test this use case. If you guys have any other tools that I can use effectively, that would also help.
You reach the limit of 1 computer, thus you must execute in distributed environment of multiple computers.
You can setup JMeter's distributed testing on your own environment or use blazemeter or other cloud based load testing tool
we can use BlazeMeter, which provides us with an easy way to handle our load tests. All we need to do is to upload our JMX file to BlazeMeter. We can also upload a consolidated CSV file with all the necessary data and BlazeMeter will take care of splitting it, depending on the amount of engines we have set.
On BlazeMeter we can set the amount of users or the combination of engines (slave systems) and threads that we want to apply to our tests. We can also configure additional values like multi locations.
1k concurrent sounds low enough that it's something else ... it's also the default amount of open file descriptor limits on a lot of Linux distributions so maybe try to raise the limit.
ulimit -Sn
will show you your current limit and
ulimit -Hn
will show you the hard limit you can go before you have to touch configuration files. Editing /etc/security/limits.conf as root and setting something like
yourusername soft nofile 50000
yourusername hard nofile 50000
yourusername - will have to be the username of the user which with you run jmeter.
After this you will probably have to restart in order for the changes to take effect. If not on Linux I don't know how to actually do this you will have to google :D
Recommendation:
As a k6 developer I can propose it as an alternative tool, but running 10k VUs on a single machine will be hard with it as well. Every VU will take some memory - like at least 1-3mb and this will go up the larger your script is. But with 32gb you could still run upto 1-2kVUs and use http.batch to make concurrent requests which might simulate the 10k VUs depending on what your actual workflow is like.
I managed to run the stages sample with 300VUs on a single 3770 i7 CPU and 4gb of ram in a virtual machine and got 6.5k+ rps to another virtual machine on a neighboring physical machine (the latency is very low) so maybe 1.5-2kVUs with a a somewhat more interesting script and some higher latency as this will give time to the golang to actually run GC while waiting for tcp packets. I highly recommend using discardResponseBodies if you don't need them and even if you need some to actually get the response for them. This helps a lot with the memory consumption a each VU
I execute jmeter script via Blazemeter and I got 2% error. Is it a acceptable rate?
In detail report I observed 1353 request are failed. I just run the script for one user. In that case non of the request doesn't fail. So is that failures are due to performance issue?
Following is summary of the report. Kindly help me to anlyse this.
Most probably your application simply cannot handle the load of 50 concurrent users. With regards to "acceptable" - we don't know. If you're load testing a fan page of your local hip hop star - even 90% error rate will be acceptable. If you're testing an algorithm which will be deployed on a Mars rover and will have to work without errors and modifications for 20 years - it is not.
Normally maximum response time, minimum throughput, acceptable number of errors, etc. are defined in SLA or NFR. If you don't have those and performing some form of stress testing of your application and want to figure out the root cause of the performance bottleneck - take the next steps:
Check your application log file(s), they should have some information regarding the failure
Check status message and code in .jtl results file. Sometimes it also makes sense to "tell" JMeter to save response data for failed samplers by adding the next lines to user.properties file:
jmeter.save.saveservice.output_format=xml
jmeter.save.saveservice.response_data.on_error=true
Make sure you add the load gradually, this way you will be able to correlate increasing error rate with increasing number of users and will be able to determine exact point of time when first error occurred
Get used to monitoring whether your application under test has enough headroom to operate in terms of CPU, RAM, Network, Disk, etc. It can be done using JMeter PerfMon Plugin
If you have ability to read and understand the code in the language your application is written in - it would be beneficial if you could run your test with profiler tool telemetry enabled, this is probably the most efficient way to identify the performance problem in your application.
first of all i already had a look at several questions which are quite similar. But i wasn't able to find a solution.
My script performs a load test it calls several different URLs(GET http) to download the content behind.
After 120 requests the memory usage increases up to 2GB and after 500 to 5-6GB
I changed already the xmx size in hope that this will solve the problem but it doesn't.
Is there any way to configure jmeter to not save the files coming from a response? Or lets say to discard immediately the downloaded files?
Is it maybe an JRE setting?
Or is there no way to solve this memory increasing problem?
Br,
Kabba
Try disabling the View Results Tree in the script as it records all results for you to inspect.
The jMeter documentation specifically mentions this:
18.3.6 View Results Tree
View Results Tree MUST NOT BE USED during load test as it consumes a
lot of resources (memory and CPU). Use it only for either functional
testing or during Test Plan debugging and Validation.
I don't think that it is something you can do via JMeter settings. As per JMeter Performance and Tuning Tips guide (make sure that you follow all the recommendations):
Default JMeter java configuration comes with 512 Mo and very little GC tuning.
First ensure you set -Xmx option value to a reasonable value regarding your test requirements.
Then change MaxNewSize option in jmeter file to respect the original ratio between MaxNewSize and -Xmx.
Finally try tuning GC options only if you master this domain.
So you can try different Garbage Collection options on JMeter's side. See How to Tune Java Garbage Collection guide for ramp-up on the domain or consult a Java developer if one is around.
Here is the scenario
We are load testing a web application. The application is deployed on two VM servers with a a hardware load balancer distributing the load.
There are tow tools used here
1. HP Load Runner (an expensive tool).
2. JMeter - free
JMeter was used by development team to test for a huge number of users. It also does not have any licensing limit like Load Runner.
How the tests are run ?
A URL is invoked with some parameters and web application reads the parameter , process results and generates a pdf file.
When running the test we found that for a load of 1000 users spread over period of 60 seconds, our application took 4 minutes to generate 1000 files.
Now when we pass the same url through JMeter, 1000 users with a ramp up time of 60 seconds,
application takes 1 minutes and 15 seconds to generate 1000 files.
I am baffled here as to why this huge difference in performance.
Load runner has rstat daemon installed on both servers.
Any clues ?
You really have four possibilities here:
You are measuring two different things. Check your timing record structure.
Your request and response information is different between the two tools. Check with Fiddler or Wireshark.
Your test environment initial conditions are different yielding different results. Test 101 stuff, but quite often overlooked in tracking down issues like this.
You have an overloaded load generator in your loadrunner environment which is causing all virtual users to slow. For example you may be logging everything resulting in your file system becoming a bottleneck for the test. Deliberately underload your generators, reduce your logging levels and watch how you are using memory for correlations so you don't create a physical memory oversubscribed condition which results in high swap activity.
As to the comment above as to JMETER being faster, I have benchmarked both and for very complex code the C based solution for Loadrunner is faster upon execution from iteration to iteration than the Java based solution in JMETER. (method: complex algorithm for creating data files on the fly for upload for batch mortgage processing. p3: 800Mhz. 2GB of RAM. LoadRunner 1.8 million iterations per hour ungoverned for a single user. JMETER, 1.2 million) Once you add in pacing it is the response time of the server which is determinate to both.
It should be noted that LoadRunner tracks its internal API time to directly address accusations of the tool influencing the test results. If you open the results set database set (.mdb or Microsoft SQL server instance as appropriate) and take a look at the [event meter] table you will find a reference for "Wasted Time." The definition for wasted time can be found in the LoadRunner documentation.
Most likely the culprit is in HOW the scripts are structured.
Things to consider:
Think / wait time: When recording,
Jmeter does not automatically put in
waits.
Items being requested: Is
Jmeter ONLY requesting/downloading
HTML pages while Load runner gets all
embedded files?
Invalid Responses:
are all 1000 Jmeter responses valid?
If you have 1000 threads from a
single desktop, I would suspect you
killed Jmeter and not all your
responses were valid.
Dont forget that the testing application itself measures itself, since the arrival of the response is based on the testing machine time. So from this perspective it could be the answer, that JMeter is simply faster.
The second thing to mention is the wait times mentioned by BlackGaff.
Always check results with result tree in jmeter.
And always put the testing application onto separate hardware to see real results, since testing application itself loads the server.