I'm planning to use Quartz scheduler to process a one-time job.
My use case is, I need to migrate BLOB from one storage to another and blob's can be as big as 100GB, so a particular job can run really long enough to get the work done.
The reason I'm using Quartz because of its clustering support, fault tolerance and retry capabilities in case job fails etc. Only thing I'm concerned about is, I might have a lot of miss fire trigger scenario and a lot of database lock which can hamper live production traffic on those database hosts. I will probably be scheduling 10s of thousands of job in one shot.
Few of the things that I figured out is
I can set a high value for org.quartz.jobStore.misfireThreshold so that miss fire does not happen. I don't really care about the time when the job get's picked up as it's background job and no SLA as such. Only thing I care about is that eventually job getting picked up and getting work done.
I can also set batch mode properties org.quartz.scheduler.batchTriggerAcquisitionMaxCount and org.quartz.scheduler.batchTriggerAcquisitionFireAheadTimeWindow. I understand the batch max count property should be like equal to the thread pool size which can give the biggest bang on performance but what should be the value of fire ahead of time window be?
I'm using Quartz with Spring boot and will be leveraging org.quartz.impl.jdbcjobstore.JobStoreCMT. What I understand is execute method of the job get wrapped in the transaction, will this cause any problem since transaction will be open for a long time as the job might take hours to complete? Is this something ok? I will be using Oracle database.
Am I missing something here? Can someone share their experience with a similar use case?
Thanks!
I have recorded a script against the application that I want to test. Now, I am having a hard time arriving at the decision that what is that number that the application will run without any issue and to find out the max number of users. Here is what I have done-
I have run the Jmeter script for 10, 50, 100, 150 users
Until 50 users, it runs like a charm. After about 80 users the throughput starts to come down and some samples do not show up in Aggregate Report.
I see heap memory problems in my console for about 150 users over period of time. Is it the application problem or my machine problem?
Do you have an article where I could read about how to come to a conclusion about THE number?
UPDATE- after increasing the heap size, it is running smoothly for 100 users. I am even more confused now
Thank you
The problem can be anywhere!
Server Performance Metrics collector:
First you need an agent running in the application server to monitor the server performance while you are running the test.
This link will give you an idea about the set up.
JMeter Best Practices:
I think that you are running your test in GUI mode with listeners. Most likely the problem is with your machine/your test. Ensure that you follow this.
Samples not showing in aggregate Report:
You already asked a question on this in SO. Do not select 'Successes' in the Log/display only section of the listener while writing the results in the jtl. It will not write the failed requests details. You might need all the results. Once the jtl is created, you can always filter 'Success' only results as and when you want.
So I've been using Boto in Python to try and configure autoscaling based on CPUUtilization, more or less exactly as specified in this example:
http://boto.readthedocs.org/en/latest/autoscale_tut.html
However both alarms in CloudWatch just report:
State Details: State changed to 'INSUFFICIENT_DATA' at 2012/11/12
16:30 UTC. Reason: Unchecked: Initial alarm creation
Auto scaling is working fine but the alarms aren't picking up any CPUUtilization data at all. Any ideas for things I can try?
Edit: The instance itself reports CPU utilisation data, just not when I try and create an alarm in CloudWatch, programatically in python or in the interface. Detailed monitoring is also enabled just in case...
Thanks!
The official answer from AWS goes like this:
Hi, There is an inherent delay in transitioning into INSUFFICIENT_DATA
state (only) as alarms wait for a period of time to compensate for
metric generation latency. For an alarm with a 60 second period, the
delay before transition into I_D state will be between 5 and 10
minutes.
John.
Apparently this is a temporary state and will likely resolve itself.
I am not sure what's going on in the backend, but if you compare the alarm history you will see AWS remove the 'unit' column if you just modify the alarm without any change as at7000ft said. So remove the unit column of your script.
Make sure that the alarm's Namespace is 'AWS/EC2'.
I know this is a long time after the original question, but in case others find this via Google, I had the same problem, and it turned out I set alarm's Namespace improperly.
It is needed to publish data with the same unit used to create the alarm. If you didn't specify one, it will be a <None> unit.
Unit can be specified in aws put-metric-data and aws-put-metric-alarm with --unit <value>
Unit <value> can be:
Seconds
Bytes
Bits
Percent
Count
Bytes/Second (bytes per second)
Bits/Second (bits per second)
Count/Second (counts per second)
None (default when no unit is specified)
Units are also case-sensitive, be carefull about that in your scripts.
For CPUUtilization, you can use Percent.
After the first data-set is sent to your alarm (it can take up to 5 minutes for a non-detailed monitored instance), the alarm will switch to the OK or ALARM state instead of the INSUFFICIENT_DATA one.
I am having the same INSUFFICIENT_DATA alarm state show up in CloudWatch for an RDS CPUUtilization > 60 alarm created with CloudFormation. ("Reason: Unchecked: Initial alarm creation" shows up under details). This is a very crude fix but I found that by selecting the alarm, clicking the Modify button, and then the Save button (without changing anything) the alarm goes to the OK state and everything is file.
I had this problem. Make sure the metric name you use to create the alarm matches the actual metric name.
You can list your metrics with:
aws cloudwatch list-metrics --namespace=<NAMESPACE, e.g. System/Linux, etc>
Find the metric and the MetricName. Make sure your alarm is configured for that metric.
As far as I know, default metric resolution is 5 minutes (which can be lowered to 1 minute if you pay up, or something like that), so if your alarm's measurement period is lower than that, then it'll remain permanently in an INSUFFICIENT_DATA state. In my case, I had a 1 minute measurement period on CPU utilization, and changing it to 5 minutes has fixed the state issue.
I had a similar problem, my alarm was constantly in INSUFFICIENT_DATA status although I can see the metric in the GUI.
Come out that this happen, because I specified the wrong Unit for the metric, when I created the Alarm. No error was reported back but it never became GREEN.
Better to avoid to specify it, if you are not sure, and AWS will do the correct match in the background.
There is a directory /var/tmp/aws-mon/ that contains a couple files. One is instance-id. The instance I was on was created from an AMI and this file retained the old instance id. I just edited it and made sure /var/tmp/aws-mon/placement/availability-zone was also correct. The alarms changed to OK almost instantly.
Also ran into this problem but for a different reason: I passed ES cluster ARN instead of domain name in my Cloudformation template. It was pretty frustrating
If suppose I will run load testing with 5000 threads or may be more ,
Will the main server under test crash at a certain level.
As EJP said, it is JMeter purpose to find the limit of tested application and how it will react under performance.
So yes it is perfectly possible.
You should read JMeter Manual.
If you are doing any kind of performance tests (load, stress, soak etc) you will want to know at what point your application server falls over i.e. its breaking point.
Once you've found out what your upper limit is, start dialing back the number of threads until you find your application's "sweet spot" for example CPU usage <80% & >70%
I want to load test a URL by hitting it few hundred times at same millisecond . I tried JMeter but I could hit 2 request at same millisecond. This seems to be problem that my machine cant create threads fast enough . Is there any solution to the issue ?
In JMeter you can use synchronizing timer setting it to 100, this way all threads will wait until there are 100 available and hit the server:
http://jmeter.apache.org/usermanual/component_reference.html#Synchronizing_Timer
Another solution is to increase the number of Threads so that you hit this throughput.
In next coming version (2.8) of JMeter you will be able to create threads on demand (created once needed).
Anyway hitting few hundred times at same millisecond is a high load so you will have to tune JMeter correctly.
Regards
Philippe
JMeter uses blocking HTTP client, in order to hit the server at the exact same time with 100 reqeusts you need 100 threads in JMeter. Even providing that, you still don't have 100 cores to actually run such code at the same time. Even if you had 100 cores, it takes some time to start a thread, so you would have to start them in advance and synchronize on some sort of barrier. And that is not supported in JMeter.
Why do you really want to run your server "at same millisecond"? An ordinary load test just calls the server with as many connections as possible, but not necessarily at the same time. Moreover, sometimes you are even adding random sleep between requests to simulate so-called think time.
As per Philippe's answer, JMeter does in fact support synchronous requests. But maybe for what you want something like Apache Bench using -c100 (or tune it to whatever works) is a better option? It's pretty basic stuff but then the overhead is a lot smaller which might help in this situation.
But I would also steal from Tomasz's answer and echo his concern that perhaps this is not really the best way to approach load testing. If you're trying to replicate real life traffic then do you really need such a high level of concurrency?
You need to use Jmeter-server and a host of client machines for load generation. Your single machine is not enough to generate the load itself.