I have a Test plan with several Thread Groups (TG) which have a different Loop Count.
Test Plan
|--TG 1: loop count=5
|--TG 2: loop count=10
|--TG 3: loop count=3
These thread groups are executed concurrently. However because the loop count of threads differ, some threads are finished while others are still running. What I would like is for JMeter to manage the execution of each Thread Group based on a ratio of the the highest Loop Count in the Test plan in order to create a more even time distribution and ensure all Thread Groups roughly end at the same time. For instance:
Test Plan
|--TG 1: loop count=5, execute every 2 occurrences of TG 2
|--TG 2: loop count=10
|--TG 3: loop count=3, execute every 3.3 occurrences of TG 2
In terms of execution, things would look like this:
[round 1] TG1: run | TG2: run | TG3: run
[round 2] TG1: wait | TG2: run | TG3: wait
[round 3] TG1: run | TG2: run | TG3: wait
[round 4] TG1: wait | TG2: run | TG3: run
etc...
Can I configure JMeter to acheive this, if so how?
Related
Currently, I have a sequence job in DataStage.
Here is the flow:
StartLoop Activity --> UserVariables Activity --> Job Activity --> Execute Command --> Endloop Activity
The job will run every 30 minutes (8 AM - 8 PM) to get real data. The first loop iteration will load data from 8 PM the previous day to 8 AM the current day, and the others will load data that happens in the last 30 minutes.
The UserVariables Activity is to pass variables (SQL statement) to filter data getting in the Job Activity. The first iteration the UserVariables pass variable A (SQL statement 1) to the Job Activity, from the second iteration, it will pass variable B (SQL statement 2) to the Job Activity.
The Execute Command I currently set the 'Sleep 1800' command for the job to sleep 30 minutes to end the iteration of the loop. But I realized now that it is affected by the running time of each iteration. So with my knowing-nothing about shell script, I have searched for solutions and have this file to sleep until a specific time when minute like 30 or 00 (delay 0-1 minute but it's fine).
The shell script is below, I ran it fine on my system but no success on making it as part of the job.
#!/bin/bash
minute=$(date +%M)
num_1=30
num_2=60
if [ $minute -le 30 ];
then
wait=$((($num_1 - $minute)*$num_2))
sleep $wait
fi
if [ $minute -gt 30 ];
then
wait=$((($num_2 - $minute)*$num_2))
sleep $wait
fi
I am now facing 2 problems right now that I need your help with.
The job runs the first iteration fine with the variable A below:
select * from my_table where created_date between trunc(sysdate-1) + 20/24 and trunc(sysdate) + 8/24;
But from the second iteration it failed with the Job Activity with the variable B below:
select * from my_table where created_date between trunc(sysdate-1/48, 'hh') + 30*trunc(to_number(to_char(sysdate-1/48,'MI'))/30)/1440 and trunc(sysdate, 'hh') + 30*trunc(to_number(to_char(sysdate,'MI'))/30)/1440;
In the parallel job, the log said:
INPUT,0: The following SQL statement failed: select * from my_table where created_date between trunc(sysdate-1/48, hh) + 30*trunc(to_number(to_char(sysdate-1/48,MI))/30)/1440 and trunc(sysdate, hh) + 30*trunc(to_number(to_char(sysdate,MI))/30)/1440.
I realized that maybe it failed to run the parallel job because it removed the single quote in hh and MI.
Is it because when passing variables from UserVariables Activity to Job Activity the variable will remove all the quotes? And how can I fix this?
2. How can I make the shell script above as part of the job like Execute Command or some other stage. I have searched for solutions and I think it's about the ExecSH Before/ After Routine Activity. But after reading from IBM pages, I still don't know where to start with it.
Sorry for adding 2 questions at 1 post that makes it so long but it's very relative to each other so it will take lots of time to answer if I separate it into 2 posts and you guys need more information about it.
Thank you!
Try escaping the single quote characters (precede each with a backslash).
Execute the shell script from an Execute Command activity ahead of the Job activity.
Suppose I have a program that loads significant content before running...but this is a one time slowdown.
Next, I write:
cat ... | parallel -j 8 --spreadstdin --block $sz ... ./mycode
Will this induce the load overhead every single job?
If it does induce the overhead, is there a way to avoid it?
As #Barmar says, ./mycode is started for each block in your example.
But since you do not use -k in your example you may be able to use --round-robin.
... | parallel -j 8 --spreadstdin --round-robin --block $sz ... ./mycode
This will start 8 ./mycodes (but not one per block) and give blocks to any process that is ready to read.
This example shows that more blocks are given to process 11 and 10 than process 4 and 5 because 4 and 5 read slower:
seq 1000000 |
parallel -j8 --tag --roundrobin --pipe --block 1k 'pv -qL {}0000 | wc' ::: 11 4 5 6 9 8 7 10
parallel doesn't know anything about the internal workings of the program you're running with it. Each instance runs independently, there's no way that one invocation's initialization can be copied over to the others.
If you want the application to initialize once and then run multiple instances in parallel, you need to design that into the application itself. It should load the data, then use fork() to create multiple processes that use this data.
I am a newbie in Robot Framework. I want to implement a For loop to check for a xpath on the page. It should wait for 10 seconds for the xpath, if the xpath is still not there then wait again for 10 seconds, if the xpath has appeared then exit the loop and move ahead. A total of 10 iterations are required to wait for the element.
I am trying the below:
|| ${count}= | Get Matching Xpath Count | xpath=//*[.='Continue'] |
|| :FOR | ${loopIndex} | IN RANGE | 10
| Wait Until Page Contains Element | xpath=//*[.='Continue'] | 10
| Exit For Loop If | ${count}>0
|| Log | Got out of loop
I am getting the error right now as:
FAIL : Element 'xpath=//*[.='Continue']' did not appear in 10 seconds.
Let me know if I have missed some information. I am using RIDE for editing.
EDIT:
I have to do it in a loop. I am asking for help regarding this so that I can use this loop example in other places of my work.
You can achieve this either with a FOR loop or with a "Wait until keyword succeeds".
See both examples:
*** Test Cases ***
test with for loop
:FOR ${loopIndex} IN RANGE 10
\ ${element_is_here} = Run Keyword and return status Page should contain element xpath=//*[.='Continue']
\ Exit For Loop If ${element_is_here}
\ sleep 10s
Log I found the element
test with WUKS
wait until keyword succeeds 10s 10s Page should contain element xpath=//*[.='Continue']
Log I found the element
The keyword is failing, so the test case is failing. What you need to do instead is either not use a loop and simply wait for 100 seconds, or use something like Run keyword and ignore error or Run keyword and continue on failure (eg: Run keyword and ignore error | Wait until page contains element | ...)
I have around 4 files named:
Test_1.csv
Test_2.csv
...
Each line in each test file has the following format:
method;request
Where, method is the URL that I call and request is the request I make to it. Everything is configured to pick up these values and form URLs.
However, first Test_1.csv must run, then Test_2.csv must run and so on. To do that, I have created 5 thread groups in the following hierarchy:
Test Plan
|
+- Step 1
|
+- HTTP request
+- CSV Data Set Config <- Reads from Test_1.csv
+- Uniform Random Timer
+- Step 2
|
+- HTTP request
+- CSV Data Set Config <- Reads from Test_2.csv
+- Uniform Random Timer
And I have also selected the Run thread groups consecutively option in my test plan. Each thread group is configured for 20 threads. Now, what I want it to do is that, Step 1 should run each and every test in Test_1.csv, then Step 2 should execute and run each and every line in Test_2.csv. However, what is happening is that, Step 1 runs the first 20 lines from Test_1.csv and then Step 2 starts, runs 20 tests and continues. Then, after all steps are done, Step 1 runs again and runs the very same 20 lines from Test1.csv. I want it to just run once, loop through all the lines in the test file and then quit and hand over control to the next thread.
How do I go about this?
Try to use the following schema:
Test Plan
Thread Group #1
Number of Threads: N
. . .
While Controller
Condition: ${__javaScript("${request}"!="<EOF>",)} - until the EOF
CSV Data Set Config
Filename: [path to your file with test-data] - Test_1.csv in this case
Variable Names: method,request
HTTP Request
Uniform Random Timer
. . .
Thread Group #2
. . .
[same for Test_2.csv]
Thread Group #3
. . .
[same for Test_3.csv]
jMeter Plugins' ParameterizedController can help you call parts of your test plan like you call functions in a regular programming language. More importantly, you can parametrise those calls with variables.
I'm doing some experiments on a computing cluster. My algorithm has two steps. The first one writes its outputs to some files which will be used by the second step. The dependecies are 1 to n meaning one step2 programs needs the output of n step1 program. I'm not sure what to do neither waist cluster resources nor keep the head node busy. My current solution is:
submit script (this runs on the head node)
for different params, p:
run step 1 with p
sleep some time based on the an estimate of how much step 1 takes
for different params, q:
run step 2 with q
step 2 algorithm (this runs on the computing nodes)
while files are not ready:
sleep a few minutes
do the step 2
Is there any better way to do this?
SGE provides both job dependencies and array jobs for that. You can submit your phase 1 computations an array job and then submit the phase 2 computation as a dependent job using the qsub -hold_jid <phase 1 job ID|name> .... This will make the phase 2 job wait until all the phase 1 computations have finished and then it will be released and dispatched. The phase 1 computations will run in parallel as long as there are enough slots in the cluster.
In a submission script it might be useful to specifiy holds by job name and name each array job in a unique way. E.g.
mkdir experiment_1; cd experiment_1
qsub -N phase1_001 -t 1-100 ./phase1
qsub -hold_jid phase1_001 -N phase2_001 ./phase2 q1
cd ..
mkdir experiment_2; cd experiment_2
qsub -N phase1_002 -t 1-42 ./phase1 parameter_file
qsub -hold_jid phase1_002 -N phase2_002 ./phase2 q2
cd ..
This will schedule 100 executions of the phase1 script as the array job phase1_001 and another 42 executions as the array job phase1_002. If there are 142 slots on the cluster, all 142 executions will run in parallel. Then one execution of the phase2 script will be dispatched after all tasks in the phase1_001 job have finished and one execution will be dispatched after all tasks in the phase1_002 job have finished. Again those can run in parallel.
Each taks in the array job will receive a unique $SGE_TASK_ID value ranging from 1 to 100 for the tasks in job phase1_001 and from 1 to 42 for the tasks in job phase1_002. From it you can compute the p parameter.