I want to write a spring boot batch application where I have a database table full of events. What I want to do is have a multi threaded spring boot batch application which will work in this way:
I want to have 5 threads running each thread will keep an offset to track what event it has read so that no other thread reads the same event again. How am I thinking to do it:
Thread 1 will pick up the event if eventId % 5 == 1
Thread 2 will pick up the event if eventId % 5 == 2
Thread 2 will pick up the event if eventId % 5 == 3
Thread 2 will pick up the event if eventId % 5 == 4
Thread 2 will pick up the event if eventId % 5 == 0
so I want to be able to keep offset in a database table for each thread. Is there a way to make Spring boot environment to work in this way?
You can partition your table according to what you described (I guess you will have 5 partitions) and use a partitioned step configured with a TaskExecutorPartitionHandler of 5 threads. With this setup you will achieve what you are looking for.
Hope this helps.
Related
I have 8 threads in JMeter, which i am executing for every 5 minutes using Task scheduler.
Now i have included 2 threads which want to run for 5 times per day only (ex: at 12am, 5am,10am...)
when the moment comes, the execution shall be 8+2 & remaining time, it shall be only 8 threads.
Is it possible to configure such usecase in Jmeter..
If you're going to use the same .jmx script and want to execute either 8 or 10 "threads" (whatever it is), you can go for:
If Controller - for conditional execution of this or that test elements
__groovy() function to check current time, an example condition which trigger the test at i.e. 5 AM would be:
${__groovy(Calendar.getInstance().get(Calendar.HOUR_OF_DAY) == 5 && Calendar.getInstance().get(Calendar.MINUTE) == 0,)}
I am using JMeter for performance testing. I am using csv file and there are 8 lines of data. I want to execute 3 threads in parallel at a time.
1,2,3 - execute in parallel
1 minute time delay
then 4,5,6 - parallel
1 minute time delay
then
7,8 - parallel execution
I am using JSR223 timer. I used the below code:
if(ctx.getThreadNum()%3==0) {
sleep(60000);
}
Here its sleeping at thread 3 and 6 but not as I mentioned above. Could someone please provide a idea on this?
Maybe it would be easier to switch to Parallel Controller like:
Parallel Controller
1
2
3
Flow Control Action for delay
Parallel Controller
4
5
6
Flow Control Action for delay
Parallel Controller
7
8
The similar behaviour can be achieved using Synchronizing Timer if you cannot or unwilling to use the Parallel Controller
I have a spring batch with reader/processor/writer. which works perfectly fine. The new requirement is the processor should process only 1 or 2 message and sleep for 1 or 2 sec depending on the load in the server (My item-processor calls 3rd party API to send some data, the 3rd party calls should not be continuous).
I have implemented chunk listener, but i m not able to identify how to read the message/item-process call,
chunklistener "readcount" returns the number of message processed in incremental.
paramChunkContext.getStepContext().getStepExecution().getReadCount() //chunkcontext
One option that i m thinking of is to divide the chunkcount/2 and based on the return(if return is 0 - sleep for 1 sec). then
thread.sleep(1000) //
Is there any other better way to perform this? the above process is a dirty job.
I have several (idle) jobs scheduled on a cluster that I want to move to another queue.
I can move a single job like this (where 1234 is the job id):
qmove newQueue 1234
But now I have hundreds of jobs that I want to move to newQueue. Is it possible to move them all? Using * as a wildcard operator does not work.
If the job ids are in sequential order, you could use Bash's brace extension. For example:
$ echo {0..9}
0 1 2 3 4 5 6 7 8 9
Transferred to moving all jobs ranging from 1000 to 2000, the qmove command would be:
qmove newQueue {1000..2000}
This might even work if there are job ids that you are not allowed to move (from other users or in running state). They should be simply ignored. (not tested)
I have trouble implementing the following scenario and Google did't help - may be I am missing something obvious?
Scenario is :
Step 1. 9 sesssions simultaneously running 3 different JDBC queries, i.e
3*Q1,3*Q2,3*Q3 all starting and running at the same time
Clarification: In the beginning of step 1, the following queries will start in 9 different sessions - Q1,Q1,Q1,Q2,Q2,Q2,Q3,Q3,Q3
Step 2. 27 sessions like
above (9 times each query)
Step 3. 54 sessions (18 times each query)
Steps must run sequentially.
To do so:
Step 1)
3 thread groups, each one with 3 threads, each thread group calling a different Qi
Step2)
3 thread groups, each one with 9 threads, each thread group calling a different Qi with scheduler delayed so that it starts after step1 has finished
Step3)
Same as step2 with 18 threads and delayed so that it starts after step 2
But I must say I don't understand why you need such behaviour