I've got a strange problem. I run my JMeter test on one computer/network and my test runs perfectly all the time. Single thread, multithread, it runs with no errors. I then run the same test on a different computer/network and I'm getting the below error. The error happens at random steps and not the same step each time.
org.apache.http.NoHttpResponseException: home-env-c.t1cloud.com:443 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:930)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:641)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:66)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1281)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1270)
at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:630)
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558)
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256)
at java.lang.Thread.run(Unknown Source)
I've searched for the error and added these two lines to user.properties but it didn't work:
httpsampler.ignore_failed_embedded_resources=true
http.connection.stalecheck$Boolean=true
If the "other" computer has different (better) hardware specifications it might be the case you're delivering more load than your server can handle so it looks like to be a bottleneck
Inspect how many requests per second you produce in both cases, it can be done by looking at Server Hits per Seconds or Transactions per Second charts or by simply looking into how many requests in the given period JMeter was able to make.
If it appears that the "other" computer produces more load than you expect/need you can slow it down using JMeter Timers in general and Constant Throughput Timer in particular.
Related
After the response time is set too small, check the result tree and display read time out,But not all fail, some don't show read time out.
There is no problem with the server returning the result. It should be the problem of jmeter, please help me analyze.
jmeter log:
2022-03-01 15:25:22,959 DEBUG o.a.j.p.h.s.HTTPHC4Impl: Sent 314 bytes
2022-03-01 15:25:23,011 DEBUG o.a.j.p.h.s.HTTPHC4Impl: IOException
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_131]
at java.net.SocketInputStream.socketRead(Unknown Source) ~[?:1.8.0_131]
at java.net.SocketInputStream.read(Unknown Source) ~[?:1.8.0_131]
at java.net.SocketInputStream.read(Unknown Source) ~[?:1.8.0_131]
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:935) ~[ApacheJMeter_http.jar:5.4.3-SNAPSHOT]
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:646) [ApacheJMeter_http.jar:5.4.3-SNAPSHOT]
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:66) [ApacheJMeter_http.jar:5.4.3-SNAPSHOT]
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1296) [ApacheJMeter_http.jar:5.4.3-SNAPSHOT]
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1285) [ApacheJMeter_http.jar:5.4.3-SNAPSHOT]
at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:638) [ApacheJMeter_core.jar:5.4.3]
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558) [ApacheJMeter_core.jar:5.4.3]
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) [ApacheJMeter_core.jar:5.4.3]
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) [ApacheJMeter_core.jar:5.4.3]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_131]
As per SocketTimeoutException description:
Signals that a timeout has occurred on a socket read or accept.
You're expecting JMeter to get the response in 50 milliseconds or less, if the server fails to respond fully in 50 milliseconds - you will get this error.
If you're absolutely sure that your server responds in less than 50 milliseconds be aware that there are also connect time and latency
As per JMeter Glossary:
Elapsed time. JMeter measures the elapsed time from just before sending the request to just after the last response has been received. JMeter does not include the time needed to render the response, nor does JMeter process any client code, for example Javascript.
So there are following operations involved:
JMeter connects to the server
JMeter sends the request
Server prepares the response
Server sends the response to the server
JMeter receives first byte of the response
JMeter receives last byte of the response
And "Response timeout" means full end-to-end sequence.
If you don't want to see this error either increase the timeout or use Duration Assertion instead, in that case JMeter will wait for the full response to arrive and only then will assert whether the response time exceeds the anticipated value
Here I use Parallel Controller but the Parallel Controller2 and Parallel Controller3 don't run and throw error in logs. The Parallel Controller1 is inside Transaction Controller. The Parallel Controller2 and Parallel Controller3 are inside Simple Controller.
Here is the error log for Controller2
2021-08-24 17:51:48,819 ERROR o.a.j.t.JMeterThread: Error while processing sampler: 'bzm - Parallel Controller2'.
java.lang.RuntimeException: Attempting to reset the thread context
at org.apache.jmeter.testelement.AbstractTestElement.setThreadContext(AbstractTestElement.java:579) ~[ApacheJMeter_core.jar:5.4.1]
at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:623) ~[ApacheJMeter_core.jar:5.4.1]
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558) ~[ApacheJMeter_core.jar:5.4.1]
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) [ApacheJMeter_core.jar:5.4.1]
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) [ApacheJMeter_core.jar:5.4.1]
at com.blazemeter.jmeter.controller.JMeterThreadParallel.run(JMeterThreadParallel.java:61) [jmeter-parallel-0.11.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:1.8.0_211]
at java.util.concurrent.FutureTask.run(Unknown Source) [?:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_211]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_211]
How is it possible to execute the calls in parallel inside another parallel controller?
I don't think it's supported, moreover it's not clear what you're trying to achieve by this.
As per documentation Parallel Controller executes its children in parallel so instead of going for nested constructions you can just place everything under a single Parallel Controller.
If you need to mimic a parallel call which in its turn triggers another parallel calls, however I don't think the use case is valid, you can reach out to the plugin developers and/or maintainers at jmeter-plugins support forum
In the meantime you can use JSR223 Test Elements to kick off the parallel requests as per your network requests pattern using JSR223 Test Elements and Groovy language like it's described in the How to Load Test AJAX/XHR Enabled Sites With JMeter article
I have a simple RedisStringTemplate which is throwing SocketTimeoutExceptions after only a couple reads of one key. I haven't set any timeout in any config, so it is using the default. This is happening in a junit run under SpringJUnit4ClassRunner, if that matters.
If I run just one test case, which does several reads and a couple of updates, it works fine.
But if I run the whole test class, which has a couple test cases which read the value, including some set up/cleanup code, which read and update the value, I get these 'Read timed out'.
To do the read, we simply do
myRedisStringTemplate.opsForValue().get(key);
to update this key, we do
myRedisStringTemplate.opsForValue().set(key, valueForKey);
In the scenario with the errors, I believe there is only one update on this key, to set the initial value. Then a couple reads on that key before the 'Read timed out' starts, in the #Before before the next test method, where we're trying to clean up data from the previous test.
This is within a Spring application.
Here's the stack trace :
java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
||||||c.i.q.c.MeteredAssignmentsCacheController:73
16:25:32.090|E| java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
com.mycompany.rest.exception.RestException: java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
at com.mycompany.myclass...
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:710)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.mycompany.spring.metrics.MetricsAspect.advice_Timed(MetricsAspect.java:85)
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at com.ryantenney.metrics.spring.TimedMethodInterceptor.invoke(TimedMethodInterceptor.java:46)
at com.ryantenney.metrics.spring.TimedMethodInterceptor.invoke(TimedMethodInterceptor.java:32)
at com.ryantenney.metrics.spring.AbstractMetricMethodInterceptor.invoke(AbstractMetricMethodInterceptor.java:58)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:643)
at com.mycompany.myclass...
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:214)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:690)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:945)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:876)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
at org.springframework.web.servlet.FrameworkServlet.doDelete(FrameworkServlet.java:885)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:653)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.springframework.data.redis.RedisConnectionFailureException: java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:46)
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:36)
at org.springframework.data.redis.connection.jedis.JedisConverters.toDataAccessException(JedisConverters.java:116)
at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:168)
at org.springframework.data.redis.connection.jedis.JedisConnection.get(JedisConnection.java:962)
at org.springframework.data.redis.connection.DefaultStringRedisConnection.get(DefaultStringRedisConnection.java:264)
at org.springframework.data.redis.core.DefaultValueOperations$1.inRedis(DefaultValueOperations.java:45)
at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:50)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:181)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:149)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:84)
at org.springframework.data.redis.core.DefaultValueOperations.get(DefaultValueOperations.java:42)
From point of view of redis, this is highly unlikely occasion. So you most likely have mistakes in your Spring or Redis configurations.
For example I can assume, that you specified timeout in redis configuration
http://redis.io/topics/clients
By default recent versions of Redis don't close the
connection with the client if the client is idle for many seconds: the
connection will remain open forever. However if you don't like this
behavior, you can configure a timeout, so that if the client is idle
for more than the specified number of seconds, the client connection
will be closed.
So if you're using connection pool for Redis in Spring, some of your connections would timeout, at some point, but would still remain in the pool.
Can you check what timeouts you currently have in your Redis configuration?
I just started cascading programming and have a cascading job which needs to run variable times of iteration. During each iteration, it ready from file (Tap) generated from previous iteration and write calculated data to two separate SinkTaps.
One Tap (Tap Final) is used to collect data from each iterations.
The other Tap (Tap intermediate) using to collect data that need to be calculated in the next iteration.
I am using SinkMode.UPDATE for "Tap final" to make this happen. It works correct at local mode. But failed at cluster mode. Complain about file already existed ("Tap final").
I am running CDH4.4 and cascading 2.5.2. Seems like there is no one has experienced the same problem.
If anyone knows any possible way to fix it, please let me know. Thanks
Caused by: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://dv-db.machines:8020/tmp/xxxx/cluster/97916 already exists
at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:126)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:419)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:332)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1269)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1266)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:606)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:601)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:601)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:586)
at cascading.flow.hadoop.planner.HadoopFlowStepJob.internalNonBlockingStart(HadoopFlowStepJob.java:105)
at cascading.flow.planner.FlowStepJob.blockOnJob(FlowStepJob.java:196)
at cascading.flow.planner.FlowStepJob.start(FlowStepJob.java:149)
at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:124)
at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:43)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
It would helpful to understand the issue better if you could add cascading flow code to your question.
It seems the job file with same name is being used between different jobs on cluster mode. One simple solution in case you are fine to not run it concurrently would be be set max concurrent steps to 1.
Flow flow = flowConnector.connect("name", sources, sinks, outPipe1, outPipe2);
flow.setMaxConcurrentSteps(jobProperties, 1);
UPDATE only works with sinks (like databases) that support in-place updating.
If you're using Hfs (a file system sink) then you'll need to use SinkMode.REPLACE.
I have several Spring Batch (2.1.9.RELEASE) jobs running in production that use org.springframework.batch.core.launch.support.RunIdIncrementer.
Sporadically, I get the following error:
org.springframework.batch.core.repository.JobInstanceAlreadyCompleteException: A job instance already exists and is complete for parameters={run.id=23, tenant.code=XXX}. If you want to run this job again, change the parameters.
at org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:122) ~[spring-batch-core-2.1.9.RELEASE.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_39]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_39]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_39]
at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_39]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) ~[spring-aop-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110) ~[spring-tx-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at org.springframework.batch.core.repository.support.AbstractJobRepositoryFactoryBean$1.invoke(AbstractJobRepositoryFactoryBean.java:168) ~[spring-batch-core-2.1.9.RELEASE.jar:na]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) ~[spring-aop-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at sun.proxy.$Proxy64.createJobExecution(Unknown Source) ~[na:na]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:111) ~[spring-batch-core-2.1.9.RELEASE.jar:na]
at org.springframework.batch.core.launch.support.CommandLineJobRunner.start(CommandLineJobRunner.java:349) [spring-batch-core-2.1.9.RELEASE.jar:na]
at org.springframework.batch.core.launch.support.CommandLineJobRunner.main(CommandLineJobRunner.java:574) [spring-batch-core-2.1.9.RELEASE.jar:na]
at (omitted for brevity)
A sampling from the various XML contexts:
<bean
id="jobParametersIncrementer"
class="org.springframework.batch.core.launch.support.RunIdIncrementer" />
<batch:job id="rootJob"
abstract="true"
restartable="true">
<batch:validator>
<bean class="org.springframework.batch.core.job.DefaultJobParametersValidator">
<property name="requiredKeys" value="tenant.code"/>
</bean>
</batch:validator>
</batch:job>
<batch:job id="rootJobWithIncrementer"
abstract="true"
parent="rootJob"
incrementer="jobParametersIncrementer" />
I use org.springframework.batch.core.launch.support.CommandLineJobRunner to execute the job:
java org.springframework.batch.core.launch.support.CommandLineJobRunner /com/XXX/job123/job123-context.xml job123 tenant.code=XXX -next
All of the jobs (that use the incrementer) have rootJobWithIncrementer as parent.
I did quite a bit of research and found that some who got this error had success changing the isolation level of the transaction manager. I fiddled with several levels, finally arriving at READ_COMMITED.
<batch:job-repository
id="jobRepository"
data-source="oracle_hmp"
transaction-manager="dataSourceTransactionManager"
isolation-level-for-create="READ_COMMITTED"/>
Based on my understanding, this type of error should only happen if the same job is executed at the same time from multiple processes -- so that there might be contention for the incrementer. In this instance, that is not the case, yet we see the error.
Any ideas as to what might be causing this problem? Should I try a different isolation level? Something else?
Thanks!
There is a similar question here, but it is not as well documented (and also lacks and answer).
This might be a long shot but it took me a long time to figure it out because the only symptom was sporadically getting the JobInstanceAlreadyCompleteException as you describe so I figured I'd suggest it.
The database I was using was Oracle and the BATCH_JOB_SEQ and BATCH_JOB_EXECUTION_SEQ I had created both had a CACHE_SIZE of 10.
This had the effect of sometimes causing the JOB_INSTANCE_ID and JOB_EXECUTION_ID to not be ordered correctly. Spring batch makes the assumption that the most recent JOB_INSTANCE is the one with max(JOB_INSTANCE_ID) (see org.springframework.batch.core.repository.dao.JdbcJobInstanceDao.FIND_LAST_JOBS_BY_NAME). Since my sequence was sometimes thrown off, this assumption did not always hold true.
I fixed it by setting the sequences to NO_CACHE.
An easy way to tell if this might be your problem is to check if your sequences are set to CACHE at all and/or to make sure that your JOB_INSTANCE_ID and JOB_EXECUTION_ID are always ascending with each new run.