I use coherence in my project with following configuration.
Everything is Ok with coherence 12.1.2, but after updating coherence to version 12.1.3, i have a problem. When i put for example 1000 item on cache one by one using NamedCache.put() there is no error, but when i put 1000 items on cache with calling NamedCache.putall once, coherence raises an exception.
My project is in java and is deployed on jboss.
cache configuration:
<distributed-scheme>
<scheme-name>my-map</scheme-name>
<service-name>MyMap</service-name>
<serializer>
<instance>
<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
<init-params>
<init-param>
<param-type>String</param-type>
<param-value>pof-config.xml</param-value>
</init-param>
</init-params>
</instance>
</serializer>
<thread-count>100</thread-count>
<local-storage>true</local-storage>
<backup-count>0</backup-count>
<backing-map-scheme>
<read-write-backing-map-scheme>
<internal-cache-scheme>
<ramjournal-scheme/>
</internal-cache-scheme>
<cachestore-scheme>
<class-scheme>
<class-name>myPackage.loader</class-name>
</class-scheme>
</cachestore-scheme>
</read-write-backing-map-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
Raised exception details:
Exception in thread "pool-7-thread-1" com.tangosol.net.RequestIncompleteException: Partial failure
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.validatePartialResponse(PartitionedCache.CDB:56)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$BinaryMap.putAll(PartitionedCache.CDB:123)
at com.oracle.common.collections.ConverterCollections$ConverterMap.putAll(ConverterCollections.java:1553)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ViewMap.putAll(PartitionedCache.CDB:5)
at com.tangosol.coherence.component.util.SafeNamedCache.putAll(SafeNamedCache.CDB:1)
at mypackage.CacheEnabledDataProvider.putOnCache(CacheEnabledDataProvider.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: Portable(com.tangosol.util.WrapperException): (Wrapped: Failed request execution for myMap service on Member(Id=1, Timestamp=2014-10-20 18:00:25.737, Address=192.168.70.101:8088, MachineId=5642, Location=site:,machine:PS,process:9876,member:Mem_1, Role=CoherenceServer) (Wrapped: Failed to store keys="9112791815, 9165497418, 9199193873, 9192139970, 9392020128, ") Assertion failed:) Assertion failed:
at com.tangosol.util.Base.ensureRuntimeException(Base.java:289)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.tagException(Grid.CDB:50)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onPartialCommit(PartitionedCache.CDB:7)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onPutAllRequest(PartitionedCache.CDB:85)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$PutAllRequest$PutJob.run(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:65)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:51)
at java.lang.Thread.run(Unknown Source)
at <process boundary>
at com.tangosol.io.pof.ThrowablePofSerializer.deserialize(ThrowablePofSerializer.java:57)
at com.tangosol.io.pof.PofBufferReader.readAsObject(PofBufferReader.java:3316)
at com.tangosol.io.pof.PofBufferReader.readObject(PofBufferReader.java:2604)
at com.tangosol.io.pof.ConfigurablePofContext.deserialize(ConfigurablePofContext.java:376)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.readObject(Service.CDB:1)
at com.tangosol.coherence.component.net.Message.readObject(Message.CDB:1)
at com.tangosol.coherence.component.net.message.responseMessage.DistributedPartialResponse.read(DistributedPartialResponse.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$PartialValueResponse.read(PartitionedCache.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.deserializeMessage(Grid.CDB:20)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:21)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:51)
at java.lang.Thread.run(Thread.java:745)
Caused by: Portable(com.tangosol.util.AssertionException): Assertion failed:
at com.tangosol.util.Base.azzertFailed(Base.java:209)
at com.tangosol.util.Base.azzert(Base.java:166)
at com.tangosol.net.cache.ReadWriteBackingMap$CacheStoreWrapper.storeAllInternal(ReadWriteBackingMap.java:5943)
at com.tangosol.net.cache.ReadWriteBackingMap$StoreWrapper.storeAll(ReadWriteBackingMap.java:5067)
at com.tangosol.net.cache.ReadWriteBackingMap.putAll(ReadWriteBackingMap.java:840)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.putAllPrimaryResource(PartitionedCache.CDB:7)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.postPutAll(PartitionedCache.CDB:27)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.putAll(PartitionedCache.CDB:14)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onPutAllRequest(PartitionedCache.CDB:62)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$PutAllRequest$PutJob.run(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:65)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:51)
at java.lang.Thread.run(Unknown Source)
at <process boundary>
at com.tangosol.io.pof.ThrowablePofSerializer.deserialize(ThrowablePofSerializer.java:57)
at com.tangosol.io.pof.PofBufferReader.readAsObject(PofBufferReader.java:3316)
at com.tangosol.io.pof.PofBufferReader.readObject(PofBufferReader.java:2604)
at com.tangosol.io.pof.PortableException.readExternal(PortableException.java:150)
at com.tangosol.io.pof.ThrowablePofSerializer.deserialize(ThrowablePofSerializer.java:59)
at com.tangosol.io.pof.PofBufferReader.readAsObject(PofBufferReader.java:3316)
at com.tangosol.io.pof.PofBufferReader.readObject(PofBufferReader.java:2604)
at com.tangosol.io.pof.ConfigurablePofContext.deserialize(ConfigurablePofContext.java:376)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.readObject(Service.CDB:1)
at com.tangosol.coherence.component.net.Message.readObject(Message.CDB:1)
at com.tangosol.coherence.component.net.message.responseMessage.DistributedPartialResponse.read(DistributedPartialResponse.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$PartialValueResponse.read(PartitionedCache.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.deserializeMessage(Grid.CDB:20)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onNotify(Grid.CDB:21)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:51)
at java.lang.Thread.run(Thread.java:745)
I found that if loader class defined in cachestore-scheme element, implements CacheStore instead of CacheLoader and leave store() and storeAll() empty, no exception is raised. According to oracle documents, implementing CacheStore is not necessary for read-only cache-store and implementing CacheLoader is sufficient. This might be a bug.
I am not sure implementing CacheStore with empty Store() and storeAll() methods when we need read-only cache-store has performance penalty or not?
I would only implement CacheLoader (not CacheStore) if you are not writing back to the database.
Related
Issue/Problem Description : I am not able to start server after spring cloud migration 2021 0 4 and Java 17 . Can any one help me on this
I have tried to upgrade various spring eureka jars . I need to start application without any errors
Error log :
java.lang.IllegalStateException: Error processing condition on org.springframework.cloud.netflix.eureka.config.DiscoveryClientOptionalArgsConfiguration.defaultEurekaClientHttpRequestFactorySupplier\n\t
at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:60)\n\t
at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:108)\n\t
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForBeanMethod(ConfigurationClassBeanDefinitionReader.java:193)\n\t
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:153)\n\t
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitions(ConfigurationClassBeanDefinitionReader.java:129)\n\t
at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:343)\n\t
at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:247)\n\t
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:311)\n\t
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:112)\n\t
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:746)\n\t
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:564)\n\t
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147)\n\t
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:731)\n\t
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408)\n\t
at org.springframework.boot.SpringApplication.run(SpringApplication.java:307)\n\t
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1303)\n\t
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1292)\n\t
at com.dt.rts.eregusa.discovery.Application.main(Application.java:12)\n\t
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\t
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\n\t
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\t
at java.base/java.lang.reflect.Method.invoke(Method.java:568)\n\t
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)\n\t
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)\n\t
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)\n\t
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)\n
Caused by: java.lang.IllegalStateException: Failed to introspect Class [org.springframework.cloud.netflix.eureka.config.DiscoveryClientOptionalArgsConfiguration] from ClassLoader [org.springframework.boot.loader.LaunchedURLClassLoader#61af1510]\n\t
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:485)\n\t
at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:361)\n\t
at org.springframework.util.ReflectionUtils.getUniqueDeclaredMethods(ReflectionUtils.java:418)\n\t
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.lambda$getTypeForFactoryMethod$2(AbstractAutowireCapableBeanFactory.java:765)\n\t
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1740)\n\t
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryMethod(AbstractAutowireCapableBeanFactory.java:764)\n\t
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineTargetType(AbstractAutowireCapableBeanFactory.java:703)\n\t
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.predictBeanType(AbstractAutowireCapableBeanFactory.java:674)\n\t
at org.springframework.beans.factory.support.AbstractBeanFactory.isFactoryBean(AbstractBeanFactory.java:1684)\n\t
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:570)\n\t
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:542)\n\t
at org.springframework.boot.autoconfigure.condition.OnBeanCondition.collectBeanNamesForType(OnBeanCondition.java:250)\n\t
at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getBeanNamesForType(OnBeanCondition.java:243)\n\t
at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getBeanNamesForType(OnBeanCondition.java:233)\n\t
at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getMatchingBeans(OnBeanCondition.java:181)\n\t
at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getMatchOutcome(OnBeanCondition.java:156)\n\t
at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:47)\n\t...
25 common frames omitted\n
Caused by: java.lang.NoClassDefFoundError: com/netflix/discovery/DiscoveryClient$DiscoveryClientOptionalArgs\n\t
at java.base/java.lang.ClassLoader.defineClass1(Native Method)\n\t
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1012)\n\t
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)\n\t
at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:524)\n\t
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:427)\n\t
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:421)\n\t
I am trying to populate a Gridgain cache which is running on a cluster on other machine. I'm getting ClientDisconnectedException while calling method on a Gridgain Cache and getting this after calling put method on the Cache.
Here is my cache Configuration:
// DPH cache
CacheConfiguration<K,V> DPHCacheCfg = new CacheConfiguration<>(DPH_CACHE);
DPHCacheCfg.setCacheMode(CacheMode.PARTITIONED); // Default.
DPHCacheCfg.setIndexedTypes(String.class, DPH.class);
DPHCacheCfg.setOffHeapMaxMemory(10 * 1024L * 1024L * 1024L);
DPHCacheCfg.setMemoryMode(CacheMemoryMode.ONHEAP_TIERED);
FifoEvictionPolicy evctPolicy = new FifoEvictionPolicy();
DPHCacheCfg.setEvictionPolicy(evctPolicy);
Her is how I put data in The Cache:
DPHCache.put(K, V); where V is some object. After certain number of puts, I am having the below exception.
avax.cache.CacheException: class org.apache.ignite.IgniteClientDisconnectedException: Operation has been cancelled (client node disconnected).
at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1615)
at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1955)
at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1155)
at com.elsevier.elssie.datafabric.LoadQuetzal.populateDPH(LoadQuetzal.java:246)
at com.elsevier.elssie.datafabric.LoadQuetzal.main(LoadQuetzal.java:163)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteClientDisconnectedException: Operation has been cancelled (client node disconnected).
at org.apache.ignite.internal.util.IgniteUtils$14.apply(IgniteUtils.java:829)
at org.apache.ignite.internal.util.IgniteUtils$14.apply(IgniteUtils.java:827)
... 11 more
Caused by: class org.apache.ignite.internal.IgniteClientDisconnectedCheckedException: Operation has been cancelled (client node disconnected).
at org.apache.ignite.internal.processors.cache.GridCacheMvccManager.disconnectedError(GridCacheMvccManager.java:406)
at org.apache.ignite.internal.processors.cache.GridCacheMvccManager.onDisconnected(GridCacheMvccManager.java:382)
at org.apache.ignite.internal.processors.cache.GridCacheSharedContext.onDisconnected(GridCacheSharedContext.java:151)
at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onDisconnected(GridCacheProcessor.java:934)
at org.apache.ignite.internal.IgniteKernal.onDisconnected(IgniteKernal.java:3023)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery(GridDiscoveryManager.java:588)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.notifyDiscovery(ClientImpl.java:2058)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.notifyDiscovery(ClientImpl.java:2039)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1435)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
This usually happens when all server nodes stop. You should check their logs to understand what exactly happened.
Also note that the client will reconnect automatically when at least one server is back. IgniteClientDisconnectedException has the reconnectFuture() method which returns the future that will be completed when the reconnection happens, so you can block the client until the cluster is functional.
I am using Titan 1.0.0 on top cassandra. I want to use OLAP services using SparkGraphComputer on the titan/cassandra graph. I have two questions
1) How to do it?
config:
https://github.com/thinkaurelius/titan/blob/titan10/titan-dist/src/assembly/static/conf/hadoop-graph/read-cassandra.properties
gremlin-code:
graph = GraphFactory.open('conf/hadoop-graph/read-cassandra.properties')
g = graph.traversal(computer(SparkGraphComputer))
g.V().count() //Here is the error
Error:
11:20:33 ERROR org.apache.spark.executor.Executor - Exception in task 3.0 in stage 0.0 (TID 3)
java.lang.RuntimeException: error communicating via Thrift
at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.<init>(ColumnFamilyRecordReader.java:267)
at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.<init>(ColumnFamilyRecordReader.java:215)
at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.<init>(ColumnFamilyRecordReader.java:331)
at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.<init>(ColumnFamilyRecordReader.java:331)
at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.initialize(ColumnFamilyRecordReader.java:171)
at com.thinkaurelius.titan.hadoop.formats.cassandra.CassandraBinaryRecordReader.initialize(CassandraBinaryRecordReader.java:39)
at com.thinkaurelius.titan.hadoop.formats.util.GiraphRecordReader.initialize(GiraphRecordReader.java:38)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:135)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:107)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Here is the full trace :
http://pastebin.com/CiuXjFB2
2) Why convert to hadoopGraph when the data is already stored on titan/cassandra?
references:
https://groups.google.com/forum/#!topic/gremlin-users/fVijONCxvSI
I have a service which uses apache HttpAsyncClient. (versions: httpasyncclient-4.0.2.jar, httpcore-4.4.3.jar, httpcore-nio-4.3.3.jar)
All requests start failing some time after starting the async client with following being initial exception -
[#|2016-03-16T22:31:59.376-0700|SEVERE|glassfish3.1.2|org.apache.http.impl.nio.client.InternalHttpAsyncClient|_ThreadID=564;_ThreadName=Thread-6;|I/O reactor terminated abnormally
org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:357)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:189)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute(CloseableHttpAsyncClientBase.java:67)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000(CloseableHttpAsyncClientBase.java:38)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:57)
at java.lang.Thread.run(Unknown Source)
Caused by: RestException(statusCode=500, code=null, message=I/O operation failed, developerMessage=RestException(statusCode=500, code=null, message=I/O operation failed, developerMessage=null)
at com.notificationservice.analytics.client.AsyncResponse$2.failed(AsyncResponse.java:178)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.failed(DefaultClientExchangeHandlerImpl.java:258)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:127)
at org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:68)
at org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:37)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:154)
at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:180)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
at java.lang.Thread.run(Unknown Source)
)
at com.notificationservice.client.AsyncResponse$2.failed(AsyncResponse.java:178)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.failed(DefaultClientExchangeHandlerImpl.java:258)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:127)
at org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:68)
at org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:37)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:154)
at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:180)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
... 1 more
Same problem happens with new versions - httpasyncclient-4.1.1.jar, httpcore-4.4.4.jar, httpcore-nio-4.4.4.jar
Any insight would be highly appreciated. Is there some IOReactorConfig parameter which needs to be changed?
I would say something is wrong with your rest parameters. StatusCode 500 comes from the server so your request are going to it.
Caused by: RestException(statusCode=500, code=null, message=I/O operation failed, developerMessage=RestException(statusCode=500, code=null, message=I/O operation failed, developerMessage=null
Have a batch which does some heavy operations. It runs for approximately 11-12 hours.
After that it moves to UNKNOWN state.
I have a question when would a batch move to UNKNOWN state?
Following is stack Trace.
org.springframework.transaction.TransactionSystemException: Could not roll back JDBC transaction; nested exception is java.sql.SQLException: Protocol violation
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:285)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:845)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:822)
at org.springframework.transaction.support.TransactionTemplate.rollbackOnException(TransactionTemplate.java:161)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:134)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:284)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135)
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:282)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:121)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:909)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.sql.SQLException: Protocol violation
at oracle.jdbc.driver.T4CTTIfun.receive
Thanks
Aditya
A batch job will move into the UNKNOWN state only when a rollback is unsuccessful, leaving the job in an uncertain state which is what it looks like happened here. The real question here is…why was the rollback unsuccessful?
Check org.springframework.batch.core.step.AbstractStep. It is full of:
try {
getJobRepository().updateExecutionContext(stepExecution);
}
catch (Exception e) {
stepExecution.setStatus(BatchStatus.UNKNOWN);
exitStatus = exitStatus.and(ExitStatus.UNKNOWN);
stepExecution.addFailureException(e);
}
So it happens on update* operations of getJobRepository() (same as #Minella stated).