Thread underneath coroutineScope does not stop - spring

i have an issue with stopping coroutine Thread. i don't have deep knowledge of Coroutines, correct me if i am wrong in any part, but i was not able to find the solution anywhere.
The issue which i am facing is that when i call coroutineScope.cancel() it does not stop the thread underneath, the coroutine is being stopped and it does not keep on functioning but the thread underneath moves to WAITING state.
The way i start my coroutine ->
scope = CoroutineScope(
newFixedThreadPoolContext(
request.size,
"Sync Proxy connection threads dispatcher"
)
)
In launching process, i save Job in ConcurrentHashMap ->
request.forEach {
jobs[it] = (scope.launch {
runConnectorProcess(it)
})
}
then based on some logic i call cancel ->
jobs[request]?.let {
it.cancel()
jobs.remove(syncRequest)
}
and if it was the last element canceled -> scope.cancel()
During debugging i found out in Thread Dump, that this Thread has State of WAITING
Logs from VisualVM ->
"Sync Proxy connection threads dispatcher" #168 daemon prio=5 os_prio=31 tid=0x00007fa78281b800 nid=0xd42f waiting on condition [0x000000030f563000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000077b305878> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2044)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

You shouldn't expect that cancelling the CoroutineScope would also close the CoroutineDispatcher that it's referencing, because the scope doesn't own the dispatcher. The dispatcher could be used by multiple scopes and individual coroutines. It's only a parameter passed to the CoroutineScope constructor, after all. If you want to shut down your CoroutineDispatcher, you need to store it in its own property so you can call close() on it when you need to.

Related

Why won't Micrometer stop sending data to datadog and just close already?

I have a Spring Boot app which due to weird restrictions needs to run once every three hours, and won't work with Quartz, so I've been running it once every three hours from OS cron and it quits when it's done.
After adding micrometer-registry-datadog (and spring-legacy) however, it never quits, it just sends metrics every 20 seconds or whatever the default period is, even after calling registry.close().
Am I doomed like the dutchman to sail the seas of processing forever, or is there an obvious error I have made?
Code: It reaches SpringApplication.exit(ctx), but it does not actually exit cleanly. (service is a TimedExecutorService.)
public void close() throws InterruptedException {
service.shutdown();
service.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
meterRegistry.close();
SpringApplication.exit(ctx);
}
This sounds like a bug. It is possible the Datadog exporter is running in a non-daemon thread. The JVM views non-daemon threads as application critical work.
So essentially the JVM thinks it shouldn't shutdown until the non-daemon thread finishes. In the case of the Datadog exporter thread, that probably won't happen.
To verify there are non-daemon threads, use jstack to generate a thread dump. (command: jstack <pid>) or dump all threads in your close method:
ThreadMXBean threadMxBean = ManagementFactory.getThreadMXBean();
for (ThreadInfo ti : threadMxBean.dumpAllThreads(true, true)) {
System.out.print(ti.toString());
}
An example thread dump output is below. Notice the word 'daemon' on the first line:
"pool-1-thread-1" #13 prio=5 os_prio=31 tid=0x00007fe885aa5000 nid=0xa907 waiting on condition [0x000070000d67b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c07e9720> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

spring-integration-aws: SqsMessageDrivenChannelAdapter threads waiting for each other when calling stop()

I was trying to stop the SqsMessageDrivenChannelAdapter when calling stop() I was receiving the following error:
java.util.concurrent.TimeoutException: null
at java.util.concurrent.FutureTask.get(FutureTask.java:205)
at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.stop(SimpleMessageListenerContainer.java:197)
at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.doStop(SimpleMessageListenerContainer.java:133)
at org.springframework.cloud.aws.messaging.listener.AbstractMessageListenerContainer.stop(AbstractMessageListenerContainer.java:329)
at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.stop(SimpleMessageListenerContainer.java:1)
at org.springframework.integration.aws.inbound.SqsMessageDrivenChannelAdapter.doStop(SqsMessageDrivenChannelAdapter.java:140)
at org.springframework.integration.endpoint.AbstractEndpoint.stop(AbstractEndpoint.java:111)
First indications I thought it was a time out was not long enough, after looking into the issue I found the stopping thread and the running thread are waiting for each other and the TimeoutException would always happen. I fixed this by the follow piece of code:
new Thread() {
public void run()
{
sqsMessageRawDataChannelAdapter.stop();
LockSupport.parkNanos(TimeUnit.SECONDS.toNanos(5));
}
}.start();
I now no longer get the TimeoutException as the thread is able to stop. Have I done something wrong, or is this an issue?
M-m-m. Looks like a deadlock in your code.
You try to stop an adapter (and therefore listener) from thread which is held by the listener.
In your terms stopping thread == running thread.
To avoid such a problem you definitely should shift the stop operation to different Thread.

Blocked threads due to ServiceRegistry

I have a relatively high volume/requests system in which we are using OSGi. We get close to 800M requests per day.
We are seeing some issues currently where threads are getting blocked. For every request that comes in, we forward the event/data to an osgi bundle using registerService to pass the payload/data to the OSGi bundle which is listening to this service.
Like this: bundleContext.registerService(Map.class.getName(), dataHolderMap, null);
Where dataHolderMap is nothing but a regular java hashmap
Here is the threaddump using JStack:
===========================================================================
"RequestThread" prio=10 tid=0x00000000421ab800 nid=0x1042 runnable [0x00007fbdd3867000]
java.lang.Thread.State: RUNNABLE
at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:295)
- locked <0x0000000700e2c590> (a org.apache.felix.framework.ServiceRegistry)
at org.apache.felix.framework.Felix.getService(Felix.java:3568)
at org.apache.felix.framework.BundleContextImpl.getService(BundleContextImpl.java:468)
at org.osgi.util.tracker.ServiceTracker.addingService(ServiceTracker.java:411)
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:932)
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:864)
at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)
at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:229)
at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:894)
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:932)
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4419)
at org.apache.felix.framework.Felix.registerService(Felix.java:3423)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:320)
at com.mypackage.person.bs.processor.Processor.sendEvent(Processor.java:56)
at com.mypackage.jetMyStream.event.support.AbstractEventSource.fireSendEvent(AbstractEventSource.java:97)
at com.mypackage.jetMyStream.event.channel.messaging.InboundMessagingChannel.fireEvent(InboundMessagingChannel.java:113)
at com.mypackage.jetMyStream.event.channel.messaging.InboundMessagingChannel.onMessage(InboundMessagingChannel.java:204)
at com.mypackage.jetMyStream.messaging.MessageService.dispatchMessageForContext(MessageService.java:349)
at com.mypackage.jetMyStream.messaging.MessageService.dispatch(MessageService.java:259)
at com.mypackage.jetMyStream.messaging.MessageServiceRequest.execute(MessageServiceRequest.java:40)
at com.mypackage.jetMyStream.util.RequestThread.run(RequestThread.java:69)
"RequestThread" prio=10 tid=0x00000000425f8800 nid=0x1041 waiting for monitor entry [0x00007fbdd3968000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.felix.framework.ServiceRegistry.registerService(ServiceRegistry.java:109)
- waiting to lock <0x0000000700e2c590> (a org.apache.felix.framework.ServiceRegistry)
at org.apache.felix.framework.Felix.registerService(Felix.java:3393)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:320)
at com.mypackage.person.bs.processor.Processor.sendEvent(BullseyeModelProcessor.java:56)
at com.mypackage.jetMyStream.event.support.AbstractEventSource.fireSendEvent(AbstractEventSource.java:97)
at com.mypackage.jetMyStream.event.channel.messaging.InboundMessagingChannel.fireEvent(InboundMessagingChannel.java:113)
at com.mypackage.jetMyStream.event.channel.messaging.InboundMessagingChannel.onMessage(InboundMessagingChannel.java:204)
at com.mypackage.jetMyStream.messaging.MessageService.dispatchMessageForContext(MessageService.java:349)
at com.mypackage.jetMyStream.messaging.MessageService.dispatch(MessageService.java:259)
at com.mypackage.jetMyStream.messaging.MessageServiceRequest.execute(MessageServiceRequest.java:40)
at com.mypackage.jetMyStream.util.RequestThread.run(RequestThread.java:69)
A couple of question on this:
Am I doing something wrong in sending a map through registerService. If so what are my alternatives?
Any ideas on how I can get this to work? We have 26 nodes and need to only process about 400 requests per second using this mechanism
Anyone had similar issues? Any pointers are highly appreciated
Thanks
Masti
"Am I doing something wrong in sending a map through registerService. If so what are my alternatives?"
YES! The service registry is NOT intended to be used this way; it is not a messaging bus.
As for alternatives... why not use a messaging bus? You could look at OSGi Event Admin but there are many other implementations of this idea.
"Any ideas on how I can get this to work? We have 26 nodes and need to only process about 400 requests per second using this mechanism"
You have to be a bit more specific about what you're trying to achieve. The question is too abstract. Please describe where the messages are coming from, where they need to be delivered, what processing (if any) you need to do in the middle, etc.

Gracefully stop DefaultMessageListenerContainer having a receiveTimeout = -1

Can somebody point me to a way to grecefully stop a DefaultMessageListenerContainer in Spring that is defined with a receiveTimeout = -1 (i.e. it is using the blocking version of javax.jms.MessageConsumer#receive) ?
Actually, I've tried both stop() and shutdown() methods of DMLC but consumer thread is stuck on the receive() call.
"heartBeatContainer-1" prio=6 tid=0x0304d800 nid=0x1d20 in Object.wait() [0x037ef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x22c48ab0> (a java.lang.Object)
at java.lang.Object.wait(Object.java:485)
at org.apache.activemq.SimplePriorityMessageDispatchChannel.dequeue(SimplePriorityMessageDispatchChannel.java:87)
- locked <0x22c48ab0> (a java.lang.Object)
at org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:452)
at org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:504)
at org.springframework.jms.connection.CachedMessageConsumer.receive(CachedMessageConsumer.java:70)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:430)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1069)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1061)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:958)
at java.lang.Thread.run(Thread.java:662)
I cannot close the connection as there are other listeners that should continue to consume messages.
That's the whole point of the receivetimeout - so the threads can check if the container has been shut down. Setting it to -1 means the threads will block in the JMS client until there's a message. There's not much overhead in timing out, checking the container state and doing another receive. If you don't like the default, set it to a higher number (as long as you can live with a delay when shutting down the container). But, setting it to -1 will cause the behavior you are seeing.

Make parent thread wait till child thread finishes in VC

According to MSDN:
The WaitForSingleObject function can wait for the following objects:
Change notification
Console input
Event
Memory resource notification
Mutex
Process
Semaphore
Thread
Waitable timer
Then we can use WaitForSingleObject to make the parent-thread wait for child ones.
int main()
{
HANDLE h_child_thread = CreateThread(0,0, child, 0,0,0); //create a thread in VC
WaitForSingleObject(h_child_thread, INFINITE); //So, parent-thread will wait
return 0;
}
Question
Is there any other way to make parent-thread wait for child ones in VC or Windows?
I don't quite understand the usage of WaitForSingleObject here, does it mean that the thread's handle will be available when the thread terminates?
You can establish communication between threads in multiple ways and the terminating thread may somehow signal its waiting thread. It could be as simple as writing some special value to a shared memory location that the waiting thread can check. But this won't guarantee that the terminating thread has terminated when the waiting thread sees the special value (ordering/race conditions) or that the terminating thread terminates shortly after that (it can just hang or block on something) and it won't guarantee that the special value gets ever set before the terminating thread actually terminates (the thread can crash). WaitForSingleObject (and its companion WaitForMultipleObjects) is a sure way to know of a thread termination when it occurs. Just use it.
The handle will still be available in the sense that its value won't be gone. But it is practically useless after the thread has terminated, except you need this handle to get the thread exit code. And you still need to close the handle in the end. That is unless you're OK with handle/memory leaks.
for the first queation - yes. The method commonly used here is "Join". the usage is language dependant.
In .NET C++ you can use the Thread's Join method. this is from the msdn:
Thread* newThread = new Thread(new ThreadStart(0, Test::Work));
newThread->Start();
if(newThread->Join(waitTime + waitTime))
{
Console::WriteLine(S"New thread terminated.");
}
else
{
Console::WriteLine(S"Join timed out.");
}
Secondly, the thread is terminated when when you are signaled with "WaitForSingleObject" but the handle is still valid (for a terminated thread). So you still need to explicitly close the handle with CloseHandle.

Resources