Following are the properties I have set -
spring.task.execution.pool.core-size=50
spring.task.execution.pool.max-size=200
spring.task.execution.pool.queue-capacity=100
spring.task.execution.shutdown.await-termination=true
spring.task.execution.shutdown.await-termination-period=10s
spring.task.execution.thread-name-prefix=async-task-exec-
I still see thread names as - "async-task-exec-7200"
Does it mean it is creating 7200 threads?
Also, another issue I observed that #Async would wait for more than 10min to get a thread and relieve the parent thread.
You specified core size of 50 and max size of 200. So your pool will normally run with 50 threads, and when there is extra work, it will spawn additional threads, you'll see "async-task-exec-51", "async-task-exec-52" created and so on. Later, if there is not enough work for all the threads, the pool will kill some threads to get back to just 50. So it may kill thread "async-task-exec-52". The next time it has too much work for 50 threads, it will create a new thread "async-task-exec-53".
So the fact that you see "async-task-exec-7200" means that over the life time of the thread pool it has created 7200 threads, but it will still never have more than the max of 200 running at the same time.
If #Async method is waiting 10 minutes for a thread it means that you have put so much work into the pool that it has already spawned all 200 threads and they are processing, and you have filled up the queue capacity of 100, so now the parent thread has to block(wait) until there is at least a spot in the queue to put the task.
If you need to consistently handle more tasks, you will need a powerful enough machine and enough max threads in the pool. But if your work load is just very spiky, and you don't want to spend on a bigger machine and you are ok with tasks waiting longer sometimes, you might be able to get away with just raising your queue-capacity, so the work will queue up and eventually your threads might catch up (if the task creation gets slower).
Keep trying combinations of these settings to see what will be right for your workload.
Related
I copied this code from the windows samples
auto workItemHandler = ref new WorkItemHandler([this](IAsyncAction ^ action)
{
while (action->Status == AsyncStatus::Started)
{
// Main Loop
}
});
m_renderLoopWorker = ThreadPool::RunAsync(workItemHandler, WorkItemPriority::High, WorkItemOptions::TimeSliced);
but have experienced some unreproducible lag sometimes (although maybe its from the gpu).
On the other hand
WorkItemOptions::TimeSliced The work items should be run simultaneously with other work items sharing a processor.
doesn't sound like a high performance option.
WorkItemOptions::None The work item should be run when the thread pool has an available worker thread.
Where you would want to use WorkItemOptions::TimeSliced vs WorkItemOptions::None?
Is it ever advisable to use CreathThread over running a task on the thread pool for persistent work.
WorkItemOptions::TimeSliced => preemptive multitasking
WorkItemOptions::None => cooperative multitasking
When do you want to use each one... difficult to say.
If you use None and all the threads in the thread pool are currently used, your task wont run until a thread finishes its job.
With TimeSliced each task is allowed a time slice, when the time is up, your task is paused and the thread switch to another task. This way, if you have 100 work items, but only 10 thread, all work items will progress, little by little, but 10x slower.
If you need to update something regularly, lets say a progress bar, you would rather use TimeSliced.
It is perfectly acceptable to use CreateThread for a long task. A render loop fit that description. This way you have your own thread to yourself to do whatever you want. Even though at the OS level, there is preemptive multitasking anyway, otherwise if your processor had only 2 cores, and you ran 3 threads, the 3rd thread would hang.
The main point of thread pools is to avoid creating new threads for every little task you want to do, because it incurs an overhead.
I have a jmeter script where in I initiate the test with concurrent 300 users(threads), then with every 10sec, 100 more threads are added. My script was running absolutely fine. But suddenly I am noticing that inspite of adding 100 threads, its adding 3 or 4 threads.
I have increased swap memory, still no go.
I am using 4GB machine, in which i have made "Xms=1g Xms=3GB"
And this is how slowly threads are getting added:
Here is how "absolutely fine" execution of your scenario should look like:
Given you have that many threads in the Finished state there is something preventing them from doing their normal job so instead of finishing execution of all Samplers of the current iteration and starting the new one the threads are being terminated somewhere in between and JMeter restarts them.
Your question doesn't contain sufficient information in order to guess the reason, inspect your test plan and make sure you have enough test data, in case of CSV Data Set Config ensure that Stop Thread on EOF is set to False, etc.
If your test plan looks okayish check jmeter.log file for any suspicious entries
I have a system which starts a new thread with each request to the application.
if application received hundreds of requests there may be not enough memory available to start a new thread so it will throw an exception.
I would like to know an ideal mechanism to handle this kind of a situation.
like, if application is receiving lots of request then if there is not enough memory or number of active threads reached the max then i will delay processing other requests.
but i have no idea how to implement this .
Easy solution: Increase thread-pool limits. This is actually viable although out of fashion these days.
More thorough solution: Use a SemaphoreSlim to limit the number of concurrently asynchronously active requests. Make sure to wait asynchronously. If you wait synchronously you'll again burn a thread while waiting. After having waited asynchronously you can resume normal synchronous blocking processing. This requires only small code changes.
Most thorough solution: Implement your processing fully async. That way you never run out of threads.
I want to start several Linux kernel threads using kthread_create (not the kthread_run), but in my driver there are some probability that some of threads will be not woken up with wake_up_process.
Is it correct to create all threads with kthread_create and not wake they up?
I think, some threads may get stuck in TASK_UNINTERRUPTABLE.
The problem is that I can't wake up the thread before the data for thread will be ready. If I do, then thread will try to parse unavailable data. And sometimes there will be data not for all threads.
Also, I can't start the thread at time when data will be available because this is too long to start the thread for my requirements.
I have an application with the following pattern:
2 long running processes that go into hibernate after some idle time
and their memory consumption goes down as expected
N (0 < N < 100) worker processes that do some work and hibernate when idle more than
10 seconds or terminate if idle more than two hours
during the night,
when there is no activity the process memory goes back to almost the
same value that was at the application start, which is expected as
all the workers have died.
The issue is that "system" section keeps growing (around 1GB/week).
My question is how can I debug what is stored there or who's allocating memory in that area and is not freeing it.
I've already tested lists:keysearch/3 and it doesn't seem to leak memory, as that is the only native thing I'm using (no ports, no drivers, no NIFs, no BIFs, nothing). Erlang version is R15B03.
Here is the current erlang:memory() output (slight traffic, app started on Feb 03):
[{total,378865650},
{processes,100727351},
{processes_used,100489511},
{system,278138299},
{atom,1123505},
{atom_used,1106100},
{binary,4493504},
{code,7960564},
{ets,489944},
{maximum,402598426}]
This is a 64-bit system. As you can see, "system" section has ~270MB and "processes" is at around 100MB (that drops down to ~16MB during the night).
It seems that I've found the issue.
I have a "process_killer" gen_server where processes can subscribe for periodic GC or kill. Its subscribe functions are called on each message received by some processes to postpone the GC/kill (something like re-arm).
This process performs an erlang:monitor if not already monitored to catch a dead process and remove it from watch list. If I comment our the re-subscription line on each handled message, "system" area seems to behave normally. That means it is a bug in my process_killer that does leak monitor refs (remember you can call erlang:monitor multiple times and each call creates a reference).
I was lead to this idea because I've tested a simple module which was calling erlang:monitor in a loop and I have seen ~13 bytes "system" area grow on each call.
The workers themselves were OK because they would die anyway taking their monitors along with them. There is one long running (starts with the app, stops with the app) process that dispatches all the messages to the workers that was calling GC re-arm on each received message, so we're talking about tens of thousands of monitors spawned per hour and never released.
I'm writing this answer here for future reference.
TL;DR; make sure you are not leaking monitor refs on a long running process.