In SonarQube (5.6.4 LTS) there is a view where background (project analysis) tasks are visualized: (Administration / Projects / Background Tasks). It seems like the tasks are run in sequence (one at a time). Some tasks could take 40 minutes which means other projects are queued up waiting for this task to finish before they could be started.
Is it possible to configure the SonarQube Compute Engine so that these tasks are run in parallel instead?
As per documentation on Background Tasks:
You can control the number of Analysis Reports that can be processed at a time in $SQ_HOME/conf/sonar.properties (see sonar.ce.workerCount - Default is 1).
Careful though: blindly increasing sonar.ce.workerCount without proper monitoring is just like shooting in the dark. The underlying resources available (CPU/RAM) are fixed (all workers run in the Compute Engine JVM), and you don't want to end-up with very limited memory for each task and/or high CPU-switching. That would kill performance for each of the tasks, rather than having only a few in parallel which will be much more efficient.
In short: better to have maximum 2 tasks in parallel that can complete under a minute (i.e. max 10 minutes to run 20 tasks), rather than 20 sluggish tasks in parallel that will overall take 15 minutes to complete because they struggle to share common CPU/RAM.
Update: with SonarQube 6.7+ and the new licence plans, "parallel processing of reports" has become a commercial feature and is only available in the Enterprise Edition.
Related
I am interested in performing weak scaling tests on an HPC cluster. In order to achieve this, I run several small tests on 1,2,4,8,16,32,64 nodes with each simulation taking less than a minute to maximum 1 hour. However, the jobs stay in queue (1 hour queue) for several days before the test results are available.
I have two questions:
Is there a way to prioritize the jobs in the job scheduler given that most tests are less than a minute for which I have to wait several days?
Can and to what extent such a job scheduling policy invite abuse of HPC resources. Consider a hypothetical example of an HPC simulation on 32 nodes, which is divided into several small 1 hour simulations that get prioritized because of the solution provided in point 1. above?
Note: the job scheduling and management system used at the HPC center is MOAB. Each cluster node is equipped with 2 Xeon 6140 CPUs#2.3 GHz (Skylake), 18 cores each.
Moab's fairshare scheduler may do what you want, or if it doesn't out of the box, may allow tweaking to prioritize jobs within the range you're interested in: http://docs.adaptivecomputing.com/mwm/7-1-3/help.htm#topics/fairness/6.3fairshare.html.
I'm trying to tune a Spark application, in order to reduce overall time execution, but I'm having a strange behaviour during a Stage execution.
Basically just 14/120 tasks needs around 20 min to finish, the others instead take 4 or 5 min to be completed.
Looking a the Spark UI, the partitioning seems good, the only difference I see is the GC Time that is very high for the 14 tasks.
I attach an image of the situation.
Do you have any idea for find the performance solution?
I had a similar problem and could resolve it by using Parallel GC instead of G1GC. You may add the following options to the executors additional Java options in the submit request
-XX:+UseParallelGC -XX:+UseParallelOldGC
Our application has a feature that requires a rake tast to run every day at a specific time over all the users. This involves computation of some of their attributes and running through db queries and sending a push notification to every user. As such, the task has been designed to run in O(n) but that would still mean growing total time to finish with increasing user base. And we want the task to finish in not more than a minute - it already is take 8 minutes at 14000 users and also ever increasing the CPU util (throughout the rest of the day, the average cpu util sits around 10% but goes up to 50% when the task runs). I want to solve two problems here - make the task run in lesser time and bring down the cpu util the task run spikes.
Tech Specs - Sinatra application serving an API for the app, running on Phusion Passenger (nginx module), using MongoDB and deployed on a c3.large ec2 instance.
P.S - I don't have much knowledge about how parallel processing and threading are done in Ruby and if it can solve this issue, but can bucketing the total users and paralelly computing those buckets be an answer? If so, how do I go about doing something like that? I want to avoid buying out a bigger server just for this purpose as rest of the time it handles the requests quite easily like I pointed out above.
I have 2 jobs, submitted under 2 templates in order to use priority levels.
I'm using Queued Scheduling, with Graceful Pre-emption and all the automatic resource adjustment enabled.
I submit the low priority job. At a later stage I allocate the higher priority job. HPC 2012 automatically takes resources from the lower priority task as the running tasks within the job complete.
HPC then does NOT reallocate those resources back the lower priority task when the higher priority one completes.
This gets even worse as the application submitting the tasks/jobs adds further tasks to running jobs as certain tasks complete, and I typically have only a handful of cores allocated to the low priority one, despite there being several hundred cores free once the high priority job completes.
Is there a way to change HPC configuration to do this?
OK, so the HPC Pack 2012 SP1 notes state that this exact issue has been fixed in SP1 for SOA mode. To clarify, it HAS also been fixed for Batch mode.
This is how it works (when using Batch mode, queued scheduling with graceful pre-emption)
Submit a low priority job
Scheduler allocates resources to the tasks under the low job.
Submit a high job
As tasks under the low priority job complete the resources are allocated to the tasks under the high priority.
Under HPC Pack 2012, the resources on the low job can be run right down to a single core and the job would carry on trying to finish with a single core allocated (unless you also put in conditions on minimum numbers allocated).
Under HPC Pack 2012 SP1, it will allocate resources to the high job more aggressively (it will take ALL the resources from the low job as the already-running tasks finish) and label the tasks as cancelled in the Activity Log.
As the resources free up as the high priority job's tasks complete these are re-allocated to the low priority job, which now just appears as a queued task, so normal task-level prioritisation/scheduling occurs, exactly as if this low task had been submitted separately. It therefore gets full resources re-allocated.
I want to run 50 tasks. All these tasks execute the same piece of code. Only difference will be the data. Which will be completed faster ?
a. Queuing up 50 tasks in a queue
b. Queuing up 5 tasks each in 10 different queue
Is there any ideal number of tasks that can be queued up in 1 queue before using another queue ?
The rate at which tasks are executed depends on two factors: the number of instances your app is running on, and the execution rate of the queue the tasks are on.
The maximum task queue execution rate is now 100 per queue per second, so that's not likely to be a limiting factor - so there's no harm in adding them to the same queue. In any case, sharding between queues for more execution rate is at best a hack. Queues are designed for functional separation, not as a performance measure.
The bursting rate of task queues is controlled by the bucket size. If there is a token in the queue's bucket the task should run immediately. So if you have:
queue:
- name: big_queue
rate: 50/s
bucket_size: 50
And haven't queue any tasks in a second all tasks should start right away.
see http://code.google.com/appengine/docs/python/config/queue.html#Queue_Definitions for more information.
Splitting the tasks into different queues will not improve the response time unless the bucket hadn't had enough time to completely fill with tokens.
I'd add another factor into the mix- concurrency. If you have slow running (more than 30 seconds or so) tasks, then AppEngine seems to struggle to scale up the correct number of instances to deal with the requests (seems to max out about 7-8 for me).
As of SDK 1.4.3, there's a setting in your queue.xml and your appengine-web.config you can use to tell AppEngine that each instance can handle more than one task at a time:
<threadsafe>true</threadsafe> (in appengine-web.xml)
<max-concurrent-requests>10</max-concurrent-requests> (in queue.xml)
This solved all my problems with tasks executing too slowly (despite setting all other queue params to the maximum)
More Details (http://blog.crispyfriedsoftware.com)
Queue up 50 tasks and set your queue to process 10 at a time or whatever you would like if they can run independently of each other. I see a similar problem and I just run 10 tasks at a time to process the 3300 or so that I need to run. It takes 45 minutes or so to process all of them but the CPU time used is negligible surprisingly.