My use case details are like below.
Sonarqube Version: 4.5.2
Ram: 16 GB
Code base size: in GBs
OS: Windows
Project Languages: Java, JavaScript
Project Type: Multi Module
Analysis takes half of the day like 12-13 hours. Need help in minimizing the same.
There could be a couple things address here
network latency - for the version you're running, you want to make sure the machine that performs the analysis is a close as possible on the network to your database
database contention - in 4.5.2, the scanner talks directly to the database. If analyses of other projects are happening concurrently, they could be interfering with each other. You can remove this problem by upgrading to the new L.T.S. version, 5.6, which fully cuts the ties from the scanner to the database. In 5.6 analysis reports are generated by the SonarQube scanner, and submitted to the server, where they're queued for processing and handled serially
Your project is just too darn big to analyze successfully in a reasonable amount of time. This may or may not be the case, but you should give it honest consideration, and potentially analyze components independently. Doing so would have the knock-on benefit of not reanalyzing the whole thing when one file in one module has changed. If you need to re-aggregate the results of the individual component analyses, you can do so with the Governance plugin($).
Related
Frequently last few days onwards our GitLab(CE) running slowly. We have a hook for the CI with Jenkins. We had installed the GitLab by OmniAuth. I don't have any more ideas regarding this because we didn`t do anything new in our instances.
We are the newbie to GitLab environment. We are working in the GitLab since December 2016 and also we never faced this kind of issue before. I hope that I will fix this problem with you people. Kindly help me to fix the issue.
Follow the below image for our Gitlab details.
How could I overcome from this issue?
These are just some as-is suggestions offered without warranty, but they may help guide you to solving the problem.
Occam's Razor
You mentioned that these issues appear to have just started most recently. This means that the VERY FIRST place to look is what may have changed around the time that these issues occurred. If you have change control for your infrastructure, start there. Make absolutely sure nobody has changed anything around the time these issues started happening. Check your logs for any warnings that may have started showing up. If your OS has a security log or logs configuration changes, check those. If you don't have good visibility/audit-ability into your environment, this may be hard, but if you can identify something that changed around the same time as these issues started occurring, that is most often going to be your problem.
Specificity
It may be helpful for you to describe what you mean by it getting slow. Is it a specific operation that is slow? Or is it all activity? If it's something specific, like triggering a Jenkins job, then you can start to isolate your search there.
It can also help to run top on your server to get a picture of what might be causing the issues. There might be a specific process running on the machine that is dominating everything else and eating all of the resources.
Hardware
First thing I would check is to make sure your hardware configuration matches the 'Hardware requirements' guidelines on gitlab's website:
https://docs.gitlab.com/ce/install/requirements.html#requirements
Based on what you've posted, the CPU and memory on your system seem adequate for several thousand users, so I'm going to assume this isn't a problem, but in case you do have thousands of users, I will add some brief information on this. Your disk configuration (other than size) is not presented in the information above, so we don't know if that is sufficient or not.
I would recommend running vmstat on the server (since it's GitLab, I am assuming this is running on Linux, since they do not recommend Windows installations) to get some basic information about what is going on. The vmstat command will give you several columns of information. To the very left there should be a column 'r'. This is the 'run queue', or the number of processes that are waiting to be run on a CPU. If the value in that column is large compared to the number of cores the system has, you probably have a CPU bottleneck. The next column, 'b', is processes that are blocked. If this is large, you probably don't have a CPU bottleneck. To the right, there are CPU columns: us, sy, id, or something along these lines. These columns are a breakdown of where the CPU is spending its time, either in the application code (us), in the OS code (sy), or waiting (id). High percentage numbers in us generally indicate that you either are running healthily or have a CPU bottleneck. High percentage numbers in sy are usually going to indicate some kind of contention, possibly a configuration issue like having too many worker threads configured for the number of CPUs you have. A high percentage number in id usually indicates that the system either isn't doing much, or can't do much because it's waiting on something like disk or an external database.
So if the 'b' and/or 'id' columns in your vmstat output have high numbers, we may want to consider the possibility of there being an I/O bottleneck. Here are a couple introductory articles on evaluating Linux IO for bottlenecks that might help you determine if this is the case:
https://bartsjerps.wordpress.com/2011/03/04/io-bottleneck-linux/
http://www.linux-mag.com/id/2001/
These articles should get you pointed in the right direction to help you decide if your disks aren't fast enough.
One thing to note, if you're seeing what appears to be a CPU bottleneck (high r values, high us values), make sure that situation makes sense for the number of users you have. The CPU bottleneck may be caused a virtualization issue, or some OS issue causing the CPU to perform poorly, not just by the CPU hardware itself being insufficient.
Topology
One thing mentioned in the gitlab requirements I linked to above is that it is not recommended to run GitLab runner on the same box as GitLab itself. This is something I would say is true for any CI software working with GitLab. If you're running GitLab Runner or Jenkins on the same box as GitLab itself, you should consider moving those to their own hardware.
If you have thousands of users, you may wish to get in contact with GitLab themselves and have consulting on how to get an enterprise-grade cluster stood up and what that looks like. There are people who are experts in the specific hardware configurations that make sense for a very large GitLab installation, and I am not one of them. However, if you don't have a large number of users, the hardware you have is probably not the issue.
Software
If you're running things like vmstat and iostat and you're not finding any specific hardware bottleneck, there may be a configuration issue. Make sure you have a good number of Unicorn Workers configured, so that the box can properly utilize your hardware.
External bottlenecks
Make sure things like network speed on the server are sufficient for its needs. Make sure users trying to reach the server aren't being bottlenecked by a misconfigured network. If you're using OmniAuth, make sure the provider is performing correctly. For example, if you're using some external authentication, and that isn't scaling/performing well, you'll get bad performance in GitLab as well. These are especially important to look at if you're not seeing much hardware utilization using the methods above.
Two aspects which can help accelerate GitLab are, in the latest April 2020 12.10 version:
the application server which switches back Puma
the caching of Git info/refs
The last point is:
When fetching changes from a Git repository, the server advertises a list of all the branches and tags in the repository, known as refs.
In some instances, we have observed up to 75% of all requests to the GitLab web server are requests for the refs.
In the best case, when all the refs are packed, this is an inexpensive operation.
However, when there are unpacked refs, Git must iterate over the unpacked refs. This causes additional disk I/O, which is slow when using high latency storage like NFS.
In GitLab 12.10, info/refs are cached to improve the performance of ref advertisement and decrease the pressure on Gitaly in situations where refs are fetched very frequently.
In testing this feature on GitLab.com, we observed read operations outnumber write operations 10 to 1, and saw median latency decrease by 70%.
For GitLab instances using NFS for Git storage, we expect even greater improvements.
See Documentation and Issue.
We are planning to migrate our enterprise application currently running on Java5 stack to Java7 stack. We are having issues with implicit gc calls (mainly major gc) causing system to be unstable for a short time(ranging from 5 mins - 30 mins). After analyzing the gc stats, we found that Compact phase is taking quite long time to complete when compared to Mark and sweep phase. I understand compaction is quite complex and time taking but its impacting the app server which is customer facing and few connections being dropped off during this phase.
Now, my question is as we are migrating to Java7, is there a better garbage collection process compared to Java5?
App servers are provided with decent system resources.
Each app server contains 32 cpu cores
contains 64 gb ram
App server is IBM webpshere server
Operating System - 64 bit IBM AIX
As said earlier, gc is happening because of implicit system calls. No explicit system calls invoking gc.
Now, my question is as we are migrating to Java7, is there a better garbage collection process compared to Java5?
Generally yes, though as #Pushkar, you should really be migrating to Java 8.
With respects to the specifics of your application(s), it sounds like you need to tune / retune the garbage collection on Java 5. If you are periodically experiencing 5 to 30 >>minutes<< of unstability due to GC, there is something rather wrong. The current behavior may be due your application or Websphere (e.g. memory leaks, excessive caching, etc), or it may be due to poor GC tuning.
In short, switching to Java 7 (or 8) might make things better "out of the box", but it is likely that you will need to put in more effort to address the underlying cause of your problems.
Finally, I'd advise the obvious things.
Implement the changes in small steps. Don't upgrade your app, websphere version, java version, etc all at the some time.
Do the upgrades of your servers one at a time. Have a roll-back plan in case you get unacceptable performance.
If possible test it all first ... including performance / load testing.
By default, java 7 uses parallelGC on server class machines. If you are using JDK 7 update 4 or later version, switch to G1 garbage collector which might give you better performance. But as #the8472 suggested,it will be good to know what settings you used in java 5 and now in your current environment.
Java 7 reached end of life around April 2015. Why not migrate to 1.8?
GC performance usually improves with java major releases (and in some cases with minor GCs).
You should take a look difference GC tuning flags, following link may help you
http://www.oracle.com/technetwork/articles/java/vmoptions-jsp-140102.html
http://stas-blogspot.blogspot.com/2011/07/most-complete-list-of-xx-options-for.html
I get that Postgres scales automatically to multicore with multiple connections, but what about when I'm running a massive query on a SINGLE connection? So frustrating that the CPU usage maxes out at 25% on my 4-core system.
I'm in process of switching from SQL Server and this is the only thing so far that really bugs me. SQL Server will use up to 100% of my CPU for a single connection/query.
I'm running 9.2 on Windows 7 Enterprise 64-bit with Xeon processor if it matters.
If there is not way to get around this, could someone address why this isn't seen as an issue? Is it because Postgres is focused on multi-user scenarios?
PostgreSQL does not currently support executing a single query across multiple CPU cores (minus background things like background writing and wal writing if you're doing a write query, but that doesn't really count). It's work that's in progress, but it's a long-term project, and is not in any current version of PostgreSQL.
This is the same on all platforms and architectures.
It is definitely an issue, but since PostgreSQL is, as you say, focused on multi user scenarios, it's not bubbled to the top of the priority queue until recently. But there are definitely people realizing it's an issue, and working on solving it for future versions, it's just not done yet.
There is a Foreign Data Wrapper that aims to add parallelism via the GPU called pg_strom. I've never used it, and it looks quite specialized, but maybe you (or someone here) has a use-case for it.
Article describing pg_strom
http://gpuscience.com/software/postgresql-gpu-pgstrom/
The code:
https://github.com/kaigai/pg_strom
It's not that it isn't seen as a problem. It's that it requires fundamental architectural changes. The use case for it is pretty specialised. It would only help on data warehouse type environments where you're executing long queries one at a time -- AND the queries are CPU bound, not disk i/o bound as they would usually be.
I'm just wondering if there's anyway to let sonar use multiple cores when analyzing the code.
Or if there's any other ways to improve performance overall.
Thanks
At the moment most of the analyzers sonar uses are single threaded. I suggested to analyse several plugins parallel. https://jira.codehaus.org/browse/SONAR-3025
You can vote if you want, but at the moment there is no way to use all cores. If you have many projects you could just analyse them in parallel manually.
Unless the sonar process is mult-threaded it's unlikely the runner can take major advantage of your multi-core machine.
The biggest performance boosts I can recommend are:
Run your build on the same machine as the Sonar database. JDBC calls across a WAN network can seriously affect your Sonar build times.
If you use the views plugin consider reducing the number of views. This creates lots of extra JDBC calls.
I'm involved with a project using DotNetNuke version 05.01.04 Community Edition. We are building our new Intranet using it, but performance is terrible.
We have five people adding pages and content to it and every 15-30 seconds they experience a pause of 10 seconds or longer before the system continues and the next screens loads.
The server is Windows 2003, 3.8GHz with 1GB of RAM. I'm told by our server admin that the CPU and memory performance don't appear to be the bottleneck.
We currently have 350 pages in the system, we a plan to add 1000. So we need to resolve this performance problem so that we can enter content and so we can go live.
I just can't see where the bottleneck is. Is there a good why to determine the bottleneck when using DotNetNuke?
Modules installed
Publish:Engage (Not currently in
use)
Page Blaster (Doesn't appear
to providing caching when users
logged in using Integrated
Authentication)
SimpleGallery
XMod
Content Manager
IIS Setup
Application recycling completely disabled (Apart from a 2am recycle)
New findings: 18th March 2010
The main bottleneck was due to version 5.1.4 having a bug which caused 1300 database roundtrips on an average page, due to broken database in-memory caching. We've upgraded to 5.2.4 which has resolved this bottleneck.
Now the next biggest bottleneck is the navigation. We've used both DDR:Menu and DDN:Nav, but both have a major impact on performance.
Is there a navigation interface out there that doesn't drain performance so badly?
I think you need to start investigating this using performance profiling tools. For the DNN application itself I'd grab something like JetBrains DotTrace or Red Gate's ANTS Performance Profiler.
For the database SQL Server Profiler would be the first choice or a tool such as Red Gate's SQL Response.
Without profiling the application these you're going to be pulling at straws.
And as Tim pointed out in his comment, installing Firebug in Firefox with the YSlow add-in to see what resources are taking longest to serve to the browser.
Mitchel Sellers has some good tutorials and checklists to go through with regards to performance in DNN. Start with Explaining High Performance DotNetNuke Configuration and Management (which points to some of his earlier articles).
I have several years of dnn development and maintainance experience, when I have this kind of problem, I start doing things from database clean up. Next thing is, find for missing indexes, and/or rebuild all the indexes periodically (sql job scheduled for that) but major performance gain would be from clean up of table
Another good considerations would be, disabling trace, debug mode to false and turn off features of dnn that you don't use (scheduler is the first one to turn off)
Edit: consider keep alive as well
Hope this helps
Is your database on that server? If so, just throw in some more RAM, or get a faster disk array...
Have you considered creating this lot of pages directly through TSQL? It's not hard to do and may save you a lot of time.