I'm just wondering if there's anyway to let sonar use multiple cores when analyzing the code.
Or if there's any other ways to improve performance overall.
Thanks
At the moment most of the analyzers sonar uses are single threaded. I suggested to analyse several plugins parallel. https://jira.codehaus.org/browse/SONAR-3025
You can vote if you want, but at the moment there is no way to use all cores. If you have many projects you could just analyse them in parallel manually.
Unless the sonar process is mult-threaded it's unlikely the runner can take major advantage of your multi-core machine.
The biggest performance boosts I can recommend are:
Run your build on the same machine as the Sonar database. JDBC calls across a WAN network can seriously affect your Sonar build times.
If you use the views plugin consider reducing the number of views. This creates lots of extra JDBC calls.
Related
Is there any tool or plugin which we can incorporate with existing Functional Automation Test suite.
Earlier till 2011-12 we were using Dynatrace Ajax edition. It was a plugin for Chrome and IE. We had configured this plugin into Selenium Test case. When these Selenium test were executing it was running all functional scenarios of application and parallel that Dynatrace tool was capturing performance stats for each actions performing on the application. At last we were getting a consolidate Performance report along with the Selenium test results.
But now I am not able to get any such kind of tool,plugin etc. which will help us to capture Performance stats of application with Selenium suit run.
Please help me in finding suitable way for it.
Yes, if you are using ChromeDriver you can enable performance logging. I believe this is what you are looking for.
DesiredCapabilities cap = DesiredCapabilities.chrome();
LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
cap.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
RemoteWebDriver driver = new RemoteWebDriver(new URL("http://127.0.0.1:9515"), cap);
The above code is what you will use the enable the logging for your session.
You don't need any kind of plugin/extensions for this. It uses Chrome's own performance logging feature.
Let me know if you are looking for this specifically.
My answer is more of an opinion than a true answer to your question, but IMHO it's not so useful to measure the performance as part of your normal functional test cycle.
Measuring performance as part of a regular functional test cycle can provide you with a lot of performance data, but it won't tell you much. If you want to measure performance effectively, you should start with the question: what operation do I care to measure? And "Everything" is not the right answer... Then you have to define what performance do expect and under which circumstances. Then you should build a test and a corresponding environment to match these requirements. In addition, performance is usually not a fixed value, as it can be affected by many factors that we can't control (like external processes that may be running in the background). Therefore you should usually define the expected performance in statistical terms, like: 90% of the times, the measured operation should take no more than 3 seconds. This means that you should run the test at least 10 times (actually much more in order to be safe) in order to determine if the performance is good enough or not.
Frequently last few days onwards our GitLab(CE) running slowly. We have a hook for the CI with Jenkins. We had installed the GitLab by OmniAuth. I don't have any more ideas regarding this because we didn`t do anything new in our instances.
We are the newbie to GitLab environment. We are working in the GitLab since December 2016 and also we never faced this kind of issue before. I hope that I will fix this problem with you people. Kindly help me to fix the issue.
Follow the below image for our Gitlab details.
How could I overcome from this issue?
These are just some as-is suggestions offered without warranty, but they may help guide you to solving the problem.
Occam's Razor
You mentioned that these issues appear to have just started most recently. This means that the VERY FIRST place to look is what may have changed around the time that these issues occurred. If you have change control for your infrastructure, start there. Make absolutely sure nobody has changed anything around the time these issues started happening. Check your logs for any warnings that may have started showing up. If your OS has a security log or logs configuration changes, check those. If you don't have good visibility/audit-ability into your environment, this may be hard, but if you can identify something that changed around the same time as these issues started occurring, that is most often going to be your problem.
Specificity
It may be helpful for you to describe what you mean by it getting slow. Is it a specific operation that is slow? Or is it all activity? If it's something specific, like triggering a Jenkins job, then you can start to isolate your search there.
It can also help to run top on your server to get a picture of what might be causing the issues. There might be a specific process running on the machine that is dominating everything else and eating all of the resources.
Hardware
First thing I would check is to make sure your hardware configuration matches the 'Hardware requirements' guidelines on gitlab's website:
https://docs.gitlab.com/ce/install/requirements.html#requirements
Based on what you've posted, the CPU and memory on your system seem adequate for several thousand users, so I'm going to assume this isn't a problem, but in case you do have thousands of users, I will add some brief information on this. Your disk configuration (other than size) is not presented in the information above, so we don't know if that is sufficient or not.
I would recommend running vmstat on the server (since it's GitLab, I am assuming this is running on Linux, since they do not recommend Windows installations) to get some basic information about what is going on. The vmstat command will give you several columns of information. To the very left there should be a column 'r'. This is the 'run queue', or the number of processes that are waiting to be run on a CPU. If the value in that column is large compared to the number of cores the system has, you probably have a CPU bottleneck. The next column, 'b', is processes that are blocked. If this is large, you probably don't have a CPU bottleneck. To the right, there are CPU columns: us, sy, id, or something along these lines. These columns are a breakdown of where the CPU is spending its time, either in the application code (us), in the OS code (sy), or waiting (id). High percentage numbers in us generally indicate that you either are running healthily or have a CPU bottleneck. High percentage numbers in sy are usually going to indicate some kind of contention, possibly a configuration issue like having too many worker threads configured for the number of CPUs you have. A high percentage number in id usually indicates that the system either isn't doing much, or can't do much because it's waiting on something like disk or an external database.
So if the 'b' and/or 'id' columns in your vmstat output have high numbers, we may want to consider the possibility of there being an I/O bottleneck. Here are a couple introductory articles on evaluating Linux IO for bottlenecks that might help you determine if this is the case:
https://bartsjerps.wordpress.com/2011/03/04/io-bottleneck-linux/
http://www.linux-mag.com/id/2001/
These articles should get you pointed in the right direction to help you decide if your disks aren't fast enough.
One thing to note, if you're seeing what appears to be a CPU bottleneck (high r values, high us values), make sure that situation makes sense for the number of users you have. The CPU bottleneck may be caused a virtualization issue, or some OS issue causing the CPU to perform poorly, not just by the CPU hardware itself being insufficient.
Topology
One thing mentioned in the gitlab requirements I linked to above is that it is not recommended to run GitLab runner on the same box as GitLab itself. This is something I would say is true for any CI software working with GitLab. If you're running GitLab Runner or Jenkins on the same box as GitLab itself, you should consider moving those to their own hardware.
If you have thousands of users, you may wish to get in contact with GitLab themselves and have consulting on how to get an enterprise-grade cluster stood up and what that looks like. There are people who are experts in the specific hardware configurations that make sense for a very large GitLab installation, and I am not one of them. However, if you don't have a large number of users, the hardware you have is probably not the issue.
Software
If you're running things like vmstat and iostat and you're not finding any specific hardware bottleneck, there may be a configuration issue. Make sure you have a good number of Unicorn Workers configured, so that the box can properly utilize your hardware.
External bottlenecks
Make sure things like network speed on the server are sufficient for its needs. Make sure users trying to reach the server aren't being bottlenecked by a misconfigured network. If you're using OmniAuth, make sure the provider is performing correctly. For example, if you're using some external authentication, and that isn't scaling/performing well, you'll get bad performance in GitLab as well. These are especially important to look at if you're not seeing much hardware utilization using the methods above.
Two aspects which can help accelerate GitLab are, in the latest April 2020 12.10 version:
the application server which switches back Puma
the caching of Git info/refs
The last point is:
When fetching changes from a Git repository, the server advertises a list of all the branches and tags in the repository, known as refs.
In some instances, we have observed up to 75% of all requests to the GitLab web server are requests for the refs.
In the best case, when all the refs are packed, this is an inexpensive operation.
However, when there are unpacked refs, Git must iterate over the unpacked refs. This causes additional disk I/O, which is slow when using high latency storage like NFS.
In GitLab 12.10, info/refs are cached to improve the performance of ref advertisement and decrease the pressure on Gitaly in situations where refs are fetched very frequently.
In testing this feature on GitLab.com, we observed read operations outnumber write operations 10 to 1, and saw median latency decrease by 70%.
For GitLab instances using NFS for Git storage, we expect even greater improvements.
See Documentation and Issue.
My use case details are like below.
Sonarqube Version: 4.5.2
Ram: 16 GB
Code base size: in GBs
OS: Windows
Project Languages: Java, JavaScript
Project Type: Multi Module
Analysis takes half of the day like 12-13 hours. Need help in minimizing the same.
There could be a couple things address here
network latency - for the version you're running, you want to make sure the machine that performs the analysis is a close as possible on the network to your database
database contention - in 4.5.2, the scanner talks directly to the database. If analyses of other projects are happening concurrently, they could be interfering with each other. You can remove this problem by upgrading to the new L.T.S. version, 5.6, which fully cuts the ties from the scanner to the database. In 5.6 analysis reports are generated by the SonarQube scanner, and submitted to the server, where they're queued for processing and handled serially
Your project is just too darn big to analyze successfully in a reasonable amount of time. This may or may not be the case, but you should give it honest consideration, and potentially analyze components independently. Doing so would have the knock-on benefit of not reanalyzing the whole thing when one file in one module has changed. If you need to re-aggregate the results of the individual component analyses, you can do so with the Governance plugin($).
I get that Postgres scales automatically to multicore with multiple connections, but what about when I'm running a massive query on a SINGLE connection? So frustrating that the CPU usage maxes out at 25% on my 4-core system.
I'm in process of switching from SQL Server and this is the only thing so far that really bugs me. SQL Server will use up to 100% of my CPU for a single connection/query.
I'm running 9.2 on Windows 7 Enterprise 64-bit with Xeon processor if it matters.
If there is not way to get around this, could someone address why this isn't seen as an issue? Is it because Postgres is focused on multi-user scenarios?
PostgreSQL does not currently support executing a single query across multiple CPU cores (minus background things like background writing and wal writing if you're doing a write query, but that doesn't really count). It's work that's in progress, but it's a long-term project, and is not in any current version of PostgreSQL.
This is the same on all platforms and architectures.
It is definitely an issue, but since PostgreSQL is, as you say, focused on multi user scenarios, it's not bubbled to the top of the priority queue until recently. But there are definitely people realizing it's an issue, and working on solving it for future versions, it's just not done yet.
There is a Foreign Data Wrapper that aims to add parallelism via the GPU called pg_strom. I've never used it, and it looks quite specialized, but maybe you (or someone here) has a use-case for it.
Article describing pg_strom
http://gpuscience.com/software/postgresql-gpu-pgstrom/
The code:
https://github.com/kaigai/pg_strom
It's not that it isn't seen as a problem. It's that it requires fundamental architectural changes. The use case for it is pretty specialised. It would only help on data warehouse type environments where you're executing long queries one at a time -- AND the queries are CPU bound, not disk i/o bound as they would usually be.
Is anybody know a good testing tool that can produce a graph containing the CPU cycle and RAM usage?
What I will do for ex. is I will run an application and while the application is running the testing tool will record CPU cycle and RAM Usage and it will make a graph as an output.
Basically what I'm trying to test is how much heavy load an application put on RAM and CPU.
Thanks in advance.
In case this is Windows the easiest way is probably Performance Monitor (perfmon.exe).
You can configure the counters you are interested in (Such as Processor Time/Commited Bytes/et) and create a Data Collector Set that measures these counters at the desired interval. There are even templates for basic System Performance Report or you can add counters for the particular process you are interested in.
You can schedule the time where you want to execute the sampling and you will be able to see the result using PerfMon or export to a file for further processing.
Video tutorial for the basics: http://www.youtube.com/watch?v=591kfPROYbs
Good Sample where it shows how to monitor SQL:
http://www.brentozar.com/archive/2006/12/dba-101-using-perfmon-for-sql-performance-tuning/
Loadrunner is the best I can think of ; but its very expensive too ! Depending on what you are trying to do, there might be cheaper alternatives.
Any tool which can either hook to the standard Windows or 'NIX system utilities can do this. This has been a defacto feature set on just about every commercial tool for the past 15 years (HP, IBM, Microfocus, etc). Some of the web only commercial tools (but not all) and the hosted services offer this as wekll. For the hosted services you will generally need to punch a hole through your firewall for them to get access to the hosts for monitoring purposes.
On the open source fron this is a totally mixed bag. Some have it, some don't. Some support one platform, but not others (i.e. support Windows, but not 'NIX or vice-versa).
What tools are you using? It is unfortunately common for people to have performance tools in use and not be aware of their existing toolset's monitoring capabilities.
All of the major commercial performance testing tools have this capability, as well as a fair number of the open source ones. The ability to integrate monitor data with response time data is key to the identification of bottlenecks in the system.
If you have a commercial tool and your staff is telling you that it cannot be done then what they are really telling you is that they don't know how to do this with the tool that you have.
It can be done using jmeter, once you install the agent in the target machine you just need to add the perfmon monitor to your test plan.
It will produce 2 result files, the pefmon file and the requests log.
You could also build a plot that compares the resource compsumtion to the load, and througput. The throughput stops increasing when some resource capacity is exceeded. As you can see in the image CPU time increases as the load increases.
JMeter perfmon plugin: http://jmeter-plugins.org/wiki/PerfMon/
I know this is an old thread but I was looking for the same thing today and as I did not found something that was simple to use and produced graphs I made this helper program for apachebench:
https://github.com/juanluisbaptiste/apachebench-graphs
It will run apachebench and plot the results and percentile files using gnuplot.
I hope it helps someone.