In our team we have a Sitecore solution that we run locally. Some developers where noticing a high start up time, while other didn't.
After investigating the differences between the local config files, I've noticed that 1 particular setting was causing this high start up time.
The settings was located in the sites.config transform. When adding a transform to enablePreview to put the enablePreview to False, the start up time increased with more than 100%.
When removing this, the start up time was much better.
This seems pretty strange to me, because disabling the preview, should make your Sitecore solution faster right?
Anyone a good explanation for this?
To speed up the start up time you could, among other things, disable SPEAK precompilation. For an extensive list regarding developer productivity, read this blog by John West.
I know this is a couple months old, but try using the Sitecore debugger by appending ?sc_debug=1&sc_prof=1&sc_trace=1&sc_ri=1 to the end of your URL.
That will allow you to see anything that takes an exceptional amount of time to load.
Another thing that could increase startup time is to disable/decrease caching and disable indexing in your dev enviornment.
While in production it speeds up the performance of the site, it also slows down startup time. You can set the tweaks in Debug web.config transformations.
Another thing that could increase start-up time is to disable caching and disable indexing for Master database because no need to indexing for master database in your dev environment.
Related
Frequently last few days onwards our GitLab(CE) running slowly. We have a hook for the CI with Jenkins. We had installed the GitLab by OmniAuth. I don't have any more ideas regarding this because we didn`t do anything new in our instances.
We are the newbie to GitLab environment. We are working in the GitLab since December 2016 and also we never faced this kind of issue before. I hope that I will fix this problem with you people. Kindly help me to fix the issue.
Follow the below image for our Gitlab details.
How could I overcome from this issue?
These are just some as-is suggestions offered without warranty, but they may help guide you to solving the problem.
Occam's Razor
You mentioned that these issues appear to have just started most recently. This means that the VERY FIRST place to look is what may have changed around the time that these issues occurred. If you have change control for your infrastructure, start there. Make absolutely sure nobody has changed anything around the time these issues started happening. Check your logs for any warnings that may have started showing up. If your OS has a security log or logs configuration changes, check those. If you don't have good visibility/audit-ability into your environment, this may be hard, but if you can identify something that changed around the same time as these issues started occurring, that is most often going to be your problem.
Specificity
It may be helpful for you to describe what you mean by it getting slow. Is it a specific operation that is slow? Or is it all activity? If it's something specific, like triggering a Jenkins job, then you can start to isolate your search there.
It can also help to run top on your server to get a picture of what might be causing the issues. There might be a specific process running on the machine that is dominating everything else and eating all of the resources.
Hardware
First thing I would check is to make sure your hardware configuration matches the 'Hardware requirements' guidelines on gitlab's website:
https://docs.gitlab.com/ce/install/requirements.html#requirements
Based on what you've posted, the CPU and memory on your system seem adequate for several thousand users, so I'm going to assume this isn't a problem, but in case you do have thousands of users, I will add some brief information on this. Your disk configuration (other than size) is not presented in the information above, so we don't know if that is sufficient or not.
I would recommend running vmstat on the server (since it's GitLab, I am assuming this is running on Linux, since they do not recommend Windows installations) to get some basic information about what is going on. The vmstat command will give you several columns of information. To the very left there should be a column 'r'. This is the 'run queue', or the number of processes that are waiting to be run on a CPU. If the value in that column is large compared to the number of cores the system has, you probably have a CPU bottleneck. The next column, 'b', is processes that are blocked. If this is large, you probably don't have a CPU bottleneck. To the right, there are CPU columns: us, sy, id, or something along these lines. These columns are a breakdown of where the CPU is spending its time, either in the application code (us), in the OS code (sy), or waiting (id). High percentage numbers in us generally indicate that you either are running healthily or have a CPU bottleneck. High percentage numbers in sy are usually going to indicate some kind of contention, possibly a configuration issue like having too many worker threads configured for the number of CPUs you have. A high percentage number in id usually indicates that the system either isn't doing much, or can't do much because it's waiting on something like disk or an external database.
So if the 'b' and/or 'id' columns in your vmstat output have high numbers, we may want to consider the possibility of there being an I/O bottleneck. Here are a couple introductory articles on evaluating Linux IO for bottlenecks that might help you determine if this is the case:
https://bartsjerps.wordpress.com/2011/03/04/io-bottleneck-linux/
http://www.linux-mag.com/id/2001/
These articles should get you pointed in the right direction to help you decide if your disks aren't fast enough.
One thing to note, if you're seeing what appears to be a CPU bottleneck (high r values, high us values), make sure that situation makes sense for the number of users you have. The CPU bottleneck may be caused a virtualization issue, or some OS issue causing the CPU to perform poorly, not just by the CPU hardware itself being insufficient.
Topology
One thing mentioned in the gitlab requirements I linked to above is that it is not recommended to run GitLab runner on the same box as GitLab itself. This is something I would say is true for any CI software working with GitLab. If you're running GitLab Runner or Jenkins on the same box as GitLab itself, you should consider moving those to their own hardware.
If you have thousands of users, you may wish to get in contact with GitLab themselves and have consulting on how to get an enterprise-grade cluster stood up and what that looks like. There are people who are experts in the specific hardware configurations that make sense for a very large GitLab installation, and I am not one of them. However, if you don't have a large number of users, the hardware you have is probably not the issue.
Software
If you're running things like vmstat and iostat and you're not finding any specific hardware bottleneck, there may be a configuration issue. Make sure you have a good number of Unicorn Workers configured, so that the box can properly utilize your hardware.
External bottlenecks
Make sure things like network speed on the server are sufficient for its needs. Make sure users trying to reach the server aren't being bottlenecked by a misconfigured network. If you're using OmniAuth, make sure the provider is performing correctly. For example, if you're using some external authentication, and that isn't scaling/performing well, you'll get bad performance in GitLab as well. These are especially important to look at if you're not seeing much hardware utilization using the methods above.
Two aspects which can help accelerate GitLab are, in the latest April 2020 12.10 version:
the application server which switches back Puma
the caching of Git info/refs
The last point is:
When fetching changes from a Git repository, the server advertises a list of all the branches and tags in the repository, known as refs.
In some instances, we have observed up to 75% of all requests to the GitLab web server are requests for the refs.
In the best case, when all the refs are packed, this is an inexpensive operation.
However, when there are unpacked refs, Git must iterate over the unpacked refs. This causes additional disk I/O, which is slow when using high latency storage like NFS.
In GitLab 12.10, info/refs are cached to improve the performance of ref advertisement and decrease the pressure on Gitaly in situations where refs are fetched very frequently.
In testing this feature on GitLab.com, we observed read operations outnumber write operations 10 to 1, and saw median latency decrease by 70%.
For GitLab instances using NFS for Git storage, we expect even greater improvements.
See Documentation and Issue.
Our Sitecore 7.2 website runs quite fast, the problem starts when we want to change a lot of items on the backend. Even just opening an item takes 5-6 seconds, and saving the same.
I have events handler for when an item is created or renamed, but no more - I can't fathom why clicking on an item and seeing its details takes so much time. Does the debug mode works for the backend too? How can I start figuring out where the bottleneck is?
To troubleshoot performance issues in Sitecore (including the content editor) your best bet is to start with the CMS Performance Tuning Guide.
There's a companion to it, the CMS Diagnostics Guide.
Depending on what you find, you may need to read up on indexing, caching, browser configuration - again, lots of the information is on SDN.
For the content editor, for instance, there are some application settings that can make a huge difference to loading items - e.g. Prefetch collapsed sections, show fields from standard templates, which warnings you show in the gutter.
The Sitecore Log Analyzer will almost certainly be very useful to you, if you add some performance counters to the log (though these can themselves impact performance).
You can also monitor caches at the admin/cache.aspx - if your cache deltas go haywire or cache sizes reach maximums then you'll have a performance hit
But I'd start by simply monitoring your server resources while you perform one of your troublesome item updates - that should at least let you know if your bottleneck is memory, CPU, SQL connectivity etc.
As time goes, not only Sitecore framework, but any other technology, hardware evolves over time.
The end advice/solution that was actual a while ago might not be applicable today.
It is far more important to be capable of finding reasons in exact your solution why certain operation is slow.
No random changes, every change has to be based on facts. Wall clock response times are to be investigated by performance profiles:
Introduction to wall clock investigation
Wall clock investigations in ASP.NET
Collecting PerfView profiles in Sitecore
Analyzing Sitecore PerfView profile
Here is a blog dedicated to Sitecore performance investigations with real life case studies.
I have drupal based website (Drupal version is 6.19), it is very heavy content website (about 400K articles in it).
By following Rule one of using Drupal, I didnt make any change on the core. but i have a lot of enabled modules and some of them were customized.
Now, I am suffering of the performace and I need to enhance it. I never used Pressflow before, but I have read some articles saying that pressflow is better than Drupal. is it safe to upgrade from Drupal to Pressflow? and if so, how to do it?
Thanks for your help
Pressflow adds the following features to Drupal.
Support for database replication
Support for Squid and Varnish reverse proxy caching
Optimization for MySQL
Optimization for PHP 5
Pressflow is a 100% api-compliant replacement for your standard Drupal Core. There are no database schema changes. So long as you are running a normal Drupal core and meet the other system requirements (PHP5.x, MySQL 5.x), Pressflow is a "drop in" replacement.
Short answer: probably not. Especially since you state that you "have a lot of enabled modules and some of them were customized."
Longer answer: Pressflow's changes are relatively small, and hardly break the APIs. However, there are some incompatibilities, most in the area of database-access and caching. Especially modules that knowingly or unknowingly don't play by Drupals coding guidelines, will probably break. My suggestion: just try, if a module breaks: fix it (and file a patch).
But the real question is: are you going to benefit from Pressflow? It is not simply "better". It allows database-replication, such as load-balancing or master-slaves. Do you intend to use that?
It introduces better support for caching proxies. Are you planning to run a squid or some other caching proxy?
It has some small changes in, for example, the area of caching, that may (but may not) help you; depending on your current usage.
My suggestion: first see how to improve performance without Pressflow. Then, once you come across an area where Drupal is of little help, but which is "fixed" in Pressflow, consider changing.
Few modules have issues with Pressflow and if they do, someone else probably found them. Try searching if any of you modules is incompatible.
Its actually slowed out websites down. This is due to too many modules setup and no caching of our blocks. I am working through things now trying to setup caching and memcache. The issue I have though is that our editors want to see changes now. So some of this might be training. The other issue I have is that we have the fimage module setup and it does not work with the minimum cache lifetime setting so we do not get that benifit at all. In theory it should speed up your site but just let it be known it might do the opposite.
I'm involved with a project using DotNetNuke version 05.01.04 Community Edition. We are building our new Intranet using it, but performance is terrible.
We have five people adding pages and content to it and every 15-30 seconds they experience a pause of 10 seconds or longer before the system continues and the next screens loads.
The server is Windows 2003, 3.8GHz with 1GB of RAM. I'm told by our server admin that the CPU and memory performance don't appear to be the bottleneck.
We currently have 350 pages in the system, we a plan to add 1000. So we need to resolve this performance problem so that we can enter content and so we can go live.
I just can't see where the bottleneck is. Is there a good why to determine the bottleneck when using DotNetNuke?
Modules installed
Publish:Engage (Not currently in
use)
Page Blaster (Doesn't appear
to providing caching when users
logged in using Integrated
Authentication)
SimpleGallery
XMod
Content Manager
IIS Setup
Application recycling completely disabled (Apart from a 2am recycle)
New findings: 18th March 2010
The main bottleneck was due to version 5.1.4 having a bug which caused 1300 database roundtrips on an average page, due to broken database in-memory caching. We've upgraded to 5.2.4 which has resolved this bottleneck.
Now the next biggest bottleneck is the navigation. We've used both DDR:Menu and DDN:Nav, but both have a major impact on performance.
Is there a navigation interface out there that doesn't drain performance so badly?
I think you need to start investigating this using performance profiling tools. For the DNN application itself I'd grab something like JetBrains DotTrace or Red Gate's ANTS Performance Profiler.
For the database SQL Server Profiler would be the first choice or a tool such as Red Gate's SQL Response.
Without profiling the application these you're going to be pulling at straws.
And as Tim pointed out in his comment, installing Firebug in Firefox with the YSlow add-in to see what resources are taking longest to serve to the browser.
Mitchel Sellers has some good tutorials and checklists to go through with regards to performance in DNN. Start with Explaining High Performance DotNetNuke Configuration and Management (which points to some of his earlier articles).
I have several years of dnn development and maintainance experience, when I have this kind of problem, I start doing things from database clean up. Next thing is, find for missing indexes, and/or rebuild all the indexes periodically (sql job scheduled for that) but major performance gain would be from clean up of table
Another good considerations would be, disabling trace, debug mode to false and turn off features of dnn that you don't use (scheduler is the first one to turn off)
Edit: consider keep alive as well
Hope this helps
Is your database on that server? If so, just throw in some more RAM, or get a faster disk array...
Have you considered creating this lot of pages directly through TSQL? It's not hard to do and may save you a lot of time.
OK we are at the end of our rope here, and I’d really appreciate feedback from the SO community.
Our basic issue is slow performance by our MOSS-based intranet--
Some environment info:
We have a MOSS standard edition for a collaboration based site.
The sitedb is 29 Gb
we have two VMWare based front end
servers. (2x 32bit CPUS each)
less than 1000 users spread over all
timezones
We have one big site
collection with subsites.
General symptoms:
Loading front page and pages that have been ‘warmed-up’ is pretty decent- but pages/sites off the beaten path are very slow to load.
We see spikes, where a page all of a sudden takes 30 seconds to load, vs its more normal 2
Here’s what we have done already:
Scaled way back on crawling
enabled object and blob caching
optimized VMware setup
followed the Microsoft IT whitepaper on MOSS sharepoint best practice (esp list size etc)
I don’t know what else to do here—Split into multiple site collections?
Switch to 64 bit front-end servers?
Would be great to hear from others who have been in similar situations.
You don't say how much memory your front end servers have - given that they are 32bit, I'll assume the maximum per worker process of roughly 2gb + change.
My advice? Switch to 64 bit, add more memory, and check that you are not using just one w3wp worker process per front end. Have a dig into "web gardens," that is to say where you configure multiple w3wp processes per front end. To start with, start with two workerprocesses per front end and see how that works out. Also make sure they are set to recycle, and that the recycling of each pair of worker processes do NOT overlap - having two+ workers means they can take turns to recycle without cutting access.
just my 0.02.
-Oisin
I think your very first task is to determine where the problem actually is -
until you know that you are wasting your time changing things.
Is the database server on a separate server or one of your web servers?
Do you see a CPU/Disk bottleneck on your front end or db servers?
It sounds like your world wide; do you see the same performance problems from networks close to the server - is it a WAN issue?
Thanks for some helpful advice all, one thing I just learned was that our object caching has basically not been doing anything! This is because the way it seems to work is that if you have rights to edit ANYWHERE on the site collection, it per default disables object caching across the portal. Since all users have rights to at least something, this means caching was doing basically nothing!
We discovered this by enabling the cache debugging, which puts a small comment in the html about what cache is being used. After changing the setting "Allow writers to view cached content" in the authenticated cache profile,
We are seeing what this does for editors, but for regular viewers, the anecdotal evidence is that it is having a big impact!
Yes, caching is the best way to reduce load on the system.
Adding RAM to the SQL server is also good. (64 bit is really a must for your SQL server, WFE not so important).
Not sure if you want to recycle the processes though. I have not evidence for this except a conversation with someone saying the recycling the processes looked to have solved one performance issue, but was introducing others.
Did I mention caching?
The SQL server should handle database up to 100Gb, but at that size they will be hard to manage for backups and the like, so splitting your site into relevant site collections is something you may need to plan for now, but this may not be relevant to performance.
Did you take a look at Plan for software boundaries (Office SharePoint Server)?
At first glance, your server fits in their recommended settings.
To improve performance you should take a look at :
64-bit servers
Limiting the number of items displayed in your document lists
(source: microsoft.com)
Definitely check the disk usage. If you have two VMs and they run of the same disk / SAN, make sure it isn't too busy. Overloaded SANs kill performance
I'll throw my hat in the ring and recommend 64 bit as well. Perhaps not as an immediate fix, but going forward, I would put a move to 64 bit as a goal for your entire farm.
I'd also take some issue w/ Nat's comment that it doesn't matter on the Web Front Ends. I won't debate benchmarks or argue memory addressability. It's really simpler than that.
Microsoft has publically stated that 2007 is the last version of SharePoint that will run on 32 bit servers. Going forward 64 bit is going to be a requirement - so like the FRAM oil filter guy says - "pay me now or pay me later"...
we have also performance problems. We upgraded on win 2008 64bit witch make some difference but not as much as expected.
The bih boost gave us changing from NTLM azthentication to Kerberos. This was our major improvement.
hope this helps to someone
I run a very similar setup, however we have over 400GB spread over 3 site collections. There are a few other things you can attempt before going down the 64bit path.
Ensure there is no Disk or Bandwidth issues between the DB server and web front ends. The network speed to the database server is critical.
Check the database server itself, if the disks are on a SAN there may be performance issues if there is contention for the phsyical drive on the san. RAM is also important, the more it can cache to memory the less time waiting for disks to respond.
Schedule crawling jobs to run outside of regular business hours
You've enabled object caching which can be huge help, be sure to ensure compression is enabled too! For lower bandwidth users sharepoint sends a large amount of CSS information to users that compresses down to 1/10th it's size if compression is enabled.
Heres another big one, ensure your companies DNS is working correctly in other locations and look into if this problem is related to external sources. IE we have a sonicwall firewall that is brutal on response times for offices connecting through it for anything.
Micrsoft has a whitepaper on performance counter monitering. It is very thorough and will help you narrow down the problem between CPU/RAM/Network/HD IO.
Hope this helps!