Loading Jenkins Job Configure page is slow w/o running jobs - performance

Question

Currently when we tried to open Jenkins configure page, it takes round 45 seconds while
for other pages like requesting a view of a job or looking into a console output it's takes less than 3 seconds.
Based on thread dumps analyse result we got a CPU 'Spike' with as description
"Your application might be suffering from high CPU."
Looking at the thread report we don't see any blocked status but one suspect status:
"1 thread is infinitely looping: DestroyJavaVM"
Unfortunately we are not able to identify the cause of this high CPU and probably the related infinitely looping.
So far we already take some steps to improve the performance by restarting Jenkins, it results to 10 seconds faster from 55 seconds to 45 seconds.
We redefine JVM min/max memory parameters as the same and increase the hype to 4 Gb, it doesn't give any affect.
We create a clean test Jenkins instance with same Jenkins version, same installed plugins and job configuration on lower hardware specs with same OS.
The load of the Job configure page at the new test Jenkins instance is less than 3 seconds.
We are not sure if we can exclude the installed plugins as the main issue based on this test.
Any other suggestions are welcome to able track the issues.
ThreadDump Analyse Report
We are using :
- Windows Server 2008 R2 Enterprise sp1
- Jenkins (Master): version 2.89.1
- Java: JRE 1.8.0_112
- Client webbrowser Google Chrome Versie 61.0.3163.100 (Officiële build) (64-bits)
Installed plugins: Ant Plugin - 1.7
Apache HttpComponents Client 4.x API Plugin - 4.5.3-2.0
Artifactory Plugin - 2.13.1
Authentication Tokens API Plugin - 1.3
Autofavorite for Blue Ocean - 1.2.1
Bitbucket Branch Source Plugin - 2.2.7
Bitbucket Pipeline for Blue Ocean - 1.3.4
Blue Ocean - 1.3.4
Blue Ocean Pipeline Editor - 1.3.4
bouncycastle API Plugin - 2.16.2
Branch API Plugin - 2.0.15
build-name-setter - 1.6.7
Common API for Blue Ocean - 1.3.4
Conditional BuildStep - 1.3.6
Config API for Blue Ocean - 1.3.4
Config File Provider Plugin - 2.16.4
Copy Artifact Plugin - 1.39
Credentials Binding Plugin - 1.13
Credentials Plugin - 2.1.16
Customize Build Now Label - 1.1
CVS Plug-in - 2.13
Dashboard for Blue Ocean - 1.3.4
Dashboard View - 2.9.11
Discard Old Build plugin - 1.05
disk-usage plugin - 0.28
Display Upstream Changes - 0.3.2
Display URL API - 2.2.0
Display URL for Blue Ocean - 2.2.0
Docker Commons Plugin - 1.9
Docker Pipeline - 1.14
Durable Task Plugin - 1.17
EnvInject API Plugin - 1.4
Environment Injector Plugin - 2.1.5
Events API for Blue Ocean - 1.3.4
External Monitor Job Type Plugin - 1.7
Favorite - 2.3.1
Folders Plugin - 6.2.1
Git client plugin - 2.6.0
Git Pipeline for Blue Ocean - 1.3.4
Git plugin - 3.6.4
GIT server Plugin - 1.7
GitHub API Plugin - 1.90
GitHub Branch Source Plugin - 2.3.1
GitHub Pipeline for Blue Ocean - 1.3.4
GitHub plugin - 1.28.1
Gradle Plugin - 1.28
HTML Publisher plugin - 1.14
i18n for Blue Ocean - 1.3.4
Icon Shim Plugin - 2.0.3
Ivy Plugin - 1.28
Jackson 2 API Plugin - 2.8.7.0
Javadoc Plugin - 1.4
JavaScript GUI Lib: ACE Editor bundle plugin - 1.1
JavaScript GUI Lib: Handlebars bundle plugin - 1.1.1
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin - 1.2.1
JavaScript GUI Lib: Moment.js bundle plugin - 1.1.1
JIRA Integration for Blue Ocean - 1.3.4
JIRA plugin - 2.5
Job Configuration History Plugin - 2.18
jQuery plugin - 1.12.4-0
JSch dependency plugin - 0.1.54.1
JUnit Plugin - 1.23
JWT for Blue Ocean - 1.3.4
LDAP Plugin - 1.18
Locale plugin - 1.2
Mailer Plugin - 1.20
Managed Scripts - 1.4
MapDB API Plugin - 1.0.9.0
Matrix Authorization Strategy Plugin - 2.2
Matrix Project Plugin - 1.12
Maven Integration plugin - 3.0
Mercurial plugin - 2.2
Metrics Disk Usage Plugin - 3.0.0
Metrics Plugin - 3.1.2.10
Monitoring - 1.70.0
MSBuild Plugin - 1.28
MSTest plugin - 0.23
NAnt Plugin - 1.4.3
OWASP Dependency-Check Plugin - 3.0.2
OWASP Markup Formatter Plugin - 1.5
PAM Authentication plugin - 1.3
Parameterized Trigger plugin - 2.35.2
Personalization for Blue Ocean - 1.3.4
Pipeline - 2.5
Pipeline Graph Analysis Plugin - 1.5
Pipeline implementation for Blue Ocean - 1.3.4
Pipeline SCM API for Blue Ocean - 1.3.4
Pipeline Utility Steps - 1.5.1
Pipeline: API - 2.24
Pipeline: Basic Steps - 2.6
Pipeline: Build Step - 2.5.1
Pipeline: Declarative - 1.2.5
Pipeline: Declarative Agent API - 1.1.1
Pipeline: Declarative Extension Points API - 1.2.5
Pipeline: Groovy - 2.42
Pipeline: Input Step - 2.8
Pipeline: Job - 2.15
Pipeline: Milestone Step - 1.3.1
Pipeline: Model API - 1.2.5
Pipeline: Multibranch - 2.16
Pipeline: Nodes and Processes - 2.17
Pipeline: REST API Plugin - 2.9
Pipeline: SCM Step - 2.6
Pipeline: Shared Groovy Libraries - 2.9
Pipeline: Stage Step - 2.3
Pipeline: Stage Tags Metadata - 1.2.5
Pipeline: Stage View Plugin - 2.9
Pipeline: Step API - 2.14
Pipeline: Supporting APIs - 2.16
Plain Credentials Plugin - 1.4
Plugin Usage - Plugin - 0.3
PostBuildScript Plugin - 2.2.1
PowerShell plugin - 1.3
promoted builds plugin - 2.31
Pub-Sub "light" Bus - 1.12
Quality Gates Plugin - 2.5
Rebuilder - 1.27
Resource Disposer Plugin - 0.8
REST API for Blue Ocean - 1.3.4
REST Implementation for Blue Ocean - 1.3.4
Role-based Authorization Strategy - 2.6.1
Run Condition Plugin - 1.0
SCM API Plugin - 2.2.5
Script Security Plugin - 1.36
Server Sent Events (SSE) Gateway Plugin - 1.15
Simple Theme Plugin - 0.3
SonarQube Scanner for Jenkins - 2.6.1
SSH Credentials Plugin - 1.13
Static Analysis Utilities - 1.93
Structs Plugin - 1.10
Subversion Plug-in - 2.9
Team Foundation Server Plug-in - 5.126.0
Throttle Concurrent Builds Plug-in - 2.0.1
Timestamper - 1.8.8
Token Macro Plugin - 2.3
Variant Plugin - 1.1
Web for Blue Ocean - 1.3.4
Windows Slaves Plugin - 1.3.1
Workspace Cleanup Plugin - 0.34

Answer 1

First, answer these questions for yourself:
Are there other CPU intensive processes running on the same Jenkins server?
Huge CPU loads could also result from high IO, how’s the disk IO?
A JVM when the box is running low on RAM and paging, will likely raise the CPU load hence poor application performance.
So your troubleshooting steps should be:
Do your CPU support multi-threading? If it’s single-threaded and you start more than one thread, this could be the result.
Find the average CPU utilization for each process, is anything eating up CPU cycles (that should not be.)
Overall RAM usage.
Disk IO performance metrics - are you seeing spikes in IO when you do not expect them?
Identify the child processes initiated by Jenkins - find the average resource utilization for each process. Are these overloading any resources (CPU, RAM, disk IO?)
Based on what you find, you should tune your box and your processes accordingly. This could mean adding resources.