I need solution to speed up maven build process.
The project is based on Adobe cq5 otherwise AEM and i have more than 10 modules in my project where the build happens in linear way.
Currently Build process takes more than 10 min to compile.
is any other specific tool available or other way to speed up the process?
Thanks
I have one suggestion,If you have 10 modules than make separate profile for each module,and build only that part in which you are changing and modifying,no need to deploy 10 modules each time unnecessarily.It will not speed up maven project build but can help you to save your time.It is a workaround but will be helpful .
Try mvn -T 4 clean install #Builds with 4 threads
Its a multi-threaded mode to run Maven and is faster. Apache documentation here.
To add to the other answers:
1) Give more memory resources to the server where the AEM instance is installed, content creation involves a lot of disk access so using SSD is a must.
2) Having a clean AEM instance helps to speed up the process. As you may know the AEM repository grows because of revisions so each time you deploy the repository size grows and it becomes slower. if it's a production environment use maintenance tools like revision clean up and compaction.
revision clean up
how to maintain repository
As per my knowledge there are no such mechanism to speed up.
better to make a build of particular module, it will deploy faster rather waiting for all 10 module to be happened.
Thanks
You can try using the suggested answers to build modules in parallel. It should speed up the build in theory.
But really there is no magic answer. You have to find the bottleneck in your build, it could be the number of dependencies, it could be a specific slow plugin, it could be hardware related, and it could be something else.
Try this: https://github.com/lptr/maven-time-tracker
It may help you find the bottleneck.
I would like to answer this question, knowing it was posted a really long time back.
Currently I am using AEM 6.3 and to recompile and deploy CORE module changes there is a simple maven trick -
This command tells us to -
run 1 thread/core
compile just the core module of the list of projects
and of course, zip package and send it to running AEM instance.
In my observation, this reduced the turnaround time by a huge margin.
Related
I have a Java microservices project with 37 different modules (and growing). Whenever my versions change, Intellij reimports the Gradle build info and rebuilds all of them, including downloading the dependencies. This takes almost TWO hours! What can be done to speed this up?
I gave the compiler 3g of RAM hoping that would help, but the difference is minimal. Are there any other settings I can tweak?
Just wanted to add that I'm on a 64-bit, 6 core box, so I should certainly be able to build faster than this.
I may be approaching the new Cache pipeline task the wrong way, but I'm trying to improve the Build task. Currently, the build task takes on average 20 minutes to complete. I've been reading about the new Cache task, but it seems that would help with caching items like Nuget and npm libraries which for us is in a different task and only takes 1 minute.
Is there a cache setup that can help with the build itself?
Caching is especially useful in scenarios where the same dependencies are downloaded over and over at the start of each run. This is often a time consuming process involving hundreds or thousands of network calls.
Caching can be effective at improving build time provided the time to restore and save the cache is less than the time to produce the output again from scratch. Because of this, caching may not be effective in all scenarios and may actually have a negative impact on build time.
Please refer to this document for details.
Here are two other ways to reduce build time:
You can try to reduce build time in CI pipelines through parallel builds. Here are the reference: blog1,blog2.
In order to obtain faster performance, you can use private agent to run the build pipeline, as private agents cache everything between builds. If you do need to clean the repo (for example to avoid problems caused by residual files from a previous build), you can select Clean option in build definition.
.NET/NuGet:
If you use PackageReferences to manage NuGet dependencies directly within your project file and have packages.lock.json file(s), you can enable caching by setting the NUGET_PACKAGES environment variable to a path under $(Pipeline.Workspace) and caching this directory.
I am suffering the slow download of maven.
I use the default configuration for Spring Boot starter project.
Group: com.example
Artifact: Demo
STS takes too much time to sync the content. is there a way to speed this up? for example, download the whole thing separately that the project needs, and then import them locally.
If you look carefully at the screenshot you'll notice that it says that Maven download operation is blocked by user operation.
The comments saying the build is slower under IDE (particularly Eclipse) are wrong in the sense that they are sort of a mental shortcuts. They are based on observation (it may indeed take longer time to achieve the end result) but that does not mean the build/download itself is slower. The thing is that Eclipse performs way more operations than just the build alone and sometimes those end up waiting on one another (as your screenshot clearly indicates).
With that in mind, if you run your build on the command line it may complete a lot faster as most likely it will not compete for resources with other tasks. But keep in mind that will keep Eclipse out of sync with what is actually on the file system. Eventually Eclipse will figure that out and try to sync. Sometimes it may not and you would have to do it manually. In both cases, depending on the size and amount of the projects and the number and complexity of changes made, it may take significant time to sync.
To summarize, it's not "slow download of maven" what you experience but multiple tasks competing for resources and waiting on each other. There is no point to pre-download all dependencies as this is not a recurring operation. Maven ONLY downloads missing dependencies. Once they are in local repo it will not try to download them again (unless you force it to).
I've tried building it:
parallel (helps a bit)
daemon=true
preDexLibraries true/false ( no big difference )
incremental true
offline
It takes >40s EVERY time. It doesn't matter if I change anything or not. Building it after "gradle clean" takes >50s.
I hope that I'm doing something wrong. Eclipse/ant could build consecutive builds WITH CHANGES in <10s.
Any help how to bring this <10s appreciated.
Yes, we have a lot of modules, some jar libraries, some maven dependancies. Still don't get why would it take >40s for two consecutive builds with no changes.
Gradle used: 2.2.1
Android studio: not really relevant, usually building it only with Gradle.
EDIT: adding some profiling logs.
:app:dexProjDebug 22.541s
:app:shrinkProjDebugMultiDexComponents 4.279s
:app:compileProjDebugJava 3.478s
:app:packageProjDebug 2.591s
:app:processProjDebugResources 2.590s
:app:packageAllProjDebugClassesForMultiDex 2.536s
:app:createProjDebugMainDexClassList 2.126s
You need to properly define inputs and outputs even for library projects with source code. (Gradle Documentation). Do you get UP-TO-DATE message when you rebuild second time onwards?
If you have no changes to the other modules, you can build the app itself using gradlew :app:build. This will eliminate the time Gradle spends on building the other modules. You can build any module separately like this as well, using gradlew :moduledirectoryname:build. If you need to build one module and the app, you can execute both tasks at once to avoid going through Gradle's configuration stage twice: gradlew :app:build :moduledirectoryname:build
If possible, update to Gradle 2.4. It's significantly faster (claiming 20-40%).
One of the most significant build time reducers is the preDex task. Check whether preDex is running in your build. It increases build time the first time you run it, but dramatically reduces build times in subsequent builds, as most of your SDKs and libraries will already be dexed. Note however that it won't run under some conditions, e.g. if you use multidex.
Btw Google are well aware of build times with Gradle and Android Studio and are going to introduce significant improvements in upcoming releases, including Jack and Jill which will allow compiling from source code directly to dex. You can already try to experiment with them - see here: http://tools.android.com/tech-docs/jackandjill.
Also worth watching this video from recent Google I/O: https://youtu.be/f7ihSQ44WO0?t=327
Download Genymotion emulator. It is very fast and good to run android apps. It will save you a lot of time.
Here is the link - GenyMotionDownload
Setting up an integration server, I’m in doubt about the best approach regarding using multiple tasks to complete the build. Is the best way to set all in just one big-job or make small dependent ones?
You definitely want to break up the tasks. Here is a nice example of CruiseControl.NET configuration that has different targets (tasks) for each step. It also uses a common.build file which can be shared among projects with little customization.
http://code.google.com/p/dot-net-reference-app/source/browse/#svn/trunk
I use TeamCity with an nant build script. TeamCity makes it easy to setup the CI server part, and nant build script makes it easy to do a number of tasks as far as report generation is concerned.
Here is an article I wrote about using CI with CruiseControl.NET, it has a nant build script in the comments that can be re-used across projects:
Continuous Integration with CruiseControl
The approach I favour is the following setup (Actually assuming you are in a .NET project):
CruiseControl.NET.
NANT tasks for each individual step. Nant.Contrib for alternative CC templates.
NUnit to run unit tests.
NCover to perform code coverage.
FXCop for static analysis reports.
Subversion for source control.
CCTray or similar on all dev boxes to get notification of builds and failures etc.
On many projects you find that there are different levels of tests and activities which take place when someone does a checkin. Sometimes these can increase in time to the point where it can be a long time after a build before a dev can see if they have broken the build with a checkin.
What I do in these cases is create three builds (or maybe two):
A CI build is triggered by checkin and does a clean SVN Get, Build and runs lightweight tests. Ideally you can keep this down to minutes or less.
A more comprehensive build which could be hourly (if changes) which does the same as the CI but runs more comprehensive and time consuming tests.
An overnight build which does everything and also runs code coverage and static analysis of the assemblies and runs any deployment steps to build daily MSI packages etc.
The key thing about any CI system is that it needs to be organic and constantly being tweaked. There are some great extensions to CruiseControl.NET which log and chart build timings etc for the steps and let you do historical analysis and so allow you to continously tweak the builds to keep them snappy. It's something that managers find hard to accept that a build box will probably keep you busy for a fifth of your working time just to stop it grinding to a halt.
We use buildbot, with the build broken down into discrete steps. There is a balance to be found between having build steps be broken down with enough granularity and being a complete unit.
For example at my current position, we build the sub-pieces for each of our platforms (Mac, Linux, Windows) on their respective platforms. We then have a single step (with a few sub steps) that compiles them into the final version that will end up in the final distributions.
If something goes wrong in any of those steps it is pretty easy to diagnose.
My advice is to write the steps out on a whiteboard in as vague terms as you can and then base your steps on that. In my case that would be:
Build Plugin Pieces
Compile for Mac
Compile for PC
Compile for Linux
Make final Plugins
Run Plugin tests
Build intermediate IDE (We have to bootstrap building)
Build final IDE
Run IDE tests
I would definitely break down the jobs. Chances are you're likely to make changes in the builds, and it'll be easier to track down issues if you have smaller tasks instead of searching through one monolithic build.
You should be able to create one big job from the smaller pieces, anyways.
G'day,
As you're talking about integration testing my big (obvious) tip would be to make the test server built and configured as close as possible to the deployment environment as possible.
</thebloodyobvious> (-:
cheers,
Rob
Break your tasks up into discrete goal/operations, then use a higher-level script to tie them all together appropriately.
This makes your build process easier to understand for other people (you're documenting as you go so anyone on your team can pick it up, right?), as well as increasing the potential for re-use. It's likely you won't reuse the high-level scripts (although this could be possible if you have similar projects), but you can definitely reuse (even if it's copy/paste) the discrete operations rather easily.
Consider the example of getting the latest source from your repository. You'll want to group the tasks/operations for retrieving the code with some logging statements and reference the appropriate account information. This is the sort of thing that's very easy to reuse from one project to the next.
For my team's environment, we use NAnt since it provides a common scripting environment between dev machines (where we write/debug the scripts) and the CI server (since we just execute the same scripts in a clean environment). We use Jenkins to manage our builds, but at their core each project is just calling into the same NAnt scripts and then we manipulate the results (ie, archive the build output, flag failing tests etc).