I've tried building it:
parallel (helps a bit)
daemon=true
preDexLibraries true/false ( no big difference )
incremental true
offline
It takes >40s EVERY time. It doesn't matter if I change anything or not. Building it after "gradle clean" takes >50s.
I hope that I'm doing something wrong. Eclipse/ant could build consecutive builds WITH CHANGES in <10s.
Any help how to bring this <10s appreciated.
Yes, we have a lot of modules, some jar libraries, some maven dependancies. Still don't get why would it take >40s for two consecutive builds with no changes.
Gradle used: 2.2.1
Android studio: not really relevant, usually building it only with Gradle.
EDIT: adding some profiling logs.
:app:dexProjDebug 22.541s
:app:shrinkProjDebugMultiDexComponents 4.279s
:app:compileProjDebugJava 3.478s
:app:packageProjDebug 2.591s
:app:processProjDebugResources 2.590s
:app:packageAllProjDebugClassesForMultiDex 2.536s
:app:createProjDebugMainDexClassList 2.126s
You need to properly define inputs and outputs even for library projects with source code. (Gradle Documentation). Do you get UP-TO-DATE message when you rebuild second time onwards?
If you have no changes to the other modules, you can build the app itself using gradlew :app:build. This will eliminate the time Gradle spends on building the other modules. You can build any module separately like this as well, using gradlew :moduledirectoryname:build. If you need to build one module and the app, you can execute both tasks at once to avoid going through Gradle's configuration stage twice: gradlew :app:build :moduledirectoryname:build
If possible, update to Gradle 2.4. It's significantly faster (claiming 20-40%).
One of the most significant build time reducers is the preDex task. Check whether preDex is running in your build. It increases build time the first time you run it, but dramatically reduces build times in subsequent builds, as most of your SDKs and libraries will already be dexed. Note however that it won't run under some conditions, e.g. if you use multidex.
Btw Google are well aware of build times with Gradle and Android Studio and are going to introduce significant improvements in upcoming releases, including Jack and Jill which will allow compiling from source code directly to dex. You can already try to experiment with them - see here: http://tools.android.com/tech-docs/jackandjill.
Also worth watching this video from recent Google I/O: https://youtu.be/f7ihSQ44WO0?t=327
Download Genymotion emulator. It is very fast and good to run android apps. It will save you a lot of time.
Here is the link - GenyMotionDownload
Related
I need solution to speed up maven build process.
The project is based on Adobe cq5 otherwise AEM and i have more than 10 modules in my project where the build happens in linear way.
Currently Build process takes more than 10 min to compile.
is any other specific tool available or other way to speed up the process?
Thanks
I have one suggestion,If you have 10 modules than make separate profile for each module,and build only that part in which you are changing and modifying,no need to deploy 10 modules each time unnecessarily.It will not speed up maven project build but can help you to save your time.It is a workaround but will be helpful .
Try mvn -T 4 clean install #Builds with 4 threads
Its a multi-threaded mode to run Maven and is faster. Apache documentation here.
To add to the other answers:
1) Give more memory resources to the server where the AEM instance is installed, content creation involves a lot of disk access so using SSD is a must.
2) Having a clean AEM instance helps to speed up the process. As you may know the AEM repository grows because of revisions so each time you deploy the repository size grows and it becomes slower. if it's a production environment use maintenance tools like revision clean up and compaction.
revision clean up
how to maintain repository
As per my knowledge there are no such mechanism to speed up.
better to make a build of particular module, it will deploy faster rather waiting for all 10 module to be happened.
Thanks
You can try using the suggested answers to build modules in parallel. It should speed up the build in theory.
But really there is no magic answer. You have to find the bottleneck in your build, it could be the number of dependencies, it could be a specific slow plugin, it could be hardware related, and it could be something else.
Try this: https://github.com/lptr/maven-time-tracker
It may help you find the bottleneck.
I would like to answer this question, knowing it was posted a really long time back.
Currently I am using AEM 6.3 and to recompile and deploy CORE module changes there is a simple maven trick -
This command tells us to -
run 1 thread/core
compile just the core module of the list of projects
and of course, zip package and send it to running AEM instance.
In my observation, this reduced the turnaround time by a huge margin.
We have a huge project with many submodules. A full build takes currently over 30mins.
I wonder how this time distributes over different plugins/goals, e.g. tests, static analysis (findbugs, pmd, checkstyle, etc ...)
Would it be possible to time the build to see where (in both dimensions: modules and goals) most time is spent?
The maven-buildtime-extension is a maven plugin that can be used to see the times of each goal:
https://github.com/timgifford/maven-buildtime-extension
If you run the build in a CI server like TeamCity or Jenkins (formerly Hudson), it will give you timestamps for every step in the build process and you should be able to use these values to determine which goals/projects are taking the most time.
I don't think there is any way built in to maven to do this. In fact, in the related question artbristol posted, there is a link to a Maven feature request for this functionality. Unfortunately, this issue is unresolved and I don't know if it will ever be added.
The other potential solution is to write your own plugin which would provide this build metadata for you.
I don't think there is a way to determine the timing of particular goals. What you can do is run the particular goals separately to see how long they take. So instead of doing a "mvn install" which runs all of your tests, checkstyle, etc.. just do "mvn checkstyle:checkstyle" to see how long that takes for a particular module.
Having everything done every time is nice when its done by an automated server (continuum/jenkins/hudson) but when you are building locally, sometimes its better to be able to just compile. Some of the things you can do are have the static analysis goals ONLY run when you pass in a certain parameter or profile. Another option is to only have them ran when maven.test.skip=false.
If you are using a continuous build, try having the static analysis only done every 4 hours, or daily.
my team is currently using buildbot to automate overnight and continuous-integration builds and regression tests. For builds and unit tests, the builder just invokes a script which syncs the sources from p4 and then runs xcodebuild. The regression tests are also launched by a shell script, and are themselves combinations of shell scripts and AppleScripts. The builds are all performed on one system but then buildbot triggers tests to happen on multiple Macs, with different architecture and OS combinations. The things I like about buildbot are the automatic triggers (so the tests run only if and when the build succeeds), and the reporting including the waterfall view to see the overall status.
However, I see a number of problems too. The "master" process which coordinates the builds is either leaky or just has a huge working set which means that it consumes a couple of gigabytes of memory. Occasionally network problems mean that a slave will be lost; rather than retrying later it will just fail the build. In fact if the slave is supposed to be triggered for a dependent build, it will fail the first build after it has otherwise successfully completed.
So what are the rest of you using to automate your Xcode builds and unit tests? How do those solutions work for you? Anything you would recommend?
I use TeamCity in a windows environment but I believe it will work on Mac and has xcode build agents.
Another similar question here.
Our team use Hudson with a script which converts the Unit Test output from XCode to NUnit format.
I know this is an older question, but for those using TFS to hold their Xcode projects, I created a custom build activity to make automating Xcode builds via TFS easier. The code is hosted on Codeplex here: http://tfsxcodebuild.codeplex.com/.
Hope someone finds it useful!
We worked out what was using all the memory - log files - older buildbot keeps all logs forever and keeps them in memory (possibly until a restart).
Newer buildbot can be configured to keep a fixed amount of history.
Buildbot build log files should be limited in size to avoid the problem.
I use the XcodeBuilder that's part of CruiseControl. Of course it helps that I wrote it. :)
But I did use it on a real multiple person project for an iPhone app (Surf) that's for sale in the app store.
Now that it's three years later I have my own answer to provide to this question. I'm using Jenkins, mainly for the plug-ins that it provides. It has a plug-in for building targets in Xcode projects (or schemes in workspaces). There's a plug-in to run the Clang static analyzer. It interfaces with my bug-tracker system, it'll automatically push builds to Testflight too.
I sometimes have issues with the incremental rebuild on visual C++ (2003 currently ). Some dependencies does not seem correctly checked and some files aren't build when they should. I suppose thoses issues come from the timestamp approach to incremental rebuild.
I don't consider it a huge issue when building debug build on my desk, however for distribuable build this is a issue.
Is it safe to use incremental build for a build server or is a full build a requirement ?
You need a build you distrubute to be recreatable again should users experience problems that need investigating.
I would not rely on an incremental build. Also, I would always delete all source from the build machine, and fetch it from scratch from the source control system before building a release. This way, you know you can repeat the build process again by fetching the same source code.
If you use an incremental build, the build will build differently each time because only a subset of the system will need to be built. I think its just good to eliminate as many possible differences between release builds as possible. So, for this reason incremental builds are out.
It's a good idea to label or somehow mark the versions of each source file in the source control system with the version number of the build. This enables you to keep track of the exact source that went into building the release. With a decent source code control system the labels can be used to track down all the changes that were made to the code between one release and the next. This can be a real help when trying to track down a bug that you know was introduced between the two releases.
Incremental builds can still be useful on a development machine when you're not distributing the build, just for saving time during the code/debug/test/repeat development cycle.
I'd argue that you should avoid the "is it safe" question. It's simply not worth looking into.
The advantage of incremental builds is the cost saving. Small per build, it adds up for all the debug builds a developer typically makes, and then adds up further for all developers on your team.
On the other hand, release builds are rare. The costs of a release build are not in developers waiting on them, they're found much more in the team of testers that need to validate it.
For that reason, I find that incremental builds are without a doubt cost-saving for debug builds, but I refuse to spend any time calculating the small saving I'd get out of incrementally building a release build.
I would always prefer to do a full clean and rebuild for any release for peace of mind if nothing else. Assuming that you're talking about externally released builds and not just builds using the Release configuration, the fact that these builds are relatively infrequent means that any time saving will be minimal in the long run.
Binaries built using with the incremental build feature are going to be bigger and slower, even for release builds, because they necessarily contain scaffolding to enable the incremental builds that isn't necessary for fully optimized builds. I'd recommend you not to use incremental builds for this reason.
Setting up an integration server, I’m in doubt about the best approach regarding using multiple tasks to complete the build. Is the best way to set all in just one big-job or make small dependent ones?
You definitely want to break up the tasks. Here is a nice example of CruiseControl.NET configuration that has different targets (tasks) for each step. It also uses a common.build file which can be shared among projects with little customization.
http://code.google.com/p/dot-net-reference-app/source/browse/#svn/trunk
I use TeamCity with an nant build script. TeamCity makes it easy to setup the CI server part, and nant build script makes it easy to do a number of tasks as far as report generation is concerned.
Here is an article I wrote about using CI with CruiseControl.NET, it has a nant build script in the comments that can be re-used across projects:
Continuous Integration with CruiseControl
The approach I favour is the following setup (Actually assuming you are in a .NET project):
CruiseControl.NET.
NANT tasks for each individual step. Nant.Contrib for alternative CC templates.
NUnit to run unit tests.
NCover to perform code coverage.
FXCop for static analysis reports.
Subversion for source control.
CCTray or similar on all dev boxes to get notification of builds and failures etc.
On many projects you find that there are different levels of tests and activities which take place when someone does a checkin. Sometimes these can increase in time to the point where it can be a long time after a build before a dev can see if they have broken the build with a checkin.
What I do in these cases is create three builds (or maybe two):
A CI build is triggered by checkin and does a clean SVN Get, Build and runs lightweight tests. Ideally you can keep this down to minutes or less.
A more comprehensive build which could be hourly (if changes) which does the same as the CI but runs more comprehensive and time consuming tests.
An overnight build which does everything and also runs code coverage and static analysis of the assemblies and runs any deployment steps to build daily MSI packages etc.
The key thing about any CI system is that it needs to be organic and constantly being tweaked. There are some great extensions to CruiseControl.NET which log and chart build timings etc for the steps and let you do historical analysis and so allow you to continously tweak the builds to keep them snappy. It's something that managers find hard to accept that a build box will probably keep you busy for a fifth of your working time just to stop it grinding to a halt.
We use buildbot, with the build broken down into discrete steps. There is a balance to be found between having build steps be broken down with enough granularity and being a complete unit.
For example at my current position, we build the sub-pieces for each of our platforms (Mac, Linux, Windows) on their respective platforms. We then have a single step (with a few sub steps) that compiles them into the final version that will end up in the final distributions.
If something goes wrong in any of those steps it is pretty easy to diagnose.
My advice is to write the steps out on a whiteboard in as vague terms as you can and then base your steps on that. In my case that would be:
Build Plugin Pieces
Compile for Mac
Compile for PC
Compile for Linux
Make final Plugins
Run Plugin tests
Build intermediate IDE (We have to bootstrap building)
Build final IDE
Run IDE tests
I would definitely break down the jobs. Chances are you're likely to make changes in the builds, and it'll be easier to track down issues if you have smaller tasks instead of searching through one monolithic build.
You should be able to create one big job from the smaller pieces, anyways.
G'day,
As you're talking about integration testing my big (obvious) tip would be to make the test server built and configured as close as possible to the deployment environment as possible.
</thebloodyobvious> (-:
cheers,
Rob
Break your tasks up into discrete goal/operations, then use a higher-level script to tie them all together appropriately.
This makes your build process easier to understand for other people (you're documenting as you go so anyone on your team can pick it up, right?), as well as increasing the potential for re-use. It's likely you won't reuse the high-level scripts (although this could be possible if you have similar projects), but you can definitely reuse (even if it's copy/paste) the discrete operations rather easily.
Consider the example of getting the latest source from your repository. You'll want to group the tasks/operations for retrieving the code with some logging statements and reference the appropriate account information. This is the sort of thing that's very easy to reuse from one project to the next.
For my team's environment, we use NAnt since it provides a common scripting environment between dev machines (where we write/debug the scripts) and the CI server (since we just execute the same scripts in a clean environment). We use Jenkins to manage our builds, but at their core each project is just calling into the same NAnt scripts and then we manipulate the results (ie, archive the build output, flag failing tests etc).