How to speed up slow fatJAR task? - performance

Question

The following part is a part of my build.gradle file:
project(":App") {
dependencies {
compile(project(':Shared'))
compile(project(':Subproject1'))
compile(project(':Subproject2'))
compile(project(':Subproject3'))
compile(project(':Subproject4'))
compile(project(':Subproject5'))
}
jar {
manifest {
attributes(
'Main-Class': 'com.my.App.Main',
)
}
}
// To run, use: $ gradle :App:fatJar
task fatJar(type: Jar) {
println "fatJarBuild(): 1"
manifest.from jar.manifest
println "fatJarBuild(): 2"
classifier = 'all'
println "fatJarBuild(): 3"
from {
configurations.runtime.collect {
println "fatJarBuild(): collect" + it.absolutePath
it.isDirectory() ? it : zipTree(it)
}
} {
exclude "META-INF/*.SF"
exclude "META-INF/*.DSA"
exclude "META-INF/*.RSA"
}
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
println "fatJarBuild(): 4"
with jar
}
}
Now, whenever I slightly modify a file in my codebase, it takes substantial time to finish fatJar task:
[... End of "./gradlew :App:fatJar --info" follows ... ]
:App:compileJava (Thread[Daemon worker Thread 13,5,main]) completed. Took 0.164 secs.
:App:processResources (Thread[Daemon worker Thread 13,5,main]) started.
:App:processResources
Skipping task ':App:processResources' as it is up-to-date (took 0.002 secs).
:App:processResources UP-TO-DATE
:App:processResources (Thread[Daemon worker Thread 13,5,main]) completed. Took 0.002 secs.
:App:classes (Thread[Daemon worker Thread 13,5,main]) started.
:App:classes
Skipping task ':App:classes' as it has no actions.
:App:classes (Thread[Daemon worker Thread 13,5,main]) completed. Took 0.0 secs.
:App:fatJar (Thread[Daemon worker Thread 13,5,main]) started.
:App:fatJar
Executing task ':App:fatJar' (up-to-date check took 0.003 secs) due to:
Input file /home/work/App/BrainThread.class has changed.
:App:fatJar (Thread[Daemon worker Thread 13,5,main]) completed. Took 5.993 secs.
BUILD SUCCESSFUL
Total time: 7.051 secs
Stopped 0 compiler daemon(s).
Received result Success[value=null] from daemon DaemonInfo{pid=30179, address=[c5e7f6f0-985b-48cc-88b0-ebc8aed7e75b port:33465, addresses:[/0:0:0:0:0:0:0:1%lo, /127.0.0.1]], idle=true, context=DefaultDaemonContext[uid=cc8b9da5-88b5-476a-9cf5-430af98f7f5a,javaHome=/usr/lib/jvm/java-8-openjdk-amd64,daemonRegistryDir=/home/user/.gradle/daemon,pid=30179,idleTimeout=10800000,daemonOpts=-XX:MaxPermSize=256m,-XX:+HeapDumpOnOutOfMemoryError,-Xmx1024m,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]} (build should be done).
You can see that BrainThread.class was modified and as an effect some heavy machinery started.
Can I make my build.gradle more time efficient?

Answer 1

Maybe someone else will come along and tell me I'm wrong, but IMO 5 seconds is not unreasonable time to build a fatjar. The task as defined
Unpacks the jar file of every single dependency,
Consolidates the unpacked files, along with your project class files and then
Repackages everything into a new jar.
Depending on how many dependencies you have, 5 seconds seems fairly ok to me. Unless the underlying Zip task is somehow optimized, I do not see this timing change.
A couple of things you can thing about though:
You could move the fatjar task out of your standard build cycle, which runs for every change. You could only run the fatjar task when you're doing a release. In most cases you should be able to test changes locally without having to build a fatjar as you have all the dependencies available anyway.
For a single file change, you're doing the unpack-repack dance with a whole lot of dependency jar. Maybe you can write an optimized fatjar task that unpacks the old fatjar, replaces the changed file and repacks it, reducing the number of file operations required.
EDIT: I just looked at the gradle shadow jar plugin, and realized that it does exactly what I described #2 above, makes incremental changes in place to the fatjar, which should give you palpable performance improvements.