What are .kotlin_builtins files and can I omit them from my uberjars? - gradle

I'm working on integrating proguard to my gradle build for an application written in Kotlin. I'm finding that proguard is stripping out the Kotlin standard library (as it should in my simple Hello World program) but it's leaving a bunch of files in my jar that have the file extension .kotlin_builtins. When I configure my gradle task to exclude those files, the program still appears to work fine. What are those files and must they ship with my executable uberjar?
Here's my build.gradle file's contents for reference:
buildscript {
ext.kotlin_version = '1.0.5'
ext.shadow_version = '1.2.4'
repositories {
mavenCentral()
maven {
url "https://plugins.gradle.org/m2/"
}
flatDir dirs: "gradle/proguard"
}
dependencies {
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
classpath "com.github.jengelman.gradle.plugins:shadow:$shadow_version"
classpath ":proguard:"
}
}
apply plugin: 'kotlin'
apply plugin: 'application'
apply plugin: 'com.github.johnrengelman.shadow'
mainClassName = 'namespace.MainKt'
defaultTasks 'run'
repositories {
mavenCentral()
}
dependencies {
compile "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
testCompile "junit:junit:4.12"
testCompile "org.jetbrains.kotlin:kotlin-test-junit:$kotlin_version"
}
shadowJar {
exclude 'kotlin/**/*.kotlin_builtins'
exclude '.keep'
}
task minify(type: proguard.gradle.ProGuardTask, dependsOn: 'shadowJar') {
libraryjars "${System.getProperty('java.home')}/lib/rt.jar"
injars 'build/libs/artful-all.jar'
outjars 'build/libs/artful-all.out.jar'
printmapping 'build/libs/out.map'
keepclasseswithmembers 'public class * { \
public static void main(java.lang.String[]); \
}'
assumenosideeffects 'class kotlin.jvm.internal.Intrinsics { \
static void checkParameterIsNotNull(java.lang.Object, java.lang.String); \
}'
}

These files contain data for declarations of standard ("built-in") Kotlin classes which are not compiled to .class files, but rather are mapped to the existing types on the platform (in this case, JVM). For example, kotlin/kotlin.kotlin_builtins contains the information for non-physical classes in package kotlin: Int, String, Enum, Annotation, Collection, etc.
There are two main scenarios when these files are used:
The compiler looks them up from kotlin-stdlib on the classpath to determine which built-in declarations are available.
The reflection library (kotlin-reflect) loads these files as resources to provide reflection capabilities for built-in declarations. For example, String::class.members returns all members of the class kotlin.String exactly in the same way as the Kotlin compiler sees those members (despite the fact that there's no kotlin/String.class file and it's erased to java.lang.String in bytecode).
The first point is clearly not applicable in your case. And if you don't use reflection on built-in classes, I think it's safe to exclude .kotlin_builtins files completely from the resulting jar.

You can optimize/omit these from yours JARs/APKs:
packagingOptions {
exclude "/META-INF/*.kotlin_module"
exclude "**/kotlin/**"
}
Even better:
packagingOptions {
exclude "/META-INF/*.kotlin_module"
exclude "**/kotlin/**"
exclude "**/*.txt"
exclude "**/*.xml"
exclude "**/*.properties"
}
Source: https://github.com/jaredsburrows/android-gif-example/blob/master/build.gradle.kts#L127

Related

new gradle format confusion and migrating causes Could not find method testCompile()

I have a build.gradle with
plugins {
id {some plugin for all projects}
id "com.diffplug.spotless" version "5.1.1"
}
AND THEN I have an allprojects {} section that defines ONE apply plugin: 'jacoco' and a subprojects {} section that declares apply plugin: 'java' with a few others
Immediately adding spotless messed with stuff and errors out that it cannot find the java plugin so then I modify ALL plugins to be in the plugins section like so
plugins {
id "java"
id "checkstyle"
id "eclipse"
id "idea"
id "jacoco"
id "com.diffplug.spotless" version "5.1.1"
id "com.dorongold.task-tree" version "1.5" //This prints out a task tree (pretty print)
}
This then results in this error
Could not find method testCompile() for arguments [junit:junit:4.11] on object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.
So for some reason the java plugin is lost. I can't figure out the right combination here to migrate everything to this new plugins section format.
How do I do that? I randomly tried putting a plugins section in allprojects and subprojects but that results in this new error
Could not find method plugins() for arguments [build_d8c2jgy4ua1m0vkv9kmvgefmc$_run_closure2$_closure5#62337fda] on root project 'providersvc-all' of type org.gradle.api.Project
How does this new plugins section work? I can't seem to migrate without it breaking everything. I just want java plugin, testCompile, and spotless to play nicely together right now
EDIT(forgot to attach the full trimmed down file that does not work):
plugins {
id "java"
id "com.diffplug.spotless" version "5.1.1"
}
ext {
//dependency versions every project usees so we version in one location all jars(less jar hell this way)
deps = [
'junit': 'junit:junit:4.11'
]
}
allprojects {
repositories {
jcenter()
mavenCentral()
maven {
//webpieces VERSIONED snapshots so you can lock on a snapshot
url "https://dl.bintray.com/deanhiller/maven"
}
//For testing locally
maven {
url uri('/tmp/myRepo/')
}
}
}
subprojects {
dependencies {
testCompile deps['junit']
}
}
thanks,
Dean
You are only applying the plugins to the root project - not the sub-projects. However, if you like to configure plugins through the subprojects configuration, you have to use the apply plugin syntax. But you don't have to use the old buildscript block for configuring the classpath and repositories if you a combination of the two.
Here is an example. I am assuming the root project is not a Java project. I have also removed your comments and inserted mine instead for the sole reason to make them easier to spot.
plugins {
id "com.diffplug.spotless" version "5.1.1" apply false // <-- Set "apply false" here
// This makes it configure which version to use on the classpath for the entire build, but without applying it.
// Notice that the Java plugin is not specified here anymore.
// This is because it is a core plugin so you can't set the version (and I am assuming you don't want it on in the root project).
}
ext {
deps = [
'junit': 'junit:junit:4.11'
]
}
allprojects {
repositories {
jcenter()
mavenCentral() // <-- You can remove this if you want as it is already present as a proxy in jcenter().
maven {
url "https://dl.bintray.com/deanhiller/maven"
}
maven {
url uri('/tmp/myRepo/')
}
}
}
subprojects {
// Here are the two plugins
apply plugin: "java"
apply plugin: "com.diffplug.spotless"
dependencies {
testImplementation deps['junit'] // <-- testCompile renamed to testImplementation as the former has been deprecated for a long time
}
}

In gradle, how to copy a subset of compiled test classes

I tried taking a portion of the compiled test classes of a Gradle project, and put them in a jar file (to create a test case). What happened was, only those classes got compiled out of all test classes. Any idea how to do it right?
Here's my build.gradle:
plugins {
id 'java'
}
group 'org.example'
version '1.0-SNAPSHOT'
repositories {
mavenCentral()
}
dependencies {
testCompile group: 'junit', name: 'junit', version: '4.12'
}
task testSampleJar(type: Zip) {
archiveName "sample.jar"
from compileTestJava {
include "org/example/samples/**"
}
}
test {
dependsOn(testSampleJar)
}
So what seems to be happening is that by just defining the testSampleJar task, the compileTestJava task gets modified to only compile under org.example.samples package. My intent was to use the outputs of compileTestJava and pick something out of them them.
Thanks,
Uri
OK, figured it out, so the two possible solutions are:
//a global filter
task testSampleJar(type: Zip) {
archiveName "sample.jar"
from compileTestJava
include "org/example/samples/**"
}
Or
//Child specification - notice the parentheses around compileTestJava
task testSampleJar(type: Zip) {
archiveName "sample.jar"
from (compileTestJava) {
include "org/example/samples/**"
}
}
I guess the original code was defining compileTestJava instead of using its output, but I don't have a full grasp of Groovy/Gradle DSL to be certain.

Use Gradle to generate Java classes from a Swagger definition into a standalone JAR

I am fairly new to both Gradle and Swagger code generator plugin for it (concretely the one that is linked from Swagger's website, i.e. https://github.com/thebignet/swagger-codegen-gradle-plugin), so I'm not sure whether my problem is with Gradle in general or with that particular plugin.
I've created a simple multi-module Spring Boot application (but the fact that I'm using Spring Boot or even Spring doesn't matter much). It's a console application; i.e. it doesn't start a webserver. In fact, it's actually a REST client consuming someone else's interface.
The application consists of four modules: spc-parent (which is just an envelope for the rest) containing spc-boot, spc-service, and spc-integration-model. Spc-boot contains just the starting point of the application, spc-service now contains a single Spring service, and spc-integration-model is meant to contain classes needed to consume the REST interface. The resulting structure will be much more complicated but I've tried to create a sort of a minimal example.
The problem lies within the spc-integration-model module. It consists of a single source file, petstore.json, and a build.gradle copied from https://github.com/thebignet/swagger-codegen-gradle-plugin (and only slightly modified). There are actually two problems (but they may have the same underlying cause).
When running gradle build (from spc-parent) for the very first time, it fails. Java sources are generated from petstore.json but they don't get compiled, which is why the service in spc-service doesn't see needed classes. However, running gradle build a second time fixes this (generated Java sources get compiled which makes it possible to compile spc-service, too).
The created JAR of spc-integration-model never contains anything besides Manifest.
My goal here is to persuade Gradle to compile the generated classes right away during the first build and also to put them into the JAR.
Now for some concrete Gradle tasks. The most interesting is spc-integration-model's build.gradle:
plugins {
id 'org.detoeuf.swagger-codegen' version '1.7.4'
id 'java'
}
apply plugin: 'org.detoeuf.swagger-codegen'
repositories {
mavenCentral()
jcenter()
}
swagger {
inputSpec = 'http://petstore.swagger.io/v2/swagger.json'
outputDir = file('build/swagger')
lang = 'java'
additionalProperties = [
'apiPackage' : 'ondra.spc.integration.client.api',
'dateLibrary' : 'java8',
'hideGenerationTimestamp': 'true',
'invokerPackage' : 'ondra.spc.integration.client',
'library' : 'resttemplate',
'modelNameSuffix' : 'Dto',
'modelPackage' : 'ondra.spc.integration.client.model'
]
importMappings = [
'Dog': 'io.swagger.petstore.client.model.Dog'
]
}
sourceSets {
swagger {
java {
srcDir file("${project.buildDir.path}/swagger/src/main/java")
}
}
}
classes.dependsOn('swagger')
ext {
spring_boot_version = springBootVersion
jackson_version = jacksonVersion
junit_version = jUnitVersion
swagger_annotations_version = swaggerAnnotationsVersion
swagger_codegen_version = swaggerCodegenVersion
}
dependencies {
swaggerCompile "org.springframework.boot:spring-boot-starter-web:$spring_boot_version"
swaggerCompile "io.swagger:swagger-annotations:$swagger_annotations_version"
compile sourceSets.swagger.output
compile "com.fasterxml.jackson.core:jackson-core:$jackson_version"
compile "com.fasterxml.jackson.core:jackson-annotations:$jackson_version"
compile "com.fasterxml.jackson.core:jackson-databind:$jackson_version"
compile "io.swagger:swagger-codegen:$swagger_codegen_version"
testCompile "junit:junit:$junit_version"
}
(Now that I'm re-reading my question I see that the local version of petstore.json is actually not used and an online version is used instead but let's leave that aside.)
The rest should be quite straightforward. spc-service:
dependencies {
compile "org.springframework:spring-context:$springVersion"
compile "org.springframework:spring-web:$springVersion"
compile project (":spc-integration-model")
}
spc-boot:
dependencies {
compile "org.springframework.boot:spring-boot:$springBootVersion"
compile "org.springframework.boot:spring-boot-autoconfigure:$springBootVersion"
compile "org.springframework:spring-web:$springVersion"
runtime "org.hibernate.validator:hibernate-validator:$hibernateVersion"
runtime "com.fasterxml.jackson.core:jackson-databind:$jacksonVersion"
compile project (":spc-service")
testCompile("org.junit.jupiter:junit-jupiter-api:$jUnitVersion")
testRuntime("org.junit.jupiter:junit-jupiter-engine:$jUnitVersion")
}
test {
useJUnitPlatform()
}
spc-parent:
subprojects {
apply plugin: 'java'
group 'ondra'
version '1.0-SNAPSHOT'
buildscript {
ext {
hibernateVersion = '6.0.9.Final'
jacksonVersion = '2.9.4'
springBootVersion = '2.0.0.RELEASE'
springVersion = '5.0.5.RELEASE'
swaggerAnnotationsVersion = '1.5.16'
swaggerCodegenVersion = '2.2.3'
jUnitVersion = '5.1.1'
}
repositories {
mavenCentral()
}
}
sourceCompatibility = 1.8
targetCompatibility = 1.8
repositories {
mavenCentral()
}
}
And spc-parent's settings.gradle:
rootProject.name = 'spc-parent'
include 'spc-boot'
include 'spc-service'
include 'spc-integration-model'
I've also put the whole application into a single archive: https://drive.google.com/open?id=1cOYIcaxnhik548w0wEGswgD2g4udATdD

Gradle: Distributing Executable, Obfuscated Jar File

I'm trying to use gradle with proguard to obfuscate the code then generate a zip file to distribute. I'd like to use the distribution plugin, but it always includes the jar that is generated by the jar task. Is there some way to force the distribution plugin to omit the original (non-obfuscated) jar and only include the obfuscated jar? I can easily add the obfuscated jar in addition to the original, but I want to distribute the obfuscated jar instead of the original so the generated execution scripts run against the obfuscated version.
Here's my abridged build.gradle file:
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath 'net.sf.proguard:proguard-gradle:5.3.3'
}
}
apply plugin: 'java'
apply plugin: 'application'
task obfuscate(type: proguard.gradle.ProGuardTask) {
configuration 'proguard.txt'
injars "build/libs/${rootProject.name}.jar"
outjars "build/libs/${rootProject.name}-release.jar"
}
jar.finalizedBy(project.tasks.obfuscate)
distributions {
main {
contents {
from(obfuscate) {
into "lib"
}
from(jar) {
exclude "*.jar"
}
}
}
}
I've tried a number of things in the distributions block to try to exclude the original jar, but nothing seems to work.
Any ideas would be greatly appreciated.
This isn't the best solution, but I was able to work around the issue by renaming the jars at the end of the obfuscation step. Now, I name the original jar something like <JAR_NAME>-original.jar and I give the obfuscated jar the original jar's name. I still wish there was a better way to do it, but this seems to work.
Here is the updated, abridged build.gradle file:
import java.nio.file.Paths
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath 'net.sf.proguard:proguard-gradle:5.3.3'
}
}
apply plugin: 'java'
apply plugin: 'application'
def jarNameWithoutExtension = jar.archiveName.with { it.take(it.lastIndexOf(".")) }
def obfuscatedJarName = "${jarNameWithoutExtension}-release.jar"
def jarFileLocation = jar.archivePath.parent
def obfuscatedFilePath = Paths.get(jarFileLocation, obfuscatedJarName)
task obfuscate(type: proguard.gradle.ProGuardTask) {
configuration 'proguard.txt'
injars jar.archivePath
outjars obfuscatedFilePath.toString()
// Rename the original and obfuscated jars. We want the obfuscated jar to
// have the original jar's name so it will get included in the distributable
// package (generated by installDist / distZip / distTar / assembleDist).
doLast {
jar.archivePath.renameTo(Paths.get(jarFileLocation, "$jarNameWithoutExtension-original.jar").toFile())
obfuscatedFilePath.toFile().renameTo(jar.archivePath)
}
}
jar.finalizedBy(project.tasks.obfuscate)

Building a fully executable Spring Boot 1.3 war from a Gradle multi project build

I'm trying to build a fully executable WAR using Spring Boot 1.3 as per https://docs.spring.io/spring-boot/docs/current/reference/html/deployment-install.html. If I build a single Gradle project, it all works fine, but I havea multi project build, where I have a "root" project and then several projects underneath it, and I cannot get it to build anything but a standard, "fat" WAR file, without the providedRuntime of Jetty and without the scripts to make it run.
Does anyone know how to do this?
In my root project, I have the following (abridged):
buildscript {
repositories {
mavenCentral()
}
ext {
springBootVersion = '1.3.0.RELEASE'
}
dependencies {
classpath "org.springframework.boot:spring-boot-gradle-plugin:$springBootVersion"
}
}
allprojects {
//Put instructions for all projects
repositories {
mavenCentral() // jcenter is missing spring-orm.4.1.6.RELEASE jar file so try mavenCentral first
jcenter {
url "http://jcenter.bintray.com/"
}
maven { url 'http://repo.opensourceagility.com/release' }
}
}
subprojects {
apply plugin: 'java'
apply plugin: 'spring-boot'
}
and then in the subproject which is a web project, and which I'm trying to build, I have:
apply plugin: 'war'
dependencies {
// Include related projects
compile project(':project-model')
compile project(':project-dynamoDB')
// Core Spring Boot - note version is set in main build.gradle file
compile 'org.springframework.boot:spring-boot-starter-web'
// Remove Tomcat (included in -web) and include Jetty instead
providedRuntime 'org.springframework.boot:spring-boot-starter-jetty'
// Other Spring modules
compile 'org.springframework.boot:spring-boot-starter-social-facebook'
compile 'org.springframework.boot:spring-boot-starter-social-linkedin'
compile 'org.springframework.social:spring-social-google:1.0.0.RELEASE'
compile 'org.springframework.boot:spring-boot-starter-thymeleaf'
compile 'org.springframework.boot:spring-boot-starter-security'
compile 'org.springframework.boot:spring-boot-starter-actuator'
compile 'org.springframework.boot:spring-boot-devtools'
compile 'org.springframework:spring-context'
compile 'org.springframework:spring-context-support'
}
configurations {
providedRuntime.exclude group: 'org.springframework.boot', module:'spring-boot-starter-tomcat'
all*.exclude group: 'org.springframework.boot', module: 'spring-boot-starter-logging' // exclude when using log4j
}
springBoot {
mainClass = 'rs.web.Weblication'
executable = true
}
bootRun {
addResources = true
}
processResources {
// exclude resources if they look like they're profile dependent but don't match the current env/profile
eachFile { d ->
if(d.name.endsWith('.xml') || d.name.endsWith('.yaml') || d.name.endsWith('.properties')) {
//def fname = d.name.replaceFirst(~/\.[^\.]+$/, '')
//if(fname.indexOf("-") > -1 && ! fname.endsWith("-" + environment)) {
// d.exclude()
//} else {
// replace #variables# listed below in properties/config files
filter(org.apache.tools.ant.filters.ReplaceTokens, tokens: [
activeProfiles: environment
])
//}
}
}
}
war {
baseName = 'project-web'
version = '1.0.0'
manifest {
attributes 'Implementation-Title': baseName,
'Implementation-Version': version
}
webXml = file('src/main/resources/web.xml')
// rename the war task which has profiles appended from warName-profile,profile2.war
// to warName-profile.profile2.war
classifier = environment.replaceAll(',','-')
}
but when I build it (./gradlew build, or ./gradlew subprojectname:build), all is well and a working WAR is created, but not an executable one.
With a single project, I have it working fine.
Ah ha, right well I build a test multi-project build and it worked OK, so it was clearly the configuration above.
I worked through a process of elimination and it turns out that the problematic area was the line
classifier = environment.replaceAll(',','-')
which is intended to rename files with environment variables as part of the name. This process seems to get in the way of the script addition; perhaps it could be applied afterwards if it's really necessary.

Resources