Build jar with binaries for required platform only (javacpp) - gradle

I want to build a fat jar for tesseract. With the following build settings I get a jar of about 68 MB with dependencies for all supported platforms:
dependencies {
implementation group: 'org.bytedeco', name: 'tesseract-platform', version: '4.1.1-1.5.3'
}
jar {
manifest { attributes 'Main-Class': 'BasicExample' }
from { configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) } }
}
In order to reduce this size, I tried to only include the dependencies for my platform following this guideline - but without success, see also this SO question and #Samuel Audet's answer to it (I want to stick with gradle). Therefore I decided to manually include only the required dependencies from the POM file:
dependencies {
implementation group: 'org.bytedeco', name: 'tesseract', version: '4.1.1-1.5.3'
implementation group: 'org.bytedeco', name: 'tesseract', version: '4.1.1-1.5.3', classifier: 'windows-x86_64'
implementation group: 'org.bytedeco', name: 'leptonica', version: '1.79.0-1.5.3', classifier: 'windows-x86_64'
}
This reduces the size of the jar to about 7 MB and works fine (at least for the basic example). Nevertheless I get warnings:
Warning: Could not load ...: java.lang.UnsatisfiedLinkError: no jnijavacpp in java.library.path: ...
Comparing the two jars I found that the small jar lacks the lib path with all the so libs along with header, cmake and pkgconfig files and I assume that this is the reason for the warnings.
So my questions are:
If these files are apparently not necessary to run the jar, why are they included normally?
How can I prevent these warnings while keeping the jar size small?
Any other way to build a jar with just the required dependencies for one platform is of course also welcomed.

I'll have to update that guide a little bit, thanks for reporting! We now also need javacpp-platform, which would give us something like this for windows-x86_64 in this case:
dependencies {
implementation group: 'org.bytedeco', name: 'tesseract', version: '4.1.1-1.5.3'
implementation group: 'org.bytedeco', name: 'tesseract', version: '4.1.1-1.5.3', classifier: 'windows-x86_64'
implementation group: 'org.bytedeco', name: 'leptonica', version: '1.79.0-1.5.3', classifier: 'windows-x86_64'
implementation group: 'org.bytedeco', name: 'javacpp', version: '1.5.3', classifier: 'windows-x86_64'
}
This is needed to fix some issues at load time as explained in this issue:
https://github.com/bytedeco/javacv/issues/1305
UPDATE: It's now possible to use Gradle JavaCPP to do this more easily this way:
plugins {
id 'java-library'
id 'org.bytedeco.gradle-javacpp-platform' version "$javacppVersion"
}
// We can set this on the command line too this way: -PjavacppPlatform=linux-x86_64,macosx-x86_64,windows-x86_64,etc
ext {
javacppPlatform = 'linux-x86_64,macosx-x86_64,windows-x86_64,etc' // defaults to Loader.getPlatform()
}
dependencies {
api "org.bytedeco:tesseract-platform:$tesseractVersion"
}

Related

Jetty AnnotationParser scanned from multiple locations warning

I have a project build with Gradle, it's actually a Vaadin project, with a servlet where I use Jetty.
At the startup (i.e gradle run) I have a lot of different warning message from AnnotationParser about duplication of classes. I copy only one because the log is quite verbose:
[INFO ] 11:22:50.375 org.eclipse.jetty.server.Server.doStart() - jetty-9.4.31.v20200723; built: 2020-07-23T17:57:36.812Z; git: 450ba27947e13e66baa8cd1ce7e85a4461cacc1d; jvm 13.0.2+8
[WARN ] 11:22:50.777 org.eclipse.jetty.annotations.AnnotationParser.addParsedClass() - javax.websocket.ClientEndpoint scanned from multiple locations: jar:file:///Users/fox/.gradle/caches/modules-2/files-2.1/javax.websocket/javax.websocket-api/1.0/fc843b649d4a1dcb0497669d262befa3918c7ba8/javax.websocket-api-1.0.jar!/javax/websocket/ClientEndpoint.class, jar:file:///Users/fox/.gradle/caches/modules-2/files-2.1/javax.websocket/javax.websocket-client-api/1.0/afcf19e889d8725576811f8d47ab6c65d9dcbd78/javax.websocket-client-api-1.0.jar!/javax/websocket/ClientEndpoint.class
In this line AnnotationParser warns me that the ClientEndpoint.class is present in two jar, the javax.websocket-api-1.0.jar and the javax.websocket-client-api-1.0.jar.
With the command gradle dependencies I could see:
...
+--- org.eclipse.jetty.websocket:javax-websocket-client-impl:9.4.31.v20200723
| +--- org.eclipse.jetty.websocket:websocket-client:9.4.31.v20200723 (*)
| \--- javax.websocket:javax.websocket-client-api:1.0
+--- org.eclipse.jetty.websocket:websocket-server:9.4.31.v20200723 (*)
\--- javax.websocket:javax.websocket-api:1.0
In my gradle.build I only have:
dependencies {
// Vaadin
implementation enforcedPlatform('com.vaadin:vaadin-bom:18.0.6')
implementation("com.vaadin:vaadin-core")
implementation group: 'com.github.appreciated', name: 'vaadin-css-grid', version: '2.0.0'
// Logging
implementation group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.13.1'
implementation group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.13.1'
implementation group: 'org.apache.logging.log4j', name: 'log4j-slf4j-impl', version: '2.13.1'
// Testing
testImplementation group: 'junit', name: 'junit', version: '4.12'
testImplementation group: 'org.skyscreamer', name: 'jsonassert', version: '1.5.0'
// Jetty
implementation group: 'org.eclipse.jetty', name: 'jetty-server', version: '9.4.31.v20200723'
implementation group: 'org.eclipse.jetty', name: 'jetty-webapp', version: '9.4.31.v20200723'
implementation group: 'org.eclipse.jetty.websocket', name: 'websocket-server', version: '9.4.31.v20200723'
implementation group: 'org.eclipse.jetty.websocket', name: 'javax-websocket-server-impl', version: '9.4.31.v20200723'
}
All these logs are really annoying (they slow down the start up really much) and I can't understand if they are dangerous or not.
Which is the right way to avoid duplications ?
Is there a technique to instruct AnnotationParser to scan only some dependencies?
I know this topic is present in different questions but I didn't find a solution for gradle or a common solution strategy.
Thanks,
Stefano
Having the same class name in multiple locations on your classpath is a bad idea.
This is the most common form of unstable operation on Java there is!
The classloaders in Java have no guarantee of load order behavior if the same class is present in multiple locations within the classloader.
In one run you might accidentally load the classes in the order you intend, and have it run properly, then at a future date you run the same program and the load order is different, now you are running with a different class version and you have unexpected behavior.
The only way to fix this is to clean up your classloader and ensure that you only have 1 version of the class you intend to use.
This is what Jetty is telling you.
As for this specific one, javax.websocket-api and javax.websocket-client-api, you want to exclude javax.websocket-client-api at the gradle level, as all of the websocket client classes are also present in the javax.websocket-api.
The javax.websocket-client-api jar is only intended for projects that only use the javax.websocket Client, without a javax.websocket Server.
Following the suggestion of joakim-erdfelt
I have modified my build.gradle and this prevent the problem:
implementation ('org.eclipse.jetty.websocket:javax-websocket-server-impl:9.4.31.v20200723') {
exclude module: 'javax.websocket-client-api'
}
The Gradle documentation is here: Excluding transitive dependencies

read excel file from build.gradle

i am new to gradle.Can someone please help me out to find out a way in which we can read some values from an excel file in build.gradle script. these values will be used to decide the testng test suites thatneed to be run dynamically
use imports to add poi jars and use them as you do in normal java program
remmber to import packges at the first very first section(top) of build.gradle file
buildscript {
repositories { mavenCentral() }
dependencies {
// classpath group: 'org.apache.commons', name: 'commons-csv', version: '1.0'
classpath group: 'org.apache.poi', name: 'poi', version: '3.13'
classpath group: 'org.apache.poi', name: 'poi-ooxml', version: '3.13'
}

How to exclude dependency with classifier (platform version) in Gradle?

In my project I have dependency on 'org.nd4j:nd4j-native-platform:0.6.0' which brings me transitive dependencies:
Gradle: org.nd4j:nd4j-native:linux-ppc64le:0.6.0
Gradle: org.nd4j:nd4j-native:macosx-x86_64:0.6.0
Gradle: org.nd4j:nd4j-native:windows-x86_64:0.6.0
Gradle: org.nd4j:nd4j-native:linux-x86_64:0.6.0
I want to exclude nd4j-native:linux-ppc64le and nd4j-native:macosx-x86_64 since my application does not support these platforms. I write in my Gradle file:
configurations {
all.collect { configuration ->
configuration.exclude(group: 'org.nd4j', module: 'nd4j-native', classifier: 'linux-ppc64le')
}
}
Gradle says:
Error:(44, 0) Could not set unknown property 'classifier' for object of type org.gradle.api.internal.artifacts.DefaultExcludeRule.
It seems that gradle does not support exclusion by classifier.
How to exclude such a transitive dependencies?
Update: Gradle allows us to exclude dependencies, but what if we have several dependencies with the same id and group but different classifiers?
I have faced the same issued. I have used a deeplearning4j library with Gradle dependency.
compile group: 'org.nd4j', name: 'nd4j-native-platform', version: '1.0.0-beta'
compile group: 'org.deeplearning4j', name: 'deeplearning4j-core', version: '1.0.0-beta'
When I use this it is also downloading other platform classifiers and its size is almost 500MB. but my use case is specific to the Windows platform so i don't need other classifiers for Linux and Android and other platforms.If I exclude the group it is also excluding the classifier for the windows also . And in Gradle as of my knowledge, we can not exclude specific classifiers.
So the question was how to remove the specific classifier. What I found strange is when I made jar file of the project and extracted jar it shows me the org.nd4j:nd4j-native:linux-ppc64le:0.6.0 and other jars but when I generate dependency tree it is not showing me the specific jar in the tree.
So in order to find out in which specific module and project the jar is coming from I made a separate maven project and with this
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>nd4j-native-platform</artifactId>
<version>1.0.0-beta</version>
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-core</artifactId>
<version>1.0.0-beta</version>
</dependency>
and then I have generated a dependency tree. It showed me the jars in the dependency tree.
What I did is I have removed the whole module and I have added the required classifier in a particular module with a specific version and it worked for me.
compile (group: 'org.deeplearning4j', name: 'deeplearning4j-core', version: '1.0.0-beta')
{
exclude group: 'org.bytedeco.javacpp-presets', module: 'opencv-platform'
exclude group: 'org.bytedeco.javacpp-presets', module: 'leptonica-platform'
exclude group: 'org.bytedeco.javacpp-presets', module: 'hdf5-platform'
}
compile (group: 'org.nd4j', name: 'nd4j-native-platform', version: '1.0.0-beta')
{
exclude group: 'org.bytedeco.javacpp-presets', module: 'openblas-platform'
}
compile group: 'org.nd4j', name: 'nd4j-native', version: '1.0.0-beta', classifier: "windows-x86_64"
compile group: 'org.bytedeco.javacpp-presets', name: 'openblas', version: '0.2.20-1.4.1'
compile group: 'org.bytedeco.javacpp-presets', name: 'openblas', version: '0.2.20-1.4.1', classifier: "windows-x86"
compile group: 'org.bytedeco.javacpp-presets', name: 'openblas', version: '0.2.20-1.4.1', classifier: "windows-x86_64"
compile group: 'org.bytedeco.javacpp-presets', name: 'opencv', version: '3.4.1-1.4.1'
compile group: 'org.bytedeco.javacpp-presets', name: 'opencv', version: '3.4.1-1.4.1',classifier: "windows-x86"
compile group: 'org.bytedeco.javacpp-presets', name: 'opencv', version: '3.4.1-1.4.1',classifier: "windows-x86_64"
compile group: 'org.bytedeco.javacpp-presets', name: 'leptonica', version: '1.75.3-1.4.1'
compile group: 'org.bytedeco.javacpp-presets', name: 'leptonica', version: '1.75.3-1.4.1',classifier: "windows-x86"
compile group: 'org.bytedeco.javacpp-presets', name: 'leptonica', version: '1.75.3-1.4.1',classifier: "windows-x86_64"
Doing this reduced my jar size to almost 250MB
How to exclude such a transitive dependencies?
I think, the only way is to exclude all transitive dependencies by it's module or group and manually provide dependencies on libraries for platforms your application supports. Because classifiers are supported in dependency declaration.
And the same way you can handle the case, when you have a number of dependencies with the same module and grooup, but with different classifiers. Just add such dependencies manually with it's classifier property.

Error building project while excluding transitive dependency in build.gradle file

I am creating a multi module project with Spring Boot for deploying my application and using gradle as my build tool.
Now, I am creating independent deployments for some modules in the project. Some of the project requires Embedded tomcat and some do not. All the common dependencies have been put on a common project and all other modules are dependent on this common project.
Most of the other deployment modules require an embedded tomcat application server and other web components (provided by org.springframework.boot', name: 'spring-boot-starter-web) so this has been included in the build.gradle for common project
Here is the build.gradle for common project:
compile group: 'org.springframework.boot', name: 'spring-boot-starter-web', version: springBootVersion
compile group: 'org.springframework.boot', name: 'spring-boot-starter-actuator', version: springBootVersion
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-jpa', version: springBootVersion
compile group: 'org.springframework.data', name: 'spring-data-jpa', version: springDataJpaVersion
Now, one of the other modules which is going to be deployed independently does not require this embedded tomcat and other mvc jars which comes with including spring-boot-starter-web but requires the other transitive dependencies from common project. As such, I want to exclude the transitive dependency for
compile group: 'org.springframework.boot', name: 'spring-boot-starter-web', version: springBootVersion
I am doing it like this in the build.gradle other project
dependencies {
compile project(':common') {
exclude group 'org.springframework.boot', module:'spring-boot-starter-web'
}
}
But while building it is throwing this error:
startup failed: build file
'/Users/user1/Documents/repo/storm/build.gradle': 30: expecting '}',
found ',' # line 30, column 44.
oup 'org.springframework.boot', module:'
changing it to:
dependencies {
compile project(':common'){
exclude group 'org.springframework.boot:spring-boot-starter-web'
}
}
throws:
Could not find method exclude() for arguments [parent-project name] on
project ':common'.
How can I exclude transitive dependency here?
You need to invoke exclude on compile not on project:
dependencies {
compile(project(':common')) {
exclude group: 'org.springframework.boot', module:'spring-boot-starter-web'
}
}

could not resolve all dependencies for configuration ':compile'

To study Gradle I am using the book Gradle in action.
There was an example of dependency definition.
dependencies {
compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.1'
}
But when I do in console gradle build I've got an error
What is the problem? My whole .gradle file looks like this
apply plugin: 'java'
dependencies {
compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.1'
}
You did not tell Gradle where to find commons-lang3 library. Easy fix is to add the following to your build script:
repositories {
mavenCentral()
}
Of course you can find this piece of information in documentation - http://www.gradle.org/docs/current/userguide/artifact_dependencies_tutorial.html#N10608
I was facing this same issue. I fixed it by using local gradle distribution instead of the default gradle wrapper. This is how it goes, make sure that you have Gradle installed and setup(PATH variable).
Open IntelliJ. Goto File->setting->Build, Exec, Deployment->Build tools->Gradle and use local gradle distribution and add your Gradle Home. Apply changes and try to run it now.

Resources