Using gradle to programmatically add resource folder to test compile/runtime classpath - gradle

We have a master build script for 60+ components. The individual components do not have build.gradle files. What I'm trying to do is programmatically (in the master build.gradle) add a resource folder to certain projects. This resource folder contains a file which must be in the classpath when unit tests are ran. I'm trying to add this in the subprojects block like this:
subprojects { proj ->
...
// this is the folder I need in the test task classpath
def resdir = sprintf("%s\\resources", project(':Common').projectDir)
sourceSets {
test {
java {
srcDir 'test'
}
resources {
srcDirs = [resdir]
}
}
}
}
...
if(proj.name == "APROJECT"){
proj.tasks['test'].getClasspath().each {
logger.info "COMPILE CLASSPATH: {}", it
}
}
}
However, if I query the classpath of the test task (see above) I do not see the folder in the classpath. Additionally, of course, the test is failing because the folder is not in the classpath.
If I put the sourceSet update in a build.gradle in the component folder, it works as expected.
Am I doing something wrong? How can I get this folder into the classpath for testing?

I wasn't able to get this to work by dynamically updating the sourceSet, however, I was able to get it to work by adding the necessary resource path to the testCompile dependency. See this for adding a class folder to a dependency.
Update: It's still not an ideal solution since the "solution" only adds the class folder to the compile path, it doesn't treat it as a resource (e.g., copy it to the runtime class folder).
Update #2: It's actually working as expected. It turns out that different tests were referencing slightly different resource paths. Adding all resource paths dynamically as noted above works fine!

Related

How to create multi project fat jar as bundle of some libraries with buildSrc

First of all, sorry for my poor english.
Goal
I want create multi project containing some custom libraries as subproject with gradle.
For centralized dependency version control, using buildSrc and setting versions (spring-boot, detekt, ktlint etc.)
my-core-project(root)
+buildSrc
+-src/main/kotlin
+--int-test-convention.gradle.kts
+--library-convention.gradle.kts
+library-A
+-src
+--main/kotlin
+--test/kotlin
+-build.gradle.kts
+library-B
+-src
+--main/kotlin
+--test/kotlin
+-build.gradle.kts
+build.gradle.kts
+setting.gradle.kts
buildSrc contains common tasks for libraries(integration test, detekt, etc.)
library-A and library-B are custom libraries based on spring boot.
There is no application module or any main method.
my goal is using method of library-A and/or library-B with another separated project with adding my-core-project to dependency.
Problem
./gradlew build created 3 jar files
my-core-project
+build/libs
+-my-core-project.jar
+library-A
+-build/libs
+--library-A.jar
+library-B
+-build/libs
+--library-B.jar
copied 3 jar files to libs directory under project which actually using these library,
tried adding dependency created jar
with implementation(files("libs/library-A.jar")), class and methods are resolved well.
but with implementation(files("libs/my-core-project.jar")),
class and methods are not unresolved.
when check my-core-project.jar, recognized that any information of sub projects contained.
Here is my setting.gradle.kts and build.gradle.kts of root directory.
# setting.gradle.kts
pluginManagement {
repositories {
mavenCentral()
gradlePluginPortal()
}
}
rootProject.name = "my-core-project"
include(
"library-A",
"library-B"
)
# build.gradle.kts
plugins {
id("java-library")
id("io.spring.dependency-management")
}
group = "com.demo"
version = "0.0.1-SNAPSHOT"
dependencies {
api(project(":library-A"))
api(project(":library-B"))
}
repositories {
mavenCentral()
}
Tried things
In my opinion, my-core-project.jar should be fatJar(uberJar),
so i added FatJar task like this
val fatJar = task("fatJar", type = Jar::class) {
archiveBaseName.set("${project.name}-fat")
from(configurations.runtimeClasspath.get().map { if (it.isDirectory) it else zipTree(it) })
with(tasks.jar.get() as CopySpec)
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
}
tasks {
"build" {
dependsOn(fatJar)
}
}
but cannot resolve class and method,
additionally occurs version conflict with other dependency of projects using this package, due to library-A created as fatJar too.
Question
Is there a simple way packaging/bundling sub-modules into one jar file?
if there are tasks like this already in gradle, prefer to use that.
Modifying fatJar task like "add jar of submodules, not contained dependencies" can solve this problem?(even couldn't try completely newbie to gradle and kts.)
if so, can you show me how to modify task?
tried shadowJar already. that solved version-conflict problem with relocate option. but still couldn't resolve package in library-A
If structure has problem, is there a good practice/example for "bundle of library"?
thanks for reading.
TL;DR
If someone faced this problem, try set archive name shorter than current one.
For someone who faced same problem, sharing some informations.
as result, resolved this problem.(maybe even not problem)
current shadowJar configure is as following
tasks.named<ShadowJar>("shadowJar").configure {
archiveBaseName.set("shorten-name")
archiveClassifier.set("")
exclude("**/*.kotlin_metadata")
exclude("**/*.kotlin_builtins")
}
exclude kotlin_metadata, kotlin_builtins
set shorten name(original project name was 30 long characters)
I have no idea but shorten jar file name has no problem.
Interesting one is, upload in artifactory package with original(long) name worked well.
I don't know Gradle declaring dependency with files has length constraints.
implementation(files("path/to/package"))
And now it works well with original name with local jar package file.

Gradle dependency destination on non-jar config file

I can create a dependency to something other than a jar file like this:
dependencies {
compile files("../other-project/config.txt")
}
The above works fine, except that config.txt ends up in the WEB-INF/lib folder of my war file. Instead I need it to be in WEB-INF/classes in the war file, and in src/main/resources for jettyRun.
How can I control where the dependency ends up? Or am I going about this the wrong way?
I can also solve this with a copy task, but this really is a dependency in that I don't need the file updated unless it changes. An unconditional copy would work, but I'd rather do this the right way.
The war task (as configured by the war plugin) puts dependencies into WEB-INF/lib, the web project's own code/resources into WEB-INF/classes, and web app content (which by default goes into src/main/webapp) into WEB-INF. Other content can be added by explicitly configuring the war task. For example:
war {
into("WEB-INF/classes") {
from "../other-project/config.txt"
}
}
One way to make this work with embedded Jetty (though maybe not the most convenient during development) is to use jettyRunWar instead of jettyRun. Another solution that comes to mind, particularly if the content to be added resides in its own directory, is to declare that directory as an additional resource directory of the web project (sourceSets.main.resources.srcDir "../other-project/someResourceDir"). This is in fact an alternative to configuring the war task. If the web project already has a dependency on the other project, you could instead configure an additional resource directory for that project.
Let's say you have configured a multi-project build with the following directory and file structure:
/combined-war
/main-project
/src
/webapp
/WEB-INF
web.xml
build.gradle
/other-project
/resources
/WEB-INF
/classes
config.txt
build.gradle
build.gradle
In order to allow jettyRun to combine the contents of the webapp directory from main-project with the contents of the resources directory in other-project you need to add a workaround to your build.gradle of main-project (I've adapted the one posted by the user siasia on gist).
Adding the same directory content to the war file is quite simple and is documented in the Gradle User Guide and and the DSL reference.
apply plugin: 'java'
apply plugin: 'war'
apply plugin: 'jetty'
import org.gradle.api.plugins.jetty.internal.JettyPluginWebAppContext
def newResourceCollection(File... resources) {
def script = '''
import org.mortbay.resource.ResourceCollection
new ResourceCollection(resources)
'''
def shell = new GroovyShell(JettyPluginWebAppContext.class.classLoader)
shell.setProperty("resources", resources as String[])
return shell.evaluate(script)
}
jettyRun.doFirst {
jettyRun.webAppConfig = new JettyPluginWebAppContext()
jettyRun.webAppConfig.baseResource = newResourceCollection(
// list the folders that should be combined
file(webAppDirName),
file("${project(':other-project').projectDir}/resources")
)
}
war {
from("${project(':other-project').projectDir}/resources")
}
Whenever you execute gradle jettyRun a new ResourceCollection is created that combines the given directories. Per default Jetty locks (at least on Windows) all the files it's serving. So, in case you want to edit those files while Jetty is running take a look at the following solutions.
Update
Since other-project in this case is not another Gradle project the two tasks in build.gradle should look like that:
jettyRun.doFirst {
jettyRun.webAppConfig = new JettyPluginWebAppContext()
jettyRun.webAppConfig.baseResource = newResourceCollection(
file(webAppDirName),
file("$projectDir/../other-project/resources")
)
}
war {
from("$projectDir/../other-project/resources")
}
I'm not aware of any solution that adds only one file (e.g. config.txt). You'll always have to add a complete directory.
As I mentioned above, it's simple enough to do an unconditional copy that solves the problem. Again, not the question I originally asked. But here's my solution that works for both war and jettyRun tasks:
processResources.doFirst {
copy {
from '../other-project/config.txt'
into 'src/main/resources'
}
}

Filter out resources from custom Test task in Gradle with goal of having one properties file for tests and a different one for production

So I have setup a way to just run integration tests using this configuration:
test {
exclude "**/*IntegrationTest*.class"
}
task integrationTest(type: Test, dependsOn: testClasses) {
include "**/*IntegrationTest*.class"
}
check.dependsOn integrationTest
Works great. But then for logging in my integration tests, I want to use a log4j.properties file from a specific directory instead of the one that is located in the src/main/resoures which is used for production.
I've tried this but didn't work:
integrationTest.classpath = files("$rootDir/test/src/main/resources/log4j.properties") + integrationTest.classpath
I also tried to just see if I could exclude the file, but could not find a way. Tried this:
processTestResources.exclude "**/*log4j.properties"
Any suggestions for including one properties file in production and another one for tests?
If you put the 'test' log4j.properties in the src/test/resources directory, it will actually come before anything in src/main/resources on the classpath of your tests/integration tests.
An alternative solution is to setup your bundling so that the log4j.properties for production is not in src/main/resources, but is added to the jar from a different directory...
jar {
from('production') {
include('log4j.properties')
}
}
If you want to keep the log4j.properties files in their current locations, you were almost there with what you tried. On the classpath you can have either jar files, or directories containing resources. So try doing this:
integrationTest.classpath = files("$rootDir/test/src/main/resources") + integrationTest.classpath
Seems like you could do something like this also so that the .jar task will include it later:
sourceSets {
main {
java {
srcDir 'src/main/java'
output.classesDir = 'build/classes/main'
}
resources {
srcDir 'src/main/resources'
include 'logback.xml'
output.resourcesDir = 'build/resources/main'
}
}
}

Gradle - can I include task's output in project dependencies

I have a task that generates java sources and a set of jars from these sources (say, project a). I would like to export these jars to dependent projects (say, project b). So here's roughly what I have right now:
//a.gradle
configurations{
generatedJars
}
task generateJars(type: JavaExec) {
//generate jars ...
outputs.files += //append generated jars here
}
dependencies{
generatedJars generateJars.outputs.files
}
//b.gradle
dependencies{
project(path: ':a', configuration: 'generatedJars')
}
It works OK, except that adding generateJars.outputs.files as a dependency does not tell gradle that it has to run generateJars task when there are no jars generated yet. I have tried adding the task itself as a dependency hoping that it would work in the same way as it does when you add a jar/zip task to an artifact configuration (e.g. artifacts{ myJarTask }), but it throws an error telling me that I cannot do that. Of course I can inject the generateJars task somewhere in the build process before :b starts evaluating, but that's clumsy and brittle, so I would like to avoid it.
I feel like I should be adding the generated jars to artifacts{ ... } of the project, but I am not sure how to make them then visible to dependent projects. Is there a better way of achieving this?
Dependent projects (project b) will need to do setup IntelliJ IDEA module classpath to point to project a's generated jars. Something rather like this (pseudo-code):
//b.gradle
idea{
module{
scopes.COMPILE.plus += project(path: ':a', configuration: 'generatedJars').files
}
}
So far I have tried simply adding a project dependecy on :a's generatedJars in :b, but Idea plugin simply adds module :a as a module-dependency and assumes that it exports its generated jars (which is probably a correct assumption), therefore not adding the generated jars to :b's classpath.
Any help would be greatly appreciated!
First, do you need a separate configuration? That is, do you have clients of a that should not see the generated Jars? If not, you can add the generated Jars to the archives configuration, which will simplify things.
Second, the correct way to add the generated Jars to the configuration is (instead of the dependencies block):
artifacts {
generatedJars generateJars
}
This should make sure that the generateJars task gets run automatically when needed.
Third, I'd omit the += after outputs.files, although it might not make a difference. You should also add the necessary inputs.
Fourth, why do you need a JavaExec task to generate the Jars? Can you instead add the generated sources to some source set and let Gradle build them?
Fifth, IDEA doesn't have a concept corresponding to Gradle's project configuration dependencies. Either an IDEA module fully depends on another module, or not at all. You have two options: either use a module dependency and make the generated sources a source folder of the depended-on module (preferably both in the Gradle and the IDEA build), or pass the generated Jars as external dependencies to IDEA. In either case, you should probably add a task dependency from ideaModule to the appropriate generation task. If this still doesn't lead to a satisfactory IDEA setup, you could think about moving the generation of the Jars into a separate subproject.
For my use case, I had a C++ project which generated some native libraries which my java project needed to load in order to run.
In the project ':native' build.gradle:
task compile(type: Exec, group: 'build') {
dependsOn ...
outputs.files(fileTree('/some/build/directory') {
include 'mylib/libmy.so'
})
...
}
In project java application build.gradle:
configurations {
nativeDep
}
// Add dependency on the task that produces the library
dependencies {
nativeDep files(project(':native').tasks.findByPath('compile'))
}
// Unfortunately, we also have to do this because gradle will only
// run the ':native:compile' task if we needed the tasks inputs for another
// task
tasks.withType(JavaCompile) {
dependsOn ':native:compile'
}
run {
doFirst {
// Use the configuration to add our library to java.library.path
def libDirs = files(configurations.nativeDep.files.collect {it.parentFile})
systemProperty "java.library.path", libDirs.asPath
}
}

How to compile single class dependency in gradle outside the main project

I have a gradle project that contains only Selenium/TestNG test classes. They are executed against a deployed war application. All works fine and now I'm adding a java utility that will query the test base and print list of tests that belong to a given TestNG group. The utility should be compiled and executed separate from the main project, as users may want to query the test base before test execution.
I added the following to build.gradle:
task listgroups(dependsOn:'buildUtil' ) <<{
ant.java(classname: 'util.TestGroupScanner', fork: true,
classpath: "src/test/java")
}
task buildUtil {
compile {
source = "src/test/java/util"
}
}
However, when calling listgroups task, I'm getting the following error:
C:\console-bg1>g listgroups
FAILURE: Build failed with an exception.
(...)
* What went wrong:
A problem occurred evaluating root project 'console-bg1'.
> Could not find method compile() for arguments [build_4emu7duna2isgubc1k8uts8k9
8$_run_closure6_closure11#d210ab] on root project 'console-bg1'.
I'm not sure how to resolve this issue and needless to say, haven't found an answer online so far. Any pointers appreciated.
The problem is in the buildUtil task, as the error suggests. The buildUtil declares a compile closure, but such closure does not exist for the default task.
Let me try to clarify what your setup is. The util.TestGroupScanner source is in the src/test/java/util directory, which you want to compile separately from other source (presumably src/main/java and src/test/java). The buildUtil task is supposed to compile sources in src/test/java/util, and the listgroups task executes the scanner utility on sources src/test/java folder.
In this case, I'd suggest you declare a new source set for your utility sources, like this:
sourceSets {
util {
java {
srcDir 'src/test/java/util'
}
}
}
This will automatically create a compile task called compileUtilJava for you, that will compile those sources. I also think you'll want to include utility classes in the classpath when executing your tool, which can be retrieved by sourceSets.util.output.classesDir. So now your listgroups task will look like:
task listgroups(dependsOn: 'compileUtilJava' ) <<{
ant.java(classname: 'util.TestGroupScanner', fork: true,
classpath: "src/test/java:" + sourceSets.util.output.classesDir)
}
One thing I have noticed about your setup, is that src/test/java/util source folder is nested under src/test/java. Gradle will assume src/test/java to be the default folder for your project test, and will therefore automatically include it, and all of its children when running tests. Since you want to keep your utility folder separate from the default setup, I would recommend you put it in src/testutil/java, to avoid any clashes. If you do, don't forget to update the sourceSets setup above with the correct source path.
To solve this with gradle, I suggest to create a specific sourceset for your util class and add a task of type JavaExec that executes this class for printing your testng groups. Have a look at the following snippet:
apply plugin:'java'
...
...
configurations{
testUtilCompile.extendsFrom testCompile
}
...
...
sourceSets{
testUtil{
java {
srcDir "src/test/java"
include "util/**"
}
}
}
task printGroups(type:JavaExec){
main = "util.TestGroupScanner"
classpath = sourceSets.testUtil.runtimeClasspath
}
regards,
René

Resources