In my code there is const val VERSION = $version.
I want to replace $version with real version string which is in my build.gradle.kts.
How can I do this?
Working example here.
One way is to use a template file (stored outside of src tree). Let's call it TemplateVersion.kt:
class Version() {
val version = "__VERSION";
}
and then in build.gradle.kts, as initial part of the compileKotlin task, we generate Version.kt from TemplateVersion.kt:
val sourceFile = File(rootDir.absolutePath + "/resources/TemplateVersion.kt")
val propFile = File(rootDir.absolutePath + "/gradle.properties")
val destFile = File(rootDir.absolutePath + "/src/main/kotlin/${targetPackage}/Version.kt")
tasks.register("generateVersion") {
inputs.file(sourceFile)
inputs.file(propFile)
outputs.file(destFile)
doFirst {
generateVersion()
}
}
tasks.named("compileKotlin") {
dependsOn("generateVersion")
}
fun generateVersion() {
val version: String by project
val rootDir: File by project
val inputStream: InputStream = sourceFile.inputStream()
destFile.printWriter().use { out ->
inputStream.bufferedReader().forEachLine { inputLine ->
val newLine = inputLine.replace("__VERSION", version)
out.println(newLine)
}
}
inputStream.close()
}
where gradle.properties is:
version=5.1.50
It is trivially easy to add more fields to Version.kt, such as a build-timestamp.
(Edit: This has been updated with a proper generateVersion task that will detect changes to gradle.properties. The compileKotlin task will invoke this task).
I have created a Kotlin MPP to share Json utilities between JVM and JS. All the code lies in the common source set and I have configured the necessary targets with their respective dependencies. Without further configuration I'm able to use the utilities from both JVM and JS but not from the common source set of another MPP, which has to do with the way Gradle handles metadata.
I already found the solution (taken from https://medium.com/xorum-io/crafting-and-publishing-kotlin-multiplatform-library-to-bintray-cbc00a4f770)
afterEvaluate {
project.publishing.publications.all {
groupId = group
if (it.name.contains('metadata')) {
artifactId = "$libraryName"
} else {
artifactId = "$libraryName-$name"
}
}
}
and I also got it to work with the Gradle Kotlin DSL:
afterEvaluate {
publishing.publications.all {
this as MavenPublication
artifactId = project.name + "-$name".takeUnless { "metadata" in name }.orEmpty()
}
}
However, this doesn't feel quite right yet.
There is no such code snippet in the official documentation.
The documentation advertises that a single dependency from the common source set should suffice to automatically resolve target specific dependencies: https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#metadata-publishing. I had to add the dependency for each target, respectively, for it to work.
this as MavenPublication is necessary because Publication has no field artifactId.
I use project.name instead of libraryName.
Is this even remotely the right way to do things or am I missing some other option which would make the whole process trivial?
Right now I'm using Kotlin 1.3.72 and Gradle 5.2.1 with enableFeaturePreview("GRADLE_METADATA") in settings.gradle.kts. I also tried it with Gradle 6.5.1 (latest) but it behaves exactly the same.
For now I'm glad that it's working at all but I suspect there is a cleaner way to do this. I'd really appreciate if someone with a bit more Gradle expertise could clear things up for me or point me into the right direction.
Edit:
gradle.build.kts for completeness. Although there isn't much going on here.
group = "org.example"
version = "1.0-SNAPSHOT"
plugins {
kotlin("multiplatform") version "1.3.72"
`maven-publish`
}
repositories {
mavenCentral()
}
kotlin {
jvm()
sourceSets {
val commonMain by getting {
dependencies {
implementation(kotlin("stdlib-common"))
}
}
val jvmMain by getting {
dependencies {
implementation(kotlin("stdlib"))
}
}
}
}
There wasn't really a problem after all. The solution is to simply add enableFeaturePreview("GRADLE_METADATA") to the consuming project too.
According to https://kotlinlang.org/docs/reference/building-mpp-with-gradle.html#metadata-publishing this shouldn't be necessary:
In earlier Gradle versions starting from 5.3, the module metadata is
used during dependency resolution, but publications don't include any
module metadata by default. To enable module metadata publishing, add
enableFeaturePreview("GRADLE_METADATA") to the root project's
settings.gradle file.
Weirdly it only works when both publishing project and consuming project have metadata enabled, even when both use the latest Gradle version.
enableFeaturePreview("GRADLE_METADATA") is enabled by default in latest gradle.
According to this you need to substitute "kotlinMultiplatform" by "" an empty string.
I finally managed to accomplish publishing to bintray with maven-publish plugin only, without outdated bintray library.
Here is my full maven.publish.gradle.kts:
import java.io.FileInputStream
import java.util.*
import org.gradle.api.publish.PublishingExtension
apply(plugin = "maven-publish")
val fis = FileInputStream("local.properties")
val properties = Properties().apply {
load(fis)
}
val bintrayUser = properties.getProperty("bintray.user")
val bintrayApiKey = properties.getProperty("bintray.apikey")
val bintrayPassword = properties.getProperty("bintray.gpg.password")
val libraryVersion: String by project
val publishedGroupId: String by project
val artifact: String by project
val bintrayRepo: String by project
val libraryName: String by project
val bintrayName: String by project
val libraryDescription: String by project
val siteUrl: String by project
val gitUrl: String by project
val licenseName: String by project
val licenseUrl: String by project
val developerOrg: String by project
val developerName: String by project
val developerEmail: String by project
val developerId: String by project
project.group = publishedGroupId
project.version = libraryVersion
afterEvaluate {
configure<PublishingExtension> {
publications.all {
val mavenPublication = this as? MavenPublication
mavenPublication?.artifactId =
"${project.name}${"-$name".takeUnless { "kotlinMultiplatform" in name }.orEmpty()}"
}
}
}
configure<PublishingExtension> {
publications {
withType<MavenPublication> {
groupId = publishedGroupId
artifactId = artifact
version = libraryVersion
pom {
name.set(libraryName)
description.set(libraryDescription)
url.set(siteUrl)
licenses {
license {
name.set(licenseName)
url.set(licenseUrl)
}
}
developers {
developer {
id.set(developerId)
name.set(developerName)
email.set(developerEmail)
}
}
organization {
name.set(developerOrg)
}
scm {
connection.set(gitUrl)
developerConnection.set(gitUrl)
url.set(siteUrl)
}
}
}
}
repositories {
maven("https://api.bintray.com/maven/${developerOrg}/${bintrayRepo}/${artifact}/;publish=1") {
credentials {
username = bintrayUser
password = bintrayApiKey
}
}
}
}
And build.gradle.kts:
plugins {
id("kotlin-multiplatform")
}
kotlin {
sourceSets {
jvm()
js() {
browser()
nodejs()
}
linuxX64()
linuxArm64()
mingwX64()
macosX64()
iosArm64()
iosX64()
val commonMain by getting {
dependencies {
}
}
val commonTest by getting {
dependencies {
implementation(kotlin("test-common"))
implementation(kotlin("test-annotations-common"))
}
}
val jsMain by getting {
dependencies {
}
}
val jsTest by getting {
dependencies {
implementation(kotlin("test-js"))
}
}
val jvmMain by getting {
dependencies {
}
}
val jvmTest by getting {
dependencies {
implementation(kotlin("test"))
implementation(kotlin("test-junit"))
}
}
val nativeMain by creating {
dependsOn(commonMain)
dependencies {
}
}
val linuxX64Main by getting {
dependsOn(nativeMain)
}
val linuxArm64Main by getting {
dependsOn(nativeMain)
}
val mingwX64Main by getting {
dependsOn(nativeMain)
}
val macosX64Main by getting {
dependsOn(nativeMain)
}
val iosArm64Main by getting {
dependsOn(nativeMain)
}
val iosX64Main by getting {
dependsOn(nativeMain)
}
}
}
apply(from = "maven.publish.gradle.kts")
Please note, there are bintray.user and bintray.apikey properties in local.properties file.
Also gradle.properties contains rest listed properties above:
libraryVersion = 0.5.22
libraryName = MultiplatformCommon
libraryDescription = Kotlin multiplatform extensions
publishedGroupId = com.olekdia
artifact = multiplatform-common
bintrayRepo = olekdia
bintrayName = multiplatform-common
siteUrl = https://gitlab.com/olekdia/common/libraries/multiplatform-common
gitUrl = https://gitlab.com/olekdia/common/libraries/multiplatform-common.git
.........
kotlin.mpp.enableGranularSourceSetsMetadata = true
systemProp.org.gradle.internal.publish.checksums.insecure = true
If you haven't created organisation in bintray you need to change in this url:
https://api.bintray.com/maven/${developerOrg}/${bintrayRepo}/${artifact}/;publish=1
developerOrg by bintrayUser, where the last one is your user name at bintray.com
I have a Gradle build file which uses ProtoBuffer plugin and runs some tasks. At some point some tasks are run for some files, which are inputs to tasks.
I want to modify the set of files which is the input to those tasks. Say, I want the tasks to be run with files which are listed, one per line, in a particular file. How can I do that?
EDIT: Here is a part of rather big build.gradle which provides some context.
configure(protobufProjects) {
apply plugin: 'java'
ext {
protobufVersion = '3.9.1'
}
dependencies {
...
}
protobuf {
generatedFilesBaseDir = "$projectDir/gen"
protoc {
if (project.hasProperty('protocPath')) {
path = "$protocPath"
}
else {
artifact = "com.google.protobuf:protoc:$protobufVersion"
}
}
plugins {
...
}
generateProtoTasks {
all().each { task ->
...
}
}
sourceSets {
main {
java {
srcDirs 'gen/main/java'
}
}
}
}
clean {
delete protobuf.generatedFilesBaseDir
}
compileJava {
File generatedSourceDir = project.file("gen")
project.mkdir(generatedSourceDir)
options.annotationProcessorGeneratedSourcesDirectory = generatedSourceDir
}
}
The question is, how to modify the input file set for existing task (which already does something with them), not how to create a new task.
EDIT 2: According to How do I modify a list of files in a Gradle copy task? , it's a bad idea in general, as Gradle makes assumptions about inputs and outputs dependencies, which can be broken by this approach.
If you would have added the gradle file and more specific that would have been very helpful. I will try to give an example from what I have understood:
fun listFiles(fileName: String): List<String> {
val file = file(fileName).absoluteFile
val listOfFiles = mutableListOf<String>()
file.readLines().forEach {
listOfFiles.add(it)
}
return listOfFiles
}
tasks.register("readFiles") {
val inputFile: String by project
val listOfFiles = listFiles(inputFile)
listOfFiles.forEach {
val file = file(it).absoluteFile
file.readLines().forEach { println(it) }
}
}
Then run the gradle like this: gradle -PinputFile=<path_to_the_file_that_contains_list_of_files> readFiles
How could I download a file within a gradle task with kotlin-dsl?
I want to download a file from an external source (URL) and store + rename it inside my projectfolder.
I tried to search it but I only found groovy solutions.
Thanks in advance!
I found a solution using https://github.com/michel-kraemer/gradle-download-task:
import de.undercouch.gradle.tasks.download.Download
...
plugins {
....
id("de.undercouch.download").version("3.4.3")
}
...
task<DefaultTask>("my-download-task") {
...
val url = " ... "
val dest = File("...")
task<Download>("download-task") {
src(url)
dest(dest)
}
dependsOn("download-task")
}
Simple with no external dependency.
buildscript {
repositories {
jcenter()
google()
}
dependencies {
classpath("com.android.tools.build:gradle:4.0.1")
}
}
tasks.register("downloadPdf"){
val path = "myfile.pdf"
val sourceUrl = "https://file-examples-com.github.io/uploads/2017/10/file-sample_150kB.pdf"
download(sourceUrl,path)
}
fun download(url : String, path : String){
val destFile = File(path)
ant.invokeMethod("get", mapOf("src" to url, "dest" to destFile))
}
How can I build a runnable jar with SBT?
Perhaps Spring-Boot isn't suitable for SBT built applications? Any suggestions on this?
I tried using sbt-assembly but it fails when I try to run it.
Note that sbt run works
mainClass in assembly := Some("com.xagongroup.xagon.app.XagonETL")
assemblyMergeStrategy in assembly := {
case PathList("META-INF", _ # _*) => MergeStrategy.discard
case _ => MergeStrategy.first
}
Stack Trace
org.springframework.beans.factory.BeanDefinitionStoreException: Failed to process import candidates for configuration class [com.x.app.XETL]; nested exception is java.lang.Ille
galArgumentException: No auto configuration classes found in META-INF/spring.factories. If you are using a custom packaging, make sure that file is correct.
at org.springframework.context.annotation.ConfigurationClassParser.processDeferredImportSelectors(ConfigurationClassParser.java:556)
at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:185)
at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:308)
at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:228)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:270)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:93)
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:687)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:525)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:303)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107)
at com.x.app.XETL$.main(XETL.scala:21)
at com.x.app.XETL.main(XETL.scala)
Caused by: java.lang.IllegalArgumentException: No auto configuration classes found in META-INF/spring.factories. If you are using a custom packaging, make sure that file is correct.
at org.springframework.util.Assert.notEmpty(Assert.java:277)
at org.springframework.boot.autoconfigure.AutoConfigurationImportSelector.getCandidateConfigurations(AutoConfigurationImportSelector.java:153)
at org.springframework.boot.autoconfigure.AutoConfigurationImportSelector.selectImports(AutoConfigurationImportSelector.java:95)
at org.springframework.context.annotation.ConfigurationClassParser.processDeferredImportSelectors(ConfigurationClassParser.java:547)
... 15 common frames omitted
I solved the issue by moving to sbt-native-packager
plugins.sbt
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.2.0")
build.sbt
scriptClasspath := Seq("*")
mainClass in Compile := Some("com.x.app.XETL")
enablePlugins(JavaAppPackaging)
Running:
packaging
sbt universal:stage
starting the app:
target\universal\stage\bin\x.bat
I've solved this without any plugins. It seems the only trick here is to not discard META-INF/spring.factories files in your build.sbt
assemblyMergeStrategy in assembly := {
case PathList("META-INF", "spring.factories") => MergeStrategy.filterDistinctLines
case PathList("META-INF", _*) => MergeStrategy.discard
case _ => MergeStrategy.first
}
In order to save someone else's time:
Brief
Spring Boot uses special jar format which has different inner structure, changes to manifest and can contain jars inside, which should be just stored in resulting archives as-is (it is called ZipEntry.STORED in contrast with ZipEntry.DEFLATED).
Solution
Neither of plugins I knew (sbt-assembly or sbt-native-packager) could do it from the box or with configuration, so I just wrote a custom task for this which can package my relatively simple project (no modules and subprojects).
File layout
...usual build folders...
project/
SpringBootJar.scala
build.sbt
build.sbt
Note the addition of spring-boot-loader in the dependencies and springBootJar task.
import SpringBootJar.Keys.springBootJar
import SpringBootJar.springBootJarTask
val springBootVersion = "2.7.6"
lazy val myProject = (project in file("."))
.settings(
name := "my-project",
scalaVersion := "3.2.1",
...
libraryDependencies ++= Seq(
"org.springframework.boot" % "spring-boot-starter-parent" % springBootVersion pomOnly(),
"org.springframework.boot" % "spring-boot-starter-jdbc" % springBootVersion,
.... other depenencies ....
"org.springframework.boot" % "spring-boot-loader" % springBootVersion % "compile"
),
springBootJar := springBootJarTask.value
)
SpringBootJar.scala
import sbt.{Keys, _}
import Keys._
import sbt.io.Using.fileOutputStream
import java.io.BufferedOutputStream
import java.nio.file.Files
import java.util.jar.{Attributes, JarEntry, JarFile, JarOutputStream, Manifest}
import java.util.zip.{CRC32, ZipEntry, ZipOutputStream}
import scala.collection.immutable.TreeSet
object SpringBootJar {
object Keys {
val springBootJar = taskKey[Unit]("Create spring boot fat jar")
}
val springBootJarTask = Def.task {
val log = streams.value.log
// check if spring-boot-loader is already a dependency
libraryDependencies.value.find(_.name == "spring-boot-loader") match {
case Some(_) => log.info("spring-boot-loader: present")
case None => sys.error(s"Consider added spring-boot-loader to libraryDependencies")
}
// name of resulting artifact
val destArtifact = target.value / s"${name.value}-${version.value}-sb.jar"
val classpath = (Runtime / fullClasspath).value
val externalClasspath = (Runtime / externalDependencyClasspath).value
val classDir = (Compile / classDirectory).value
val resourceDir = (Compile / resourceDirectory).value
// create directory structure
val rootDir = IO.createUniqueDirectory(target.value)
val springBootLoaderDir = rootDir / "org" / "springframework" / "boot" / "loader"
val appClassesDir = rootDir / "BOOT-INF" / "classes"
val libsDir = rootDir / "BOOT-INF" / "lib"
// get the location of the spring-boot-loader library on the file system
log.info("Seeking for spring-boot-loader jar")
val springBootLoaderFile = classpath.find(_.data.getName.startsWith("spring-boot-loader")) match {
case Some(af) => af.data
case None => sys.error("Couldn't find spring-boot-loader, interrupting")
}
// create dir structure
log.info("Creating directory structure")
IO.createDirectories(Seq(springBootLoaderDir, appClassesDir, libsDir))
// creating manifest
log.info("Creating MANIFEST")
val manifest = new Manifest()
val mainAttributes = manifest.getMainAttributes
mainAttributes.put(Attributes.Name.MANIFEST_VERSION, "1.0")
mainAttributes.put(Attributes.Name.SPECIFICATION_TITLE, name.value)
mainAttributes.put(Attributes.Name.SPECIFICATION_VERSION, version.value)
mainAttributes.put(Attributes.Name.SPECIFICATION_VENDOR, organization.value)
mainAttributes.put(Attributes.Name.IMPLEMENTATION_TITLE, name.value)
mainAttributes.put(Attributes.Name.IMPLEMENTATION_VERSION, version.value)
mainAttributes.put(Attributes.Name.IMPLEMENTATION_VENDOR, organization.value)
mainAttributes.put(new Attributes.Name("Start-Class"), (Compile / mainClass).value.getOrElse(""))
mainAttributes.put(Attributes.Name.MAIN_CLASS, "org.springframework.boot.loader.JarLauncher")
// copy all from org/springframework/boot/loader/ to the same path
locally {
log.info("Copying org/springframework/boot/loader/ classes")
val jar = new JarFile(springBootLoaderFile)
val entries = jar.entries()
while (entries.hasMoreElements()) {
val entry = entries.nextElement()
if (!entry.isDirectory && entry.getName.startsWith("org/springframework/boot/loader/")) {
val targetPath = entry.getName.stripPrefix("org/springframework/boot/loader/")
val targetFile = new File(springBootLoaderDir, targetPath)
targetFile.getParentFile.mkdirs()
Files.copy(jar.getInputStream(entry), targetFile.toPath)
}
}
jar.close()
}
// copying all own app files
log.info("Copying app classes")
IO.copyDirectory(classDir, appClassesDir)
// copying app resources
log.info("Copying app resources")
IO.copyDirectory(resourceDir, appClassesDir)
// copying all external libs
log.info("Copying libs")
IO.copy(externalClasspath.filterNot(_.data.base.startsWith("spring-boot-loader")).map {
f => f.data -> libsDir / f.data.getName
})
// creating jar
log.info("Creating jar")
jar(
(rootDir ** "*").get.map { file => (file, file.relativeTo(rootDir).get.getPath) },
destArtifact,
manifest,
Some(System.currentTimeMillis())
)
log.info("Cleaning up")
IO.delete(rootDir)
log.info(s"Created: ${destArtifact.getAbsolutePath}")
}
private def jar(sources: Traversable[(File, String)], outputJar: File, manifest: Manifest, time: Option[Long]): Unit = {
val localTime = time.map(t => t - java.util.TimeZone.getDefault.getOffset(t))
if (outputJar.isDirectory)
sys.error("Specified output file " + outputJar + " is a directory")
else {
val outputDir = outputJar.getParentFile match {
case null => new File(".")
case parentFile => parentFile
}
IO.createDirectory(outputDir)
val emptyCRC = new CRC32().getValue // The CRC32 for an empty value, needed to store directories in zip files
withJarOutput(outputJar, manifest, localTime) { output =>
writeZip(sources.toSeq, output, localTime) { (file, name) =>
val entry = new JarEntry(name)
if (file == null || file.isDirectory) {
entry.setSize(0)
entry.setMethod(ZipEntry.STORED)
entry.setCrc(emptyCRC)
} else if (file.ext == "jar") {
val jarBytes = Files.readAllBytes(file.toPath)
entry.setMethod(ZipEntry.STORED)
entry.setSize(jarBytes.length)
entry.setCompressedSize(jarBytes.length)
entry.setCrc({
val crc = new CRC32()
crc.update(jarBytes)
crc.getValue
})
}
entry
}
}
}
}
private def withJarOutput(file: File, manifest: Manifest, time: Option[Long])(f: ZipOutputStream => Unit) = {
fileOutputStream(false)(file) { fileOut =>
val zipOut = {
val os = new JarOutputStream(fileOut)
val e = new ZipEntry(JarFile.MANIFEST_NAME)
e.setTime(time.getOrElse(System.currentTimeMillis))
os.putNextEntry(e)
manifest.write(new BufferedOutputStream(os))
os.closeEntry()
os
}
try
f(zipOut)
finally
zipOut.close()
}
}
private def writeZip(sources: Seq[(File, String)], output: ZipOutputStream, time: Option[Long])(
createEntry: (File, String) => ZipEntry
): Unit = {
val files = sources.flatMap {
case (file, name) => if (file.isFile) (file, normalizeToSlash(name)) :: Nil else Nil
}.sortBy {
case (_, name) => name
}
val now = System.currentTimeMillis
def addDirectoryEntry(file: File, name: String): Unit = {
output.putNextEntry {
val e = createEntry(file, name)
e.setTime(time.getOrElse(now))
e
}
output.closeEntry()
}
def addFileEntry(file: File, name: String): Unit = {
output.putNextEntry {
val e = createEntry(file, name)
e.setTime(time.getOrElse(IO.getModifiedTimeOrZero(file)))
e
}
IO.transfer(file, output)
output.closeEntry()
}
// Calculate directories and add them to the generated Zip
allDirectoryPaths(files).foreach(addDirectoryEntry(null, _))
// Add all files to the generated Zip
files foreach { case (file, name) => addFileEntry(file, name) }
}
private def normalizeToSlash(name: String): String = {
val sep = java.io.File.separatorChar
if (sep == '/') name else name.replace(sep, '/')
}
private def relativeComponents(path: String): List[String] =
path.split("/").toList.dropRight(1)
private def directories(path: List[String]): List[String] =
path.foldLeft(List(""))((e, l) => (e.head + l + "/") :: e)
private def directoryPaths(path: String): List[String] =
directories(relativeComponents(path)).filter(_.length > 1)
private def allDirectoryPaths(files: Iterable[(File, String)]) =
TreeSet[String]() ++ (files flatMap { case (_, name) => directoryPaths(name) })
}
Usage
sbt springBootJar or springBootJar in sbt console.
Conclusion
This was tested with scala 3.2.1 in project and sbt 1.7.1. I don't have time to make it into a plugin, and I don't mind if anyone reuse my findings to make more general and ready to use solution.