Gradle doesn't collect strings starting with -D - spring-boot

Is that a Gradle or Groovy bug?
I want to pass JVM parameters from Gradle to forked JVM, which is unfortunately not done automatically. This is supposed to work, build.gradle:
...
bootRun {
jvmArgs = System.properties.iterator().findAll{it.key.startsWith('myapp')}.collect {
"-D${it.key}=${it.value}"}
}
...
It is executed as:
gradle bootRun -Dmyapp.port=34501 -Dmyapp.member.name=server1
The method collect always return empty collecting if string starts with -D. If it starts with anything else it returns expected two element String collection. If I put space before -D it also works however it breaks the build further downstream on :findMainClass misinterpreting -Dmyapp.port=... with main class name. It simply has to start with -D.
I also tried different string concatenation but as far as the result is a string starting with -D it doesn't work.
Is it a bug or I'm missing something. This is my first Gradle project and I'm not a Groovy developer.
Should I report is bug? Where, Groovy or Gradle?
Notes:
I'm running Gradle from IntelliJ IDE 2016.1.2
Using Gradle 3.5
Forked JVM runs Spring Boot application
UPDATE
Big apologies, my bad! The truth is, the JVM parameters are passed down using the formula above; the problem is with how I measured it that the weren't. I simply put printouts:
println "jvmArgs: ${jvmArgs}"
println "jvmArgs.size: ${jvmArgs.size}"
println "jvmArgs.class: ${jvmArgs.class}"
..and aborting bootRun if jvmArgs.size == 0, to avoid slow application start; that is I wasn't really checking if parameters were passed or not in the application itself. And it turned out they were.
FYI the outputs were:
jvmArgs: []
jvmArgs.size: 0
jvmArgs.class: java.lang.ArrayList
The class of jvmArgs is reported as a standard ArrayList, but behaves more like a input stream consumer, whatever array is jvmArgs assigned to, that array is scanned for all strings starting with "-D", those are consumed (by what?), passed to some ProcessBuilder (??) and jvmArgs is left only with remaining elements.
Take this example:
jvmArgs = ["-Daaa=bbb", "foo", "bar"]
jvmArgs = ["stuff", "-Dccc=ddd", "morestuff"]
jvmArgs = ["-Deee=fff"]
println "jvmArgs: ${jvmArgs}"
..it prints jvmArgs: [] and Spring Boot application is launched with -Daaa=bbb -Dccc=ddd -Deee=fff.
Can someone explain what causes this magic stream like property of jvmArgs, which otherwise claims to be a simple ListArray?

This works for me, but I don't have an explanation for the observed behavior. Hope it helps anyway.
def array = System.properties.iterator().findAll{
it.key.startsWith('myapp')
}.collect {
"-D${it.key}=${it.value}"
}
jvmArgs.addAll(array)
EDIT
jvmArgs = ["value"] calls setJvmArgs which, if I haven't missed something, goes from JavaExec to JavaExecHandleBuilder and later JvmOptions. Here, some parameters get removed. Entries beginning with -D gets added to systemproperties instead.
setJvmArgs( ["-Dtest=1", "xx"])
println getJvmArgs() //[xx]
println systemProperties //[test:1]
Does your Application does't have access to that properties?
https://github.com/gradle/gradle/blob/master/subprojects/core/src/main/java/org/gradle/process/internal/JvmOptions.java#L183
EDIT: what's happening in the background
In Groovy, a property assignment calls the setter instead, accessing it will call the getter. They are interchangeably. If you omit the setter and getter pair, it will be generated for you and will be visible in the bytecode. But you can even omit the property itself, only write the getter and setter pair and use it as a property.
class Foo {
def setBar(String foo) {println "no thanks"}
String getBar() {"test"}
}
f = new Foo()
f.bar="write Var" // println "no thanks"
println f.bar instanceof String // --> f.getBar() inst... true
println f.bar //
So you never assigned a List to a variable, but called setJvmArgs(List). You can list all args with getAllJvmArgs() btw.
In combination with delegation strategies and dynamic Properties/Methods, this can be a blessing for DSL programming, but a curse to debug...
http://groovy-lang.org/style-guide.html#_getters_and_setters
and google for groovy propertyMissing/groovy metaprogramming/groovy Resolve Strategies if you like to learn more about this topic.

Related

How can I create an incremental build to derive output filenames from input filenames?

I must write a plugin to compile files of a certain type not covered by an existing plugin.
My initial requirements are simply that if the input file is changed or if the output is missing, then the task will not be up-to-date. In other words, I want the core "working with files" paradigm to work for me.
My first attempt was to declare both inputs and outputs to see if the project would work as most would expect.
class TestPlugin implements Plugin<Project> {
void apply (Project project) {
project.task('compile') {
doFirst {
inputs.files.eachWithIndex { inFilename, idx ->
def inFile = project.file(inFilename)
def outFilename = outputs.files[idx]
def outFile = project.file(outFilename)
logger.info "converting ${inFile} to ${outFile}"
outFile.append "something"
}
}
}
}
}
apply plugin: TestPlugin
compile {
inputs.file "test.in"
outputs.file "test.out"
}
And it does. At this point it does all I need except for one thing: I have to define outputs in correlation with inputs. But defining the outputs is complicated enough to warrant factoring that part of the code into the plugin's task.
My next attempt was to try to get the task to define its outputs but it fails because when the "populate outputs" code executes, inputs is empty and so no outputs are added.
class TestPlugin implements Plugin<Project> {
void apply (Project project) {
project.task('compile') {
inputs.files.each { outputs.files.add(it.replaceFirst(~/\.in$/, '.out')) }
doFirst {
inputs.files.eachWithIndex { inFilename, idx ->
def inFile = project.file(inFilename)
def outFilename = outputs.files[idx]
def outFile = project.file(outFilename)
logger.info "converting ${inFile} to ${outFile}"
outFile.append "something"
}
}
}
}
}
apply plugin: TestPlugin
compile { inputs.file "test.in" }
The above fails with a "path may not be null..." error caused by indexing an empty outputs list (because inputs is empty at the time the task's outer block iterates over inputs.
I tried populating outputs in Project.afterEvaluate and Project.beforeEvaluate but neither worked. The beforeEvaluate closure never executed. The afterEvaluate closure executed, well, after the project was evaluated which means it executed after the task was set as either up-to-date or out-of-date and so it's not useful for what I need.
I also tried Project.configure but that didn't work either.
Then I discovered lazy configuration but since it's incubating I think I should avoid it. I'm assuming that the Java and C plugins don't use incubating features and that leaves me wondering how those plugins are accomplishing the equivalent of lazy configuration.
Then I found Gradle plugin for custom language but it doesn't have an answer. It does, however, have a comment leading me to look at a lot of gradle source code. Again, I don't think I should have to reinvent the wheel.
Lastly, if I have to push the decision to compile into doFirst and not bother with declaring outputs (and thereby abandon task up-to-datedness), then so be it. I just need to know and move on.
For more context, I'm migrating a build away from ant. The two udemy.com classes I took helped me a lot but didn't go far enough to lead me in a confident direction to solve the stated problem. I asked a similar question to one of the instructors and its community to no avail.

How can I define a global variable or constant using jenkins pipeline shared library?

While I was able to define methods, defining variables using Jenkins Shared library seems to an unresolved quest so far.
I added /vars/True.groovy with this body
def call() {
return true
}
And now inside Jenkinsfile, I tried to test if it works as expected:
println "evaluation ${ true == True }"
But surprise, this fails as it considers true != class True. My impression is that true == True() may work but that's not the point, I need a real variable because that whole purpose was to avoid some errors caused by people using wrong case.
So what is the magic trick?
I've found a way to achieve this, but with a slight caveat: the variables must be defined/wrapped within a class. However, this does have the upside of providing better organization, in order not to pollute the global space too much.
For example, we often reuse four standard "magic strings" for build statuses, which I wanted to save as global constants to facilitate interoperability. So I created a global status class, defined in vars/status.groovy:
class status {
final String STARTED = "STARTED"
final String SUCCESS = "SUCCESS"
final String FAILURE = "FAILURE"
final String ABORTED = "ABORTED"
}
The constants can then be used by referring to their parent class:
echo status.STARTED
echo status.SUCCESS
echo status.FAILURE
echo status.ABORTED
Specials thanks to #mkobit for pointing me in the right direction!
It looks like Global Variables defined in the vars directory must be lower/camel/maybe some other special casing. This isn't stated anywhere in on the Defining global variables section, but there is this note at the top:
The vars directory hosts scripts that define global variables accessible from Pipeline. The basename of each *.groovy file should be a Groovy (~ Java) identifier, conventionally camelCased. The matching *.txt, if present, can contain documentation, processed through the system’s configured markup formatter (so may really be HTML, Markdown, etc., though the txt extension is required).
Here is what I tried:
vars/MyTrue.groovy
class MyTrue implements Serializable {
}
vars/myTrue.groovy
class myTrue implements Serializable {
}
vars/mytrue.groovy
class mytrue implements Serializable {
}
vars/doesCasingMatter.groovy
class DoesCasingMatter implements Serializable {
}
And in my pipeline script to test if they are instances or Class types (no script security enabled here):
echo("MyTrue: ${Class.isInstance(MyTrue)}")
echo("myTrue: ${Class.isInstance(myTrue)}")
echo("mytrue: ${Class.isInstance(mytrue)}")
echo("What bindings are there?: ${binding.variables}")
This prints out:
[Pipeline] echo
MyTrue: true
[Pipeline] echo
myTrue: false
[Pipeline] echo
mytrue: false
[Pipeline] echo
What bindings are there?: [steps:org.jenkinsci.plugins.workflow.cps.DSL#e96256, myTrue:myTrue#8a1ddc5, mytrue:mytrue#392ff649]
Which seems to indicate that something about the class name determines how it gets compiled and created. The first example, which is similar to your vars/True.groovy, is only imported as a class and not instantiated. The other two are compiled and instantiated and are bound to the script with their defined class names. I think you will have to define your classes differently if you want them to be global variables.
Variable True is definitely a variable but it holds a reference to object of type True(that you defined in /vars/True.groovy). You have two options
The good one:
Use it this way
println "evaluation ${ true == True() }"
The strange one:
You can override equals() method in /vars/True.groovy
public boolean equals(obj) {
return obj == true;
}
Then this should work
println "evaluation ${ true == True }"
But it would be really strange and can lead to misunderstandings.

Why are the configuration styles inconsistent in gradle?

I am new to gradle and a few things of gradle confuses me. Some things appear like inconsistent coding / configuration style.
For example, when we configure the repository to be jcenter or mavencentral we call a function / method e.g. jcenter.
repositories {
jcenter()
}
However, in the same file, when we try to configure a dependency we do not call functions / methods anymore.
dependencies {
classpath 'com.android.tools.build:gradle:2.3.1'
}
And then there are clearly variables getting values
productFlavors {
prod {
versionName = "1.0-paid"
}
mock {
versionName = "1.0-free"
}
}
I am sure there is a reason behind this perceived inconcistency but could not find anything when I read through the documentation. Could anybody explain the reason?
Actually these examples are not so different.
classpath 'com.android.tools.build:gradle:2.3.1'
is a function call as well. Groovy (the language in which gradle build scripts are written) allows you to leave out the parenthesis around the arguments in many cases.
This is the flexibility (I prefer this to inconsistency) that is delivered by Groovy the language that Gradle is using.
In Groovy you can call a function/method with or without the parenthesis if its name is followed by matching arguments but if there are no arguments you must add parenthesis to make it a call to a function and make it distinct from the closure it represents.
Here is an example using groovysh
groovy:000> def a(){println "a"}
===> true
groovy:000> a
===> org.codehaus.groovy.runtime.MethodClosure#95e33cc
groovy:000> a()
a
===> null
groovy:000> def b(arg){println arg}
===> true
groovy:000> b
===> org.codehaus.groovy.runtime.MethodClosure#d771cc9
groovy:000> b "argument"
argument
===> null
groovy:000> b("argument")
argument
===> null
groovy:000>

Closure defined in root not visible in child

I have root project and subproject (:child).
Root build looks like like this:
def foo = {
println("foo")
}
allprojects {
task bar << {
println(project.name + ":bar")
}
afterEvaluate {
foo()
}
}
Running gradle bar prints:
foo
foo
:bar
:child:bar
child:bar
parent:bar
This make sense. However, IRL I need foo to be called by the child's build file (because I want it to be called only by some of the submodules).
The documentation seems to be clear enough: In a multi-project build, sub-projects inherit the properties and methods of their parent project
However, moving the "afterEvaluate" block above into child/build.gradle results in an error: Could not find method foo() for arguments [] on project ':child' of type org.gradle.api.Project.
Why does this happen and how do I fix this? I have tried a whole bunch of different variations - moving the def around (to buildscript, allprojects, to ext, to allprojects.ext, making it a variable in ext, instead of a method etc.), referring to it differently (as rootProject.foo, rootProject.foo(), ext.foo() etc.) - nothing seems to work.
Any ideas?
Vars need to be declared in the ext namespace for them to be propagated downstream. Try:
ext.foo = {
println("foo")
}
ref: https://docs.gradle.org/current/dsl/org.gradle.api.plugins.ExtraPropertiesExtension.html

Gradle looping inside ProcessResources and filter

In a jar task, I want to replace some texts in a conf file.
jar {
ext {
excludedClasses = ["com.MyClass1", "com.MyClass2"]
}
doFirst {
println 'Jar task started execution'
println 'Excluded classes ' + excludedClasses
exclude(excludedClasses)
}
doLast {
println 'Jar task finished execution'
}
processResources {
filesMatching('**/moduleconfiguration/conf.json') { f ->
excludedClasses.each { c ->
filter {
println it
it.replace(c, "com.MyClass3")
}
}
}
}
}
But the above code tries to replace c from all *.class files, resulting in an illegal jar. I want it to make replacements only in '**/moduleconfiguration/conf.json' file.
How can I achieve that?
UPDATE
Looks like I am suffering from the same problem happening here: https://issues.gradle.org/browse/GRADLE-1566. This issue has already been resolved but reoccurs if I use an each loop inside processResources.
Meanwhile, I have found 2 solutions to my problem as follows:
Solution 1: Changing order of filter and each loop. i.e. Looping inside filter
filesMatching('**/moduleconfiguration/conf.json') { f ->
filter {
excludedClasses.each { c ->
println it
it = it.replace(c, "com.MyClass3")
}
it
}
}
Solution 2: Using regex instead of each loop
filesMatching('**/moduleconfiguration/conf.json') { f ->
filter {
println it
def regex = excludedClasses.join("|") // watch for .(dot) or other regex chars here
it.replaceAll(regex, "com.MyClass3")
}
}
I am still wondering why the scope of filtering changes to all files if I use each loop within the filesMatching method closure. Is this a groovy thing or gradle thing? I would be very thankful if someone could explain what is happening there.
UPDATE 2
println output of values of delegate, this and owner at different positions for problematic case:
:processResources
Inside filesMatching. delegate:file '.../configuration/conf.json' this:root project 'projectName' owner:build_95q5jrf5z5ao0hk03tsevn2t0$_run_closure10_closure90#6f3e18b8
Problematic Case inside loop before filter. delegate:build_95q5jrf5z5ao0hk03tsevn2t0$_run_closure10_closure90_closure91#4587ec31 this:root project 'projectName' owner:build_95q5jrf5z5ao0hk03tsevn2t0$_run_closure10_closure90_closure91#4587ec31
Problematic Case inside loop before filter. delegate:build_95q5jrf5z5ao0hk03tsevn2t0$_run_closure10_closure90_closure91#4587ec31 this:root project 'projectName' owner:build_95q5jrf5z5ao0hk03tsevn2t0$_run_closure10_closure90_closure91#4587ec31
:classes
:jar
Jar task started execution
Excluded classes [MyClass1.class, MyClass2.class]
Problematic Case inside loop inside filter. delegate:build_95q5jrf5z5ao0hk03tsevn2t0$_run_closure10_closure90_closure91_closure92#3a0d0128 this:root project 'projectName' owner:build_95q5jrf5z5ao0hk03tsevn2t0$_run_closure10_closure90_closure91_closure92#3a0d0128
.
.
.
.
.
println output of values of delegate, this and owner at different positions for solution 1:
:processResources
Inside filesMatching. delegate:file '.../configuration/conf.json' this:root project 'projectName' owner:build_95q5jrf5z5ao0hk03tsevn2t0$_run_closure10_closure90#6ece61a3
Solution 1 Inside filter before loop. delegate:build_95q5jrf5z5ao0hk03tsevn2t0$_run_closure10_closure90_closure91#64af2ad7 this:root project 'projectName' owner:build_95q5jrf5z5ao0hk03tsevn2t0$_run_closure10_closure90_closure91#64af2ad7
Solution 1 Inside filter inside loop. delegate:build_95q5jrf5z5ao0hk03tsevn2t0$_run_closure10_closure90_closure91_closure92#22c74276 this:root project 'projectName' owner:build_95q5jrf5z5ao0hk03tsevn2t0$_run_closure10_closure90_closure91_closure92#22c74276
Solution 1 Inside filter inside loop. delegate:build_95q5jrf5z5ao0hk03tsevn2t0$_run_closure10_closure90_closure91_closure92#22c74276 this:root project 'projectName' owner:build_95q5jrf5z5ao0hk03tsevn2t0$_run_closure10_closure90_closure91_closure92#22c74276
.
.
.
.
.
:classes
:jar
Jar task started execution
Excluded classes [MyClass1.class, MyClass2.class]
Update
Based on your second update and some testing on my side, it doesn't seem to directly be what I originally suggested. It does definitely appear to be something with delegation, but I can't pin it down.
You can illustrate the difference in your original (problematic) example by changing the filter { line to f.filter {. This explicitly tries to execute the FileCopyDetails#filter method instead of whichever happens to be in scope.
You should be able to see that when calling f.filter only the matched file is filtered. When you call filter on its own, it is calling the task level one. However, since you are already in the middle of copying files, it only applies the filter to files that occur alphabetically after the first one that matches.
For example, if you have this folder structure:
+ resources/main
- abc.json
- configuration.json
- def.json
- efg.json
The def.json and efg.json will be filtered, but the first two will not.
This still doesn't answer why it doesn't call the correct filter method, but it at least confirms that it is calling the task level one.
Original Answer
I believe this is due to the Groovy closures delegating to different objects. I believe you are actually calling Copy#filter in the original (problematic) case, and FileCopyDetails#filter in the two solutions. The copy task's filter method will apply a filter to everything, whereas the details filter method will be for the specific file in your filesMatching.
You should be able to illustrate the different delegates by printing them out:
Original
processResources {
filesMatching('**/moduleconfiguration/conf.json') { f ->
excludedClasses.each { c ->
// print the class of this closure's delegate
println delegate.class
filter {
it.replace(c, "com.MyClass3")
}
}
}
}
Solution 1
filesMatching('**/moduleconfiguration/conf.json') { f ->
filter {
// print the class of this closure's delegate
println delegate.class
excludedClasses.each { c ->
println it
it = it.replace(c, "com.MyClass3")
}
it
}
}
My expectation is that you will see the original delegating to your Copy task (i.e. processResources) and the solutions delegating to the FileCopyDetails. By default all closures in Groovy are "owned" by the object in which they were declared, and also will "delegate" to the owner. When Gradle has API's that take a closure as an argument (such as the filter methods), they are usually reconfigured to have a specific delegate (usually the enclosing container object) and to use the "delegate first" strategy to lookup methods.
See Groovy's documentation on delegation strategies.

Resources