I've got custom tasks in build.gradle file. I need to periodically recompile sources. The idea is to call tasks.compileJava.execute() in infinite looping custom task. The problem is that (as I understand) task will be executed only once (it doesn't depends on task type/inputs/outputs - even custom tasks will be executed only once).
How can I force gradle to execute task more than once (check inputs/outputs and mark it as UP-TO-DATE if it make sence)?
A task cannot be executed more than once, and Task.execute() should never be called from user code (not even once). Continuous task execution is a planned feature and will require changes to the Gradle codebase.
Related
I have an application for which the testing is quite extensive. Essentially, we must run the application a few hundred thousand time on different input. So I have built a custom Gradle task which manages forking processes and reaping finished processes. Each of the thousands of test runs generate a file that goes in a results directory. Full testing can take about a week when distributed across 10 cluster nodes.
The issue is, if I need to stop the testing for whatever reason, then there is currently no way for me to start back up where I left off. Gradle's incremental build and caching features (to my understanding) really only work when tasks finish, and it will just rerun the entire task from scratch if the previous invocation was interrupted (ctrl-c).
I could build in some detection of the results files and only rerun segments for which there is no results file. However, this will not work properly when the application is rebuilt, and then testing legitimately must start from scratch.
So how can I reliably detect which testing segments are up to date when the previous task invocation was interrupted?
Annotated tasks
For any Gradle task, if its output files exist and its inputs (including predecessor tasks) are all up-to-date, Gradle will treat that task as up-to-date and not run it again. You tell Gradle about inputs and outputs by annotating properties of the class you write to define the task.
You can make use of this by breaking your custom Gradle testing task into a number of smaller test tasks and have each of those task definitions declare annotated outputs. The test reports are probably the most suitable for those outputs. Then the test tasks which have a report will not have to be re-run if you stop the build halfway.
A whole application rebuild will always need all tests to be re-run
However, if your whole application is rebuilt then those test tasks will no longer be up-to-date as their predecessor build tasks will not be up-to-date. This makes sense of course: a new application build needs its tests to be run again to check it still works as intended.
Multimodule builds may mean only part of an application needs rebuilding
It may be that there are parts of the application that are not being rebuilt, and test tasks that depend solely on those intact parts of the application. If the chain of predecessor tasks for any previously completed test task are all up-to-date, then Gradle will not re-run those tests again either.
This would be more likely to be the case for more test tasks if your application, if appropriate, is separated into different Gradle subprojects in a multimodule build. Each then would have its own chain of tasks which may not have to be re-run if only part of the application's code or other inputs is changed.
I'm looking for a way to disable most of my sub project processing stack if I know it's already built and published. I'm managing a project with 400 sub projects, and I can shave tones of time by checking if I really need to build a sub project or not.
My attempt to do this end I have a block inside a task that calls
task disablePublish{
if (the resultant product is already published.)
{
project.tasks.all {
task -> task.enabled = false
}
}
}
...
publish.dependsOn += [disablePublish]
When I call publish, this seems to work great, if the rpm is already published, it doesn't publish, and builds it and publishes if it's not.
However it seems to break building if I don't call publish, if the rpm is published.
Skipping task 'mytask' as task onlyIf is false.
And no rpm gets built.
if I comment out the
project.tasks.all {
task -> task.enabled = false
}
build works great.
So I'm not sure if others have a great solution to this design or ideas to try.
Thank you for your time and consideration.
I'm looking for a way to disable most of my sub project processing stack if I know it's already built [and published].
Take a look at incremental builds. Gradle can determine whether a task needs to be run or not, if the inputs and outputs of the task are properly defined. This should work without a problem for tasks that build the project. Checking whether a project was already published (and thus a build is not required) may be difficult, as it would be hard to define a task output.
if I really need to build a sub project or not
This actually indicates that you should not solve your problem on the task level, but rather on the project level. Are there any dependencies between the projects or are they just part of the same multi-project build because it is handy to run a single command for all projects? Take a look at the difference between multi-project builds and composite builds.
My attempt to do this end I have a block inside a task that calls
This is not completely true. There is a task and there is a code block, but they are not connected in the way you think they are. Actually, your code would work in the same way if it would look like this:
if (the resultant product is already published.)
{
project.tasks.all {
task -> task.enabled = false
}
}
task disablePublish {
}
Why is this the case? Well, whatever code is put inside the closure when a task is created does not define what happens when the task is executed. Instead, the code is run directly to configure the task. Only (internal) task actions, doFirst and doLast closures are run when the task gets executed (either because it was mentioned on the command line or because of dependsOn):
println 'Before creating the task'
task myTask {
println 'Task is configured'
doLast {
println 'Task is executed'
}
}
println 'After creating the task'
Just put the code above in a build.gradle file and run gradle myTask from the same directory. The statement 'Task is executed' will be printed last. Now just call gradle without any task defined on the command line. Gradle will print all statements beside 'Task is executed', showing that each task is configured on every invocation of gradle.
This shows that your 'task' logic gets applied every time Gradle is run. The task disablePublish however remains empty and does nothing at all.
When I call publish, this seems to work great, if the rpm is already published, it doesn't publish, and builds it and publishes if it's not.
Well, that depends on the condition of the if block, as this condition basically enables or disables your whole project. Of course, if the condition is false, the project will work completely normal, if the condition is true, no task will be executed at all.
However it seems to break building if I don't call publish, if the rpm is published.
If it is published (and your if condition confirms this), your whole project won't do anything, as every task is disabled. Since this logic gets applied regardless of any task, this has nothing to do with the publish task.
In project on which I work (based on gradle) there is one very big module (gradle subproject).
During build on CI two tasks from this subproject are executed sequentially and it leads to significant execution time.
The project uses org.gradle.parallel=true, but when I created simple project to check how independent tasks from same subproject are executed with this property I found out that tasks are executed sequentially.
My question:
Is possible to execute two independent tasks from same gradle subproject in parallel to shorten theirs execution time? (Assuming that they doesn't produce output in same place and don't use any shared state)
From the documentation (see Parallel execution):
Most builds consist of more than one project and some of those projects are usually independent of one another. Yet Gradle will only run one task at a time by default, regardless of the project structure (this will be improved soon). By using the --parallel switch, you can force Gradle to execute tasks in parallel as long as those tasks are in different projects.
I think the most important part here is "as long as those tasks are in different projects": if your two long-running tasks belong to the same subproject you won't be able to make them executed in parallel (not in current Gradle version)
In project on which I work (based on gradle) there is one very big module (gradle subproject).
During build on CI two tasks from this subproject are executed sequentially and it leads to significant execution time.
The project uses org.gradle.parallel=true, but when I created simple project to check how independent tasks from same subproject are executed with this property I found out that tasks are executed sequentially.
My question:
Is possible to execute two independent tasks from same gradle subproject in parallel to shorten theirs execution time? (Assuming that they doesn't produce output in same place and don't use any shared state)
From the documentation (see Parallel execution):
Most builds consist of more than one project and some of those projects are usually independent of one another. Yet Gradle will only run one task at a time by default, regardless of the project structure (this will be improved soon). By using the --parallel switch, you can force Gradle to execute tasks in parallel as long as those tasks are in different projects.
I think the most important part here is "as long as those tasks are in different projects": if your two long-running tasks belong to the same subproject you won't be able to make them executed in parallel (not in current Gradle version)
Is there a way to do an up-to-date check of the tasks only, without executing the tasks that are not up-to-date? The motivation behind this is debugging a script with a lot of lengthy tasks and a complicated task tree.
Checking gradle documentation and code there is no such option. Thinking about it this makes a lot of sense as gradle up-to-date logic is being executed during the execution phase as its result may be dependent on the actual outputs of the previous tasks. In other words, in order to know whether a task is up-to-date all its task dependencies has to be either executed or resolved as UP-TO-DATE, thus your ask is not possible.