In project on which I work (based on gradle) there is one very big module (gradle subproject).
During build on CI two tasks from this subproject are executed sequentially and it leads to significant execution time.
The project uses org.gradle.parallel=true, but when I created simple project to check how independent tasks from same subproject are executed with this property I found out that tasks are executed sequentially.
My question:
Is possible to execute two independent tasks from same gradle subproject in parallel to shorten theirs execution time? (Assuming that they doesn't produce output in same place and don't use any shared state)
From the documentation (see Parallel execution):
Most builds consist of more than one project and some of those projects are usually independent of one another. Yet Gradle will only run one task at a time by default, regardless of the project structure (this will be improved soon). By using the --parallel switch, you can force Gradle to execute tasks in parallel as long as those tasks are in different projects.
I think the most important part here is "as long as those tasks are in different projects": if your two long-running tasks belong to the same subproject you won't be able to make them executed in parallel (not in current Gradle version)
Related
I have an application for which the testing is quite extensive. Essentially, we must run the application a few hundred thousand time on different input. So I have built a custom Gradle task which manages forking processes and reaping finished processes. Each of the thousands of test runs generate a file that goes in a results directory. Full testing can take about a week when distributed across 10 cluster nodes.
The issue is, if I need to stop the testing for whatever reason, then there is currently no way for me to start back up where I left off. Gradle's incremental build and caching features (to my understanding) really only work when tasks finish, and it will just rerun the entire task from scratch if the previous invocation was interrupted (ctrl-c).
I could build in some detection of the results files and only rerun segments for which there is no results file. However, this will not work properly when the application is rebuilt, and then testing legitimately must start from scratch.
So how can I reliably detect which testing segments are up to date when the previous task invocation was interrupted?
Annotated tasks
For any Gradle task, if its output files exist and its inputs (including predecessor tasks) are all up-to-date, Gradle will treat that task as up-to-date and not run it again. You tell Gradle about inputs and outputs by annotating properties of the class you write to define the task.
You can make use of this by breaking your custom Gradle testing task into a number of smaller test tasks and have each of those task definitions declare annotated outputs. The test reports are probably the most suitable for those outputs. Then the test tasks which have a report will not have to be re-run if you stop the build halfway.
A whole application rebuild will always need all tests to be re-run
However, if your whole application is rebuilt then those test tasks will no longer be up-to-date as their predecessor build tasks will not be up-to-date. This makes sense of course: a new application build needs its tests to be run again to check it still works as intended.
Multimodule builds may mean only part of an application needs rebuilding
It may be that there are parts of the application that are not being rebuilt, and test tasks that depend solely on those intact parts of the application. If the chain of predecessor tasks for any previously completed test task are all up-to-date, then Gradle will not re-run those tests again either.
This would be more likely to be the case for more test tasks if your application, if appropriate, is separated into different Gradle subprojects in a multimodule build. Each then would have its own chain of tasks which may not have to be re-run if only part of the application's code or other inputs is changed.
I have a multi-module Maven project that consists of ~30 modules. 4 of these modules contains a large number of junit tests in separate testcases, so when I'm running tests all modules except these are executed quickly, but each of these modules running separately could take several minutes. What I want to achieve is
When project is compiled I want to use as much threads as CPUs available (-T1C)
When tests are running I want to use as much threads as CPUs available, and I want to run testcases in all modules as parallel as possible, but not more than number of CPUs. If I just run tests with -T1C it assign thread per module, so tests inside long-running modules are not running in parallel. But if I set forkCount=1C in long-running modules each of them occupies as many threads as CPUs are, so when all of them are running CPUs are overused. I know that I can set up something like forkCount=0.5C but in this case it will be required to manually adjust these numbers.
So is it a way to specify some build-level limitation having everything else as parallel as possible?
Thanks.
How can I execute several Maven goals on the same project concurrently? My exact scenario is after having installed sevaral gigabytes of artifacts locally I would start deployment to Nexus, but since it will take a while, I would also like to start executing additional goals with exec plugin - the two tasks are not generally dependent on one another, but they add up to a significant time waste.
I was hoping Maven has any sort of idiom for that, but other solutions are welcome. I would need a way to fail the build if any of the tasks fails (bonus points for a solution that would notify the other goal to fail fast).
In project on which I work (based on gradle) there is one very big module (gradle subproject).
During build on CI two tasks from this subproject are executed sequentially and it leads to significant execution time.
The project uses org.gradle.parallel=true, but when I created simple project to check how independent tasks from same subproject are executed with this property I found out that tasks are executed sequentially.
My question:
Is possible to execute two independent tasks from same gradle subproject in parallel to shorten theirs execution time? (Assuming that they doesn't produce output in same place and don't use any shared state)
From the documentation (see Parallel execution):
Most builds consist of more than one project and some of those projects are usually independent of one another. Yet Gradle will only run one task at a time by default, regardless of the project structure (this will be improved soon). By using the --parallel switch, you can force Gradle to execute tasks in parallel as long as those tasks are in different projects.
I think the most important part here is "as long as those tasks are in different projects": if your two long-running tasks belong to the same subproject you won't be able to make them executed in parallel (not in current Gradle version)
I've got custom tasks in build.gradle file. I need to periodically recompile sources. The idea is to call tasks.compileJava.execute() in infinite looping custom task. The problem is that (as I understand) task will be executed only once (it doesn't depends on task type/inputs/outputs - even custom tasks will be executed only once).
How can I force gradle to execute task more than once (check inputs/outputs and mark it as UP-TO-DATE if it make sence)?
A task cannot be executed more than once, and Task.execute() should never be called from user code (not even once). Continuous task execution is a planned feature and will require changes to the Gradle codebase.