This is not an exactly Gradle problem. But a more general build question. We have some projects structured as the following:
-- A (by team 1)
-- build.gradle
-- settings.gradle
-- B (by team 1,3)
--build.gradle
-- C (by team 2)
-- D (by team 3)
They are separate CVS modules. A, B and D are java projects and managed by two teams (team 1,3). C is an C++ project managed by another team (2). B use C via JNI. (The fact that C is C++ is not that important, the key is that it is developed and managed by another team.) There are two applications, A is the entrance of the application 1, A->B->C; while D is the entry point of the application 2, D->B->C. The development often involves change in all three levels. All three teams sit together and communicate constantly. In practice, we (team 1) might need some changes for application 1 in C; team 2 works on C and gives us a temporary copy; we might need some further changes after integration. We will go back and forth for several rounds for one problem. Similarly for application 2.
Currently, A,B and D are managed by various ant scripts and C by make. We started to explore new tools, in particular, Gradle. In our first cut, A includes B as a sub-project (in the Gradle sense) and they always are built and published together. We also always use the head of C (compiled ourselves from source in windows or grabed the latest Jenkin build) and when we are happy with the build, we tag all three projects. We recently adopt an internal Artifactory repository. We are thinking about how to mange the dependency and versioning. Once we finished, we will introduce it to team 3 for module D as well.
We can try to include C as a subproject for A and then always build from the scratch for all three. Similarly include C and B as subprojects for D. The application name can be in the version name for example.
alternatively we can always depends on a fixed version of C in the repository.
In 2, we cannot totally depends on the head/snapshot of C because that might involve their development and might be unstable. But we need their change so frequent that it seems inpractical to put a new version for every C changes.
We are wondering what is the best practice for such a setup? A quick search in internet does not yield much results. Can anyone give us some advices or point me to some books or documents?
Thank you so much.
As I can read from the whole description it seems that during development time (and in production also I suppose) all 3 projects are really tight coupled. So I'll use the first scenario for convenience of work. I'll also build all the projects together, tagged them together and keep same versioning pattern - in general even if it's separated this is a single project.
The first scenario can be carried on till no 3rd party (external) application/library/team uses any of the projects (I suppose it could only be C project). Then backward compatibility issues/pull requests and other may happen and the mentioned projects should be separated. If there's no chance that such situation takes place You shouldn't bother too much. But remember about good (semantic) versioning, and tagging the repository to be always sure which version is deployed to which environment.
UPDATE
After question update.
If You've dependency paths like A->B->C and D->B->C I'll reorganize the project structure. B should become a subproject of both A and D or maybe not a subproject but an external library that is added to these projects. B and C should be a separate project with B dependent on C.
All three projects (A,D, B with C) should be separately versioned (especially B with C) because this is a common part for clients (A and D).
Now, for a convenient development You can add to A and D snapshot libs that will be built often on CI server and then update to artifactory. This will allow You to introduce changes to B and have them visible fast in the projects A and D. But stable release of B (and hence C) should be maintained totally separately.
I hope I understood the problem well and helped You a bit.
P.S. Please consider also using gitflow - a branching model. Then You can used SNAPSHOT versions in dev branch and a stable version of B in release.
Related
I have been educating myself about monorepos as I believe it is a great solution for my team and the current state of our projects. We have multiple web products (Client portal, Internal Portal, API, Core shared code).
Where I am struggling to find the answer that I want to find is versioning.
What is the versioning strategy when all of your projects and products are inside a monorepo?
1 version fits all?
Git sub-modules with independent versioning (kind of breaks the point of having a mono repo)
Other strategy?
And from a CI perspective, when you commit something in project A, should you launch the whole suite of tests in all of the projects to make sure that nothing broke, even though there was no necessarily a change made to a dependency/share module?
What is the versioning strategy when all of your projects and products are inside a monorepo?
I would suggest that one version fits all for the following reasons:
When releasing your products you can tag the entire branch as release-x.x.x for example. If bugs come up you wouldn't need to check "which version was of XXX was YYY using"
It also makes it easier to force that version x.x.x of XXX uses version x.x.x of YYY. In essence, keeping your projects in sync. How you go about this of course depends on what technology your projects are written in.
And from a CI perspective, when you commit something in project A, should you launch the whole suite of tests in all of the projects to make sure that nothing broke, even though there was no necessarily a change made to a dependency/share module?
If the tests don't take particularly long to execute, no harm can come from this. I would definitely recommend this. The more often your tests run the sooner you could uncover time dependent or environment dependent bugs.
If you do not want to run tests all the time for whatever reason, you could query your VCS and write a script which conditionally triggers tests depending on what has changed. This relies heavily on integration between your VCS and your CI server.
I am using TFS 2015 to perform automated builds of my libraries in a cross-platform environment. I just added support for building my libraries on macOS but I am unable to figure out how to order the builds. Here is my situation:
I have libraries A, B, and C (in separate build definitions). Library B depends on library A, and library C depends on library B. Libraries A, B, and C have a small bit of overlap in terms of shared files in TFS, so when a user checks in files from these overlapped directories, all of the libraries are built, but in random order. I need to be able to build library A first, then library B, then library C.
Any help in configuring this (in a way other than creating a single build definition)?
Unfortunately, there is no such feature to control the order of vNext build and move items up / down in the queue.
There is no logical order and without any configuration to control this.
I have created a uservoice for you, you could vote up and monitor it. TFS PM and Admin will kindly review the suggestion.
Order TFS vNext Builds
https://visualstudio.uservoice.com/forums/330519-team-services/suggestions/19960372-order-tfs-vnext-builds
For now you have to use the workaround, creating a new single build definition to combine them.
I thought I understood the concept of components in ClearCase UCM after reading all the posts on here, however, I'm pretty sure I'm missing something.
I created two projects A and B that each contain a component, Comp_A and Comp_B respectively, and they also each contain a shared component Comp_C as well as a rootless component Comp_D to contain the composite baselines.
Everything works fine. However, how does this prevent duplication of code? When making a change to the shared component in Project A, that change needs to be delivered (using an inter-project delivery of a baseline) for it to be visible in Project B, which means that all the files of the shared component that had changes made to them end up being duplicated in both projects' development and integration streams.
How can I set up a component to be shared by two projects without its code becoming duplicated?
Example: when I make a change to a file in shared component C in project A, deliver it to project A's Integration stream, create a baseline, and then proceed to deliver that baseline to project's B's Development stream followed by a delivery to project B's Integration stream and do a version tree of that file, I see 4 versions of the same file that are all identical. Granted, we're still talking about the same element, but isn't there a way in which two projects can share a component (only modifiable in one) without this happening?
I think what I'm looking for is for one project to be the "producer" and other projects to be the "consumers" of this component.
UCM Baseline are a way to reference a precise version of a set of files (the UCM components)
If you make a new baseline in project A / component C, that does not influence project B which keeps referencing its own initial version of the component C (with possibly its own changes)
When you deliver a baseline inter-project:
you end up referencing the exact same versions of the files labelled by that baseline label if component C in project B had not changed: no duplication there.
or you end up merging your changes from project A into the component C of project B, which had new versions of its own. Again, no duplication, simply a report of changes in order to integrate them in another development stream for a given component: there might be merge conflicts that need to be resolved.
when I make a change to a file in shared component C in project A, deliver it to project A's Integration stream, create a baseline, and then proceed to deliver that baseline to project's B's Development stream followed by a delivery to project B's Integration stream and do a version tree of that file, I see 4 versions of the same file that are all identical.
One way to avoid that is to:
reference C as non-modifiable UCM component in project A or B, which means you can, in any Stream, rebase and change its baseline at will, without having to deliver C ever.
Change C in a dedicated Project C, in which C is modifiable, and in which you make a C baseline (which you then can rebase in project A or B, no need for inter-project delivery)
There are multiple modules in our applications and each of them have their separate version and depend on other modules(external to our organization). They all have a parent POM which has it's own version, independent from the children's version.
When one of those modules change, they're converted to snapshots.
For the following example:
Parent v14.0
- module1 v1.5.0
- dependency1(module2 v15.0.0)
- dependency2(external-jar v12.0.1)
- module2 v15.0.0
- module3 v3.1.0
If there would be a change in module2, then module2's version would become v15.0-SNAPSHOT, then module1 becomes v1.5-SNAPSHOT. The parent remains the same.
The purpose for not having the same version on parent+modules, is that we want to localize the updates made to some modules and not affect the others' versions.
This has been designed like this a long time ago and there are several bash scripts to support the updates, although they're not handling all the cases. In any case, we don't have a one-click release process and we feel we are quite far from it with this approach.
We don't know how to convince the management towards a single version approach on all modules. How do you feel about the above? Did you ever encountered a project using the above structure and how well did it go?
Thank you!
I've had to deal with such situations before. There is an actual benefit from having decentralized versions, especially in cases where your product is made out of a large number of modules and this is because of the following facts:
You don't have to release all of them as a whole, if only a handful have been changed (which, from my observation is almost always the case).
You don't have to create unnecessary tags in your version control for code which hasn't changed since the previous release.
You don't have to waste an excessive amount of time releasing modules which don't need to be released.
You know with certainty which modules have changed in a release, which helps a lot when you need to investigate a complex bug, which seems to be dating back a while.
You can actually release certain modules/aggregators before the actual release date of the complete product, allowing for more testing time and a feeling of completeness for a given part of the product.
You can make feature branch releases much easier and implement a continuous delivery in a better way.
You can re-use the same code across multiple development branches without wondering if that branched version matches the one for your branch (or at least with less confusion).
What we ended up doing was:
Extract a parent or set of parents (with no sub-modules).
Try to use fixed versions for parents as much as possible. This is a bit of a caveat, as you must change all modules that inherit it, but in the end it improves the stability.
Extract each of the modules whose versions are independent of the rest to separate modules.
Extract sets of modules whose versions must always move along together into aggregators.
Create jobs in your CI server that can do releases or manually release these modules.
Use the versions-maven-plugin.
I think it's a lot more mature of a project and company's development principles to use decentralized versions and I must admit that in the beginning I was very reluctant to this approach. You might not realize or understand the benefits immediately, but with some practice and a proper setup, you will start seeing the upsides. I'm not saying there aren't caveats like... for example bumping the version of a parent, or having to know in which modules to bump the version of one of your modules.
From my experience, this module actually works better in the end, once you've become used to working with it.
From my experience: Everywhere we tried this, it eventually failed.
Although Maven supports this approach it is not advisable because of the additional effort.
I try to use the following criteria when choosing whether to use distinct projects or a multimodule structure:
If all projects have the same release cycle, I put them in a common multi module structure. In that case, I give them all the same version and release them together.
If a part of the project is used by different other projects (organizational projects), I always split them of and give them a separate lifecycle and a separate version.
If one part of my project stabilizes, I split it off, and give it a separate lifecycle (Maven refactoring)
To do it different alway results in homebrew solutions that neither scale well, nor are easy to maintain.
Hope that helps.
Suppose I have a project "MyFramework" that has some code, which is used across quite a few solutions. Each solution has its own source control management (SVN).
MyFramework is an internal product and doesn't have a formal release schedule, and same goes for the solutions.
I'd prefer not having to build and copy the DLLs to all 12 projects, i.e. new developers should to be able to just do a svn-checkout, and get to work.
What is the best way to share MyFramework across all these solutions?
Since you mention SVN, you could use externals to "import" the framework project into the working copy of each solution that uses it. This would lead to a layout like this:
C:\Projects
MyFramework
MyFramework.csproj
<MyFramework files>
SolutionA
SolutionA.sln
ProjectA1
<ProjectA1 files>
MyFramework <-- this is a svn:externals definition to "import" MyFramework
MyFramework.csproj
<MyFramework files>
With this solution, you have the source code of MyFramework available in each solution that uses it. The advantage is, that you can change the source code of MyFramework from within each of these solutions (without having to switch to a different project).
BUT: at the same time this is also a huge disadvantage, since it makes it very easy to break MyFramwork for some solutions when modifiying it for another.
For this reason, I have recently dropped that approach and am now treating our framework projects as a completely separate solution/product (with their own release-schedule). All other solutions then include a specific version of the binaries of the framework projects.
This ensures that a change made to the framework libraries does not break any solution that is reusing a library. For each solution, I can now decide when I want to update to a newer version of the framework libraries.
That sounds like a disaster... how do you cope with developers undoing/breaking the work of others...
If I were you, I'd put MyFrameWork in a completely seperate solution. When a developer wants to develop one of the 12 projects, he opens that project solution in one IDE & opens MyFrameWork in a seperate IDE.
If you strong name your MyFramework Assemby & GAC it, and reference it in your other projects, then the "Copying DLLs" won't be an issue.
You just Build MyFrameWork (and a PostBuild event can run GacUtil to put it in the asssembly cache) and then Build your other Project.
The "best way" will depend on your environment. I worked in a TFS-based, continuous integration environment, where the nightly build deployed the binaries to a share. All the dependent projects referred to the share. When this got slow, I built some tools to permit developers to have a local copy of the shared binaries, without changing the project files.
Does work in any of the 12 solutions regularly require changes to the "framework" code?
If so your framework is probably new and just being created, so I'd just include the framework project in all of the solutions. After all, if work dictates that you have to change the framework code, it should be easy to do so.
Since changes in the framework made from one solution will affect all the other solutions, breaks will happen, and you will have to deal with them.
Once you rarely have to change the framework as you work in the solutions (this should be your goal) then I'd include a reference to a framework dll instead, and update the dll in each solution only as needed.
svn:externals will take care of this nicely if you follow a few rules.
First, it's safer if you use relative URIs (starting with a ^ character) for svn:externals definitions and put the projects in the same repository if possible. This way the definitions will remain valid even if the subversion server is moved to a new URL.
Second, make sure you follow the following hint from the SVN book. Use PEG-REVs in your svn:externals definitions to avoid random breakage and unstable tags:
You should seriously consider using
explicit revision numbers in all of
your externals definitions. Doing so
means that you get to decide when to
pull down a different snapshot of
external information, and exactly
which snapshot to pull. Besides
avoiding the surprise of getting
changes to third-party repositories
that you might not have any control
over, using explicit revision numbers
also means that as you backdate your
working copy to a previous revision,
your externals definitions will also
revert to the way they looked in that
previous revision ...
I agree with another poster - that sounds like trouble. But if you can't want to do it the "right way" I can think of two other ways to do it. We used something similar to number 1 below. (for native C++ app)
a script or batch file or other process that is run that does a get and a build of the dependency. (just once) This is built/executed only if there are no changes in the repo. You will need to know what tag/branch/version to get. You can use a bat file as a prebuild step in your project files.
Keep the binaries in the repo (not a good idea). Even in this case the dependent projects have to do a get and have to know about what version to get.
Eventually what we tried to do for our project(s) was mimic how we use and refer to 3rd party libraries.
What you can do is create a release package for the dependency that sets up a path env variable to itself. I would allow multiple versions of it to exist on the machine and then the dependent projects link/reference specific versions.
Something like
$(PROJ_A_ROOT) = c:\mystuff\libraryA
$(PROJ_A_VER_X) = %PROJ_A_ROOT%\VER_X
and then reference the version you want in the dependent solutions either by specific name, or using the version env var.
Not pretty, but it works.
A scalable solution is to do svn-external on the solution directory so that your imported projects appear parallel to your other projects. Reasons for this are given below.
Using a separate sub-directory for "imported" projects, e.g. externals, via svn-external seems like a good idea until you have non-trivial dependencies between projects. For example, suppose project A depends on project on project B, and project B on project C. If you then have a solution S with project A, you'll end up with the following directory structure:
# BAD SOLUTION #
S
+---S.sln
+---A
| \---A.csproj
\---externals
+---B <--- A's dependency
| \---B.csproj
\---externals
\---C <--- B's dependency
\---C.csproj
Using this technique, you may even end up having multiple copies of a single project in your tree. This is clearly not what you want.
Furthermore, if your projects use NuGet dependencies, they normally get loaded within packages top-level directory. This means that NuGet references of projects within externals sub-directory will be broken.
Also, if you use Git in addition to SVN, a recommended way of tracking changes is to have a separate Git repository for each project, and then a separate Git repository for the solution that uses git submodule for the projects within. If a Git submodule is not an immediate sub-directory of the parent module, then Git submodule command will make a clone that is an immediate sub-directory.
Another benefit of having all projects on the same layer is that you can then create a "super-solution", which contains projects from all of your solutions (tracked via Git or svn-external), which in turn allows you to check with a single Solution-rebuild that any change you made to a single project is consistent with all other projects.
# GOOD SOLUTION #
S
+---S.sln
+---A
| \---A.csproj
+---B <--- A's dependency
| \---B.csproj
\---C <--- B's dependency
\---C.csproj