My organization has several maven repositories on bintray. Most of these repositories contain a single package of the same name as the repository.
These packages seem highly related and I would like to reorganize them by creating a single repository and moving all these packages into it.
What is the best way to move packages between own repositories given the following constraints:
the package must remain available in the old repository as a link to the package in the new repository.
some of packages are linked to jcenter and they must remain linked after the moving.
I'll quote here a response from my conversation with JFrog support:
Since it is not possible to link a package from one repository to another within the same organisation, the best option would be to move these packages to the new repository while maintaining the link to jcenter. This leave the old repositories empty and then you would be able to delete them entirely. It is not possible to retain a link from the old repository to the new one.
Please notice that the move operation is an internal process and can only be done by us.
Unfortunately this approach doesn't satisfy my first requirement that the package must remain available for downloading from the old repository.
Related
I need a tool that will help to find all artifacts that reference another artifact.
When I rebuild an artifact, I need to update/rebuild all artifacts that were using the old version. But I work in a big organisation, and nobody knows really where the artifact is spread in the organisation, so nobody is ever completely sure that everybody use the latest versions.
What I need would be a tool - maybe an artifactory plugin or feature, or a maven plugin doing a lookup in the repository - that indexes all the known poms, and is able to make a listing of all artifacts that have the updated artifact in their dependencies, either directly and transitively. Thus a list of artifacts I would need to rebuild. Quite the opposite of dependency:tree.
Filtering that list by repository, groupId, packaging, etc. is a nice to have. But I can live without.
Any idea?
You can use the Artifactory Query Language with the REST API to do that. For example, if you want to find all builds that use "MySuperAwesomeDependency-1.0.2" your AQL statement would be something like:
//Find builds that use a dependency that is a snapshot
builds.find({"module.dependency.item.name":{"$match":"MySuperAwesomeDependency-1.0.2*"}})
The key in the above statement would be the module.dependency.item.name, which allows you to search for dependencies by name, assuming you store the dependencies in Artifactory.
I have a Gradle build set up in Azure DevOps, which compiles the code in an Azure DevOps git repository, then publishes the generated JARs (as Maven artifacts) to Azure Artifacts, as explained here. Code in other Azure DevOps git repositories can then reference these components as dependencies. This is fine for formal releases of these components (with unique version numbers), but I also need a way to get this working for in-progress snapshot releases. The problem is that I cannot publish an artifact with the same version number (e.g. 1.2.3-SNAPSHOT) more than once. This seems to be because packages in Azure are immutable.
From my understanding, that would mean that Azure Artifacts cannot be used to store in-progress snapshot artifacts. Is that correct?
If it is, is there any alternative that still uses Azure DevOps? I can see that I can publish artifacts to Azure Blob Storage, but presumably this is something you have to pay for on top of existing use of Azure Artifacts. I can also see that there's a number of GitHub Maven plugins for treating a GitHub repo as a Maven repo, but I can't find anything similar for using an Azure DevOps repo as a place to publish Maven artifacts.
In case it makes a difference, I'm talking about the cloud-based Azure stuff, nothing on-premise.
Specifically for maven, there's a difference in how a jar is packaged in a SNAPSHOT as compared to other versions.
Eg. let's say I version my jar as 1.0.0-PRERELEASE, now this version is immutable. Once I publish this to any artifact repository it cannot be overridden. Usually package management systems store the artifact in this kind of logical path - <group_id>/<artifact_id>/<version>/<artifact_id>_<version>.jar.
But for SNAPSHOTs this is slightly different. In maven, SNAPSHOTs are mutable. This means you can override a SNAPSHOT. And this makes sense as by definition SNAPSHOTs are like an image of your code in active development. How SNAPSHOTs are stored in a package management feed is something like <group_id>/<artifact_id>/<version(SNAPSHOT)>/<artifact_id>_<version>_<jar_timestamp>.jar.
This naming convention of the jar in this case makes it mutable. In a single SNAPSHOT version, there can be multiple jars present with different timestamps. And when maven looks for a SNAPSHOT dependency to download, it always picks the most recent one (again logical as that's the most recent snapshot).
At present azure artifacts implements these concepts exactly how they are supposed to. Perhaps this wasn't there before and has been pushed in an update.
http://maven.apache.org/guides/getting-started/index.html#What_is_a_SNAPSHOT_version
https://xebia.com/blog/continuous-releasing-of-maven-artifacts/
The premise of Package Management is that a package is immutable. This enables a whole bunch of caching options that would otherwise not exist. Packages are stored in your local package cache, possibly on a proxy feed package cache and all of these elements assume that packages with the same name+version are unchanged and will serve the cached version instead of the latest version you've pushed. Most package systems are built on this premise, including Nuget and NPM.
The trick to creating development snapshots is to use semantic versioning and adding a unique suffix to your version. For example 1.2.3-SNAPSHOT.1 followed by 1.2.3-SNAPSHOT.2, there are tools available for Azure Pipelines, like GitVersion that can automatically generate a unique version + suffix that you can pass into the version for your artifact.
If you don't want to "mess up" your main package feed, you can setup a second feed for development purposes which holds all your intermediate packages, you can then either promote one of these packages to your main feed or you can run a specific pipeline (configuration) to push the final package to the feed used for your stable packages.
It looks like this is mostly a feature and not a bug as to preserve the immutability of the build results - meaning, that no matter when that build is ran, it will always return the same result. See: How to update a maven dependency with a same version number in Azure Artifacts & Azure Artifact Publishing Fails for Artifact Version Containing '+'
I have a package in Bintray called "tripod" which is here:
https://bintray.com/bbende/bbende-maven/tripod
The tripod package is a Maven project with a standard multi-module setup where the root pom artifactId is tripod, with sub-modules of tripod-search-api, tripod-search-solr, and tripod-search-lucene.
In my personal Maven repo, all of these artifacts are published under the groupId com.bbende.tripod as shown here:
https://dl.bintray.com/bbende/bbende-maven/com/bbende/tripod/
When I requested to link the tripod package to JCenter using the link on the main package page for "tripod" it ended up linking only the artifact for the top-level pom as seen here:
http://jcenter.bintray.com/com/bbende/tripod/
How are you supposed to get the entire package linked to JCenter?
I believe this is the same issue here which looks like it requires manual intervention from someone at bintray/jcenter:
Bintray does not sync one of the artifacts of the package to the jcenter
A Bintray package is linked to jcenter with a specific path prefix, so that all submodules and versions under the base path will automatically be linked to jcenter without further manual intervention.
It seems like your package is linked under the base path containing only one of the submodules, therefore, the other submodules are not linked to jcenter.
In such cases, you should contact JFrog support in order to fix the inclusion path. In this case, we have already fixed it and all submodules are correctly linked to jcenter.
Best Regards,
Itamar
I've been using maven since a year for managing my projects' dependencies, but I recently came to know that there is a concept of Maven Repository Manager.
I would like to ask What is a Maven Repository Manager and what is the purpose of using maven repository manager.
A "Maven Repository Manager" is basically a server that stores copies of all of your libraries so that they can be downloaded when a project is built. When you use Maven, you are using a repository manager already called "Maven Central." See here: https://maven.apache.org/repository-management.html
When you are working with a large project or corporation, they may host an alternative to Maven Central, like Sonatype Nexus. There are two reasons why they do.
First, a big corporation might have libraries that are intended only for internal use that are used across a large number of projects. For example, if you worked at Amazon, you might have libraries for completing credit card transactions. That shouldn't necessarily be shared with the rest of the world, so you don't want to put it in Central; you need to put it someplace private.
Second, it reduces bandwidth. If every developer at Amazon only used Maven Central, then that would be lots of network traffic. A repository acts as a "proxy" to Central. It searches internally for a library, and then if it doesn't find it, it downloads it from Central and then saves it for the next time someone asks for it.
To solve problems like:
how do you get your binary to your server in the first place?
which version do you want?
when you deploy a new version how do you revert to an old version?
which employees can access which binaries?
And so forth.
In my multi-module Maven project, suppose I have two modules, car and horse. They both depend on a JAR file, transport.jar, a file not available in any online Maven repositories. As such, I need to find a way to make these modules depend on a file found somewhere in the project folder structure.
From what I understand, the default Maven solution would be to manually register the JAR file in the local repository. While this would work on a development machine, it breaks on the build server, which clears its local repository before each build.
I've been searching online on how to do this on and off for a while and found some helpful things, but nothing that completely works.
For instance, a common answer is to add a dependency to the file using <scope>system</scope>. However, not only do others claim that it's extremely bad practice to do so, it also doesn't work on the build server. (On a side note, I would also like to point out that using absolute paths to the JAR is also out of the question due to, again, it being built on several different machines.)
A more useful method I found was to define a local repository in the POM file, pointing towards the path file:${project.basedir}/lib. (Such as in this article) Unfortunately, if I place the JAR and repository definition in the car POM, I cannot successfully add a dependency to the JAR in horse. I've tried both with and without an additional reference to car in horse, as well as defining a second repository in horse, pointing to file:${project.basedir}/../car/lib. This problem would also remain if I tried to make a third module, transport-lib, specifically for wrapping the JAR dependency.
I could most likely add the JAR file to both modules and define two separate module-local repositories, but I really don't want to unless I have to due to the need to keep the two (often updated) JARs in sync etc.
So, my question is as follows: Can someone give me a confirmed-to-work method to have two modules depend on the same JAR file inside the project, given the parameters and restrictions mentioned?
Best solution is to use a repository manager like Archiva, Artifactory or Nexus and install that artifact into the repository manager. Afterwards you can use this artifact directly in your pom files without any issue.
Don't use the scope system, cause it will cause other problem after a release for other etc.