How to deal with vanished Maven repository (caching)? - maven

There used to be a repository that supplied us a certain version of a certain package - au.csiro.aehrc.variant-spark:variant-spark_2.12:jar:0.4.0-a0-dev1. Let's pretend the repository was at 12.3.4.567. In my settings.xml, I have an entry for it:
<repository>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
<id>3rdParty</id>
<url>https://12.3.4.567:8081/repository/maven-group</url>
</repository>
When that repository was up and running, I was able to acquire the package and my mvn build succeeds. Other, newer developers can't build with the same configuration, because that repository no longer can supply them the jar file. If I comment out that repository in my settings.xml, my build fails:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 57.864 s
[INFO] Finished at: 2022-12-16T06:35:31-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project my-project:
Could not resolve dependencies for project gov.x.y:my-project:jar:1.1.0-SNAPSHOT:
Could not find artifact au.csiro.aehrc.variant-spark:variant-spark_2.12:jar:0.4.0-a0-dev1
in central (https://repo1.maven.org/maven2) -> [Help 1]
If I remove the comment characters in my settings.xml, it builds again.
I don't have a good feel for how Maven caching works. There appears to be some kind of caching of mvn artifacts on my local machine that gets found based on a repository key. I'm afraid that at some point the cache will be modified, and I will no longer be able to build.
How can I make my development world a safer place and also make au.csiro.aehrc.variant-spark:variant-spark_2.12:jar:0.4.0-a0-dev1 (and its dependencies) available to other developers? (BTW, I haven't been able to find that version of variant-spark on any public repositories.) Is there a way I can work backwards from my local Maven artifacts such that we could create a local repository to supply the artifacts?
I'm open to hearing about kludgy, simple safeguards as well as how to do things the right way.

You have now found out why it is so important to use artifacts published to Maven Central or somewhere similar that won't go away. Your build server should have found this a while back - you may want to consider setting one up for the next time something breaks.
Right now, I would suggest:
back up your computer (or at least the .m2 folder).
set up a local repository manager like Nexus or Artifactory and install these files in it in the same location so others can use it immediatlye.
create a virtual repository that invisibly merges this local repository with Maven Central
tell everybody to set up their settings.xml to use that as a mirror.
This should get everybody up and running again.
Then, strongly reconsider if depending on an artifact only present in your office is a good idea. Is it elsewhere under a different name? Can you utilize something else? Can you get the source so you can fork and maintain it yourself?

Related

Jenkins - deploy artifacts to Maven repository

I have a basic Springboot Maven project and I want to be able to deploy it and make API call as it works in local.
I have a remote linux machine with Jenkins on it, and I am able to make a build of my application correctly. Now I want to deploy this build in the same linux machine, in a certain folder /deploy.
Right now I have added a Post Build Action on Jenkins to Deploy artifacts to Maven repository having the following parameters:
and right now I did not make any changes to my pom.xml or my maven settings.xml.
The error that I get is the following:
[INFO] Deployment in http://localhost:8080/deploy (id=test2,uniqueVersion=true)
Deploying the main artifact reag.login-0.0.1-SNAPSHOT.jar
Downloading: http://localhost:8080/deploy/reag/login/reag.login/0.0.1-SNAPSHOT/maven-metadata.xml
ERROR: Failed to retrieve remote metadata reag.login:reag.login:0.0.1-SNAPSHOT/maven-metadata.xml: Could not transfer metadata reag.login:reag.login:0.0.1-SNAPSHOT/maven-metadata.xml from/to deploymentRepo (http://localhost:8080/deploy): Access denied to: http://localhost:8080/deploy/reag/login/reag.login/0.0.1-SNAPSHOT/maven-metadata.xml , ReasonPhrase:Forbidden.
org.apache.maven.artifact.deployer.ArtifactDeploymentException: Failed to retrieve remote metadata reag.login:reag.login:0.0.1-SNAPSHOT/maven-metadata.xml: Could not transfer metadata reag.login:reag.login:0.0.1-SNAPSHOT/maven-metadata.xml from/to deploymentRepo (http://localhost:8080/deploy): Access denied to: http://localhost:8080/deploy/reag/login/reag.login/0.0.1-SNAPSHOT/maven-metadata.xml , ReasonPhrase:Forbidden.
The machine where I'm working is protected by username and password, I tried to put them in the settings.xml file but nothing changes. Does anyone know which are the steps to make this process work?
Thanks in advance.
Local artifact deployment is done by mvn clean install.
The install goal copies your artifact into your local maven repository (Default [USER_HOME]/.m2/repository).
If you want to deploy your artifact to another service, like a Nexus Maven Repository, then you need to deploy and also provide the credentials for that machine in your settings.xml, or even better setup a private/public key authentication for the machines.
I think Jenkins is trying to do the remote deploy, which is only working if you run a Maven Repository on your machine (like the Nexus)
"deploy" in Maven parlance means "upload the built artifacts and their metadata (such as pom files) to a repository manager.
It does not mean "copy the artifact to any location that you want".
Therefore as #funfried mentioned, you need to actually have a repository manager running.
If that is the case then the following is how you set up the maven configuration.
Maven links credentials to servers via the id element.
Your settings.xml file would have your credentials setup something like:
<servers>
<server>
<id>newhope-nexus</id>
<username>steve</username>
<password>{1T7Jmp/PBoQH4cvFjZDTaDe/F/Z+D9rJ925rf+3H1LY=}</password>
</server>
</servers>
And then your project model (project pom or parent pom) should define it's distributionManagement:
<distributionManagement>
<repository>
<id>newhope-nexus</id>
<url>http://newhope:8081/nexus/content/repositories/releases/</url>
</repository>
<snapshotRepository>
<id>newhope-nexus</id>
<url>http://newhope:8081/nexus/content/repositories/snapshots/</url>
</snapshotRepository>
</distributionManagement>
Note that the ids all match.
The mvn deploy from Jenkins should then work correctly.

How to update release artifacts in Maven?

I have the following case:
- Maven project compiles fine on one machine where artifact A (release one) exists in that machine's local Maven repository,
- artifact A was removed from the remote repository.
How can we be sure that the project builds fine on a new machine where local Maven repository is empty?
Try the install task install-file to put the jar into your local repo from your local hard drive:
mvn install:install-file -Dfile=<path-to-file>
See maven.apache.org for more details.
Few cases :
One, if you own artifact A and use <repository> to keep the versions in sync. You can anytime rebuild your project on the new machine to fetch the artifact.
Second, If artifact A is owned by you and you didn't use repositories until now, you can simply bump up its version and deploy it on the remote server(<repository>) and consume the new version on the machine where you want to. Use <repositories> in your pom.xml to specify the remote path as follows :
<repository>
<id>clojars</id>
<name>Clojars Maven Repository</name>
<url>http://clojars.org/repo/</url>
</repository>
Third, if you don't own the artifact A and it exists in a public maven central repositories, all you need to do is rebuild the correct dependency to fetch it over the net.
Worst case, the artifact A used is owned by you and doesn't exist in a public central repository. You can still copy-paste the jar file using any mode and move it to the machine's /m2/repository where you require it. But do note in such case you shouldn't reach this case ever again.
Also in the last case, you can copy it into your project and use it as suggested by #Adam
mvn install:install-file -Dfile=<path-to-file>

Maven can't find dependencies [dependencyResolutionException]

I added a Maven dependency to my project and my project compiles locally, while it doesn't compile on server. It can not resolve the newly added dependency.
This is my pom.xml file:
<repositories>
<repository>
<id>rep</id>
<name>Repository</name>
<url>http://artifacts.com/rep</url>
<releases>
<enabled>true</enabled>
<updatePolicy>always</updatePolicy>
</releases>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>com.project.rest</groupId>
<artifactId>common</artifactId>
<version>2.0.5</version>
</dependency>
</dependencies>
And this my console output with an error:
Downloading: http://artifacts.com/rep/com/project/rest/common/2.0.5/common-2.0.5.pom
[WARNING] The POM for com.project.rest:common:jar:2.0.5 is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.154s
[INFO] Finished at: Tue Feb 03 06:58:35 BRT 2015
[INFO] Final Memory: 9M/152M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project server: Could not resolve dependencies for project org.server:
server:jar:2.5.1-SNAPSHOT: The following artifacts could not be resolved: com.project.rest:common:jar:2.0.5:
Could not find artifact com.project.rest:common:jar:2.0.5 in rep (http://artifacts.com/rep) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
(I hid all real URLs and package.)
What could be the problem?
The first line "Downloading:..." only says that maven tries to download the artifact. It is no statement about success. If maven is successful you will get another line starting with "Downloaded: ..."
So in your case maven was not able to download the file. Check the logged url in your browser if it does exist and if it is protected.
BTW <updatePolicy>always</updatePolicy> is quite uncommon for release repos, because releases should not change any more.
Make sure your repository is configured properly. Try to search, in your repo, for
com.project.rest:common of version 2.0.5.
Is this your own project? some jar that you have built? are you sure you deployed it to your repo? if it is not in your repo, try to search for it in your local repo (usually .m2/repository/com/project...)
DependencyResolutionException
(source: Maven Confluence)
This error generally occurs when Maven could not download dependencies. Possible causes for this error are:
The POM misses the declaration of the <repository> which hosts the artifact.
The repository you have configured requires authentication and Maven failed to provide the correct credentials to the server. In this case, make sure your ${user.home}/.m2/settings.xml contains a <server> declaration whose <id> matches the <id> of the remote repository to use. See the Maven Settings Reference for more details.
The remote repository in question uses SSL and the JVM running Maven does not trust the certificate of the server.
There is a general network problem that prevents Maven from accessing any remote repository, e.g. a missing proxy configuration.
You have configured Maven to perform strict checksum validation and the files to download got corrupted.
Maven failed to save the files to your local repository, see LocalRepositoryNotAccessibleException for more details.
In Maven 3 if you just had a failed download and have fixed it (e.g. by uploading the jar to a repository) it will cache the failure. To force a refresh add -U to the command line.
In case of a general network-related problem, you could also consult the following articles:
Configuring a Proxy
Security and Deployment Settings
Guide to Remote Repository Access through Authenticated HTTPS
The project you mentioned doesn't contains POM i.e. it is not a MAVEN project. The .m2(look this at c drive or where you have installed) repository contains all the dependencies folders look there whether it contains the required project or not i.e. com.project.rest.
Like suggested by console output, you can check the Maven Wiki page (http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException) for more information on where the problem might lay.
Given the information you provided. I would guess we are looking at the point one issue:
The POM misses the declaration of the which hosts the artifact.
Maven repositories are of three types: local, central and remote.
In your case, your local repository contains the project specific artifacts, but the central repository doesn't.
Local repository is on your local machine. When you run a Maven build, Maven automatically downloads all the dependency jars into the local repository.
Central repository is web repository provided by a community and it contains a number of commonly used libraries, found here https://repo1.maven.org/maven2/.
Remote repository is developer's own custom repository containing required libraries or other project jars. Developer defines these repositories in POM file using tags.
I would suggest the following:
check your repository tag, if the repository is correct
check if the repository contains the artifacts specified in your dependency tag
If you are using IntelliJ IDE (Jetbrains) and code is correct then
Check the TOGGLE OFFLINE MODE button. If we enable offline mode, then IDE do not let maven download a dependency and it cannot found the library that causes error.
So make sure your project is build in ONLINE mode.

Maven can't find artefact from Nexus

We have an Atlassian Bamboo instance building and deploying our projects (snapshots) to Nexus, around 11:20 pm every day.
Another Bamboo instance runs a test plan A at midnight and fails because it can't find an artefact from Nexus (although it's looking in the right repositories), artefact that was built by the other Bamboo instance and is actually in Nexus.
A test plan B starts around 00:30 on the same instance and this one finds the artefact. In the morning, when I manually launch the plan A, it works well.
So I suspect a cache/metadata issue, but I couldn't figure out what is the right configuration to set, either in Nexus or in the Maven settings.
It's running Maven 2.2.1. Other plans running Maven 3.0.5 for a different version of our project don't seem to have the problem. Nexus is 2.7.2-03.
The error is "2 required artifacts are missing." and the list of Nexus groups in "from the specified remote repositories:" is the right one. Those groups are configured like that in the project's pom:
<snapshots>
<enabled>true</enabled>
<updatePolicy>always</updatePolicy>
</snapshots>
Any idea about how to fix this issue?
Thank you!
Best practices we've identified for moving Maven 2 builds to Maven 3:
1) All of the builders (individual users running builds on personal machines plus Jenkins/CI) should be converted at the same time. In other words, once the POM for an artifact has been adjusted as needed and the built artifact deployed to the remote repository with Maven 3, then Maven 2 should not be used to build that same artifact ever again. This is because Maven 3 uses timestamped snapshots, while Maven 2 does not. Also, Maven 2 and Maven 3 handle dependency resolution and repository metadata differently. Mixing the two doesn't work well.
One symptom is someone commits changes, builds, and deploys to the remote repo with Maven 3, then another developer tries to download the changed jar with Maven 2 and gets the old one. The team may prevent the artifacts from being built with Maven 2 using the Enforcer plugin.
2) All builders should clean out their local artifact repositories. Maven 2 and Maven 3 handle dependency resolution and repository metadata differently, and a fresh repository insulates the developer from strange, hard to find problems. (Since Maven will then have to download the world on the first build, consider doing the first Maven 3 build before you go to lunch or leave work for the day.)
3) Someone with delete privileges on the remote repository (e.g. Nexus, Artifactory) should log into the remote repo, find the artifact(s) in the Snapshots repository and delete the snapshot version during which the Maven 2 to 3 migration occurred.
For example, say that the team is working on 2.0-SNAPSHOT, and there's a task to move from Maven 2 to 3 for this version. When the POM changes for the task are complete, the developer logs into the remote repo, finds com.company.some-group:myArtifact(s), and removes the 2.0-SNAPSHOT version entirely. This will remove all snapshots built by Maven 2.

maven plugin: "Failed to read artifact descriptor for"

For my build process, I need two custom maven plugins (see here). It works fine when deploying it to my local maven repository with mvn install.
However, I need to make it accessible for other developer's in my team and integration testing.
I noticed that you could setup a light-weight maven repository on github so that's what I did for now: https://github.com/rweng/mvn-repo
This works fine for normal dependencies like ch.yax.yocto.yocto-server, however, it fails for my plugins with the message
[WARNING] The POM for com.arcanio.maven.plugin:velocity:jar:0.1-SNAPSHOT is missing, no dependency information available**
[ERROR] Plugin com.arcanio.maven.plugin:velocity:0.1-SNAPSHOT or one of its dependencies could not be resolved: Failed to read artifact descriptor for com.arcanio.maven.plugin:velocity:jar:0.1-SNAPSHOT: Could not find artifact com.arcanio.maven.plugin:velocity:pom:0.1-SNAPSHOT -> [Help 1]
I tried substituting the github url in my repositories section through file:///, Though I doubt the problem lies here. I also doubt that this is a proxy problems, as many users with the same error reported.
Is is possible that mvn install deploys s.th. different than my deployment command
mvn -DaltDeploymentRepository=snapshot-repo::default::file:/Users/robin/Code/mvn-repo/ clean deploy
Thanks in advance for any hints how to solve this.
EDIT
I just moved the plugin from my local repository to the github repository and notice the following changes.
D com/arcanio/maven/plugin/maven-metadata.xml
D com/arcanio/maven/plugin/maven-metadata.xml.md5
D com/arcanio/maven/plugin/maven-metadata.xml.sha1
D com/arcanio/maven/plugin/velocity/0.1-SNAPSHOT/maven-metadata.xml
D com/arcanio/maven/plugin/velocity/0.1-SNAPSHOT/maven-metadata.xml.md5
D com/arcanio/maven/plugin/velocity/0.1-SNAPSHOT/maven-metadata.xml.sha1
D com/arcanio/maven/plugin/velocity/0.1-SNAPSHOT/velocity-0.1-20130206.084855-1.jar
D com/arcanio/maven/plugin/velocity/0.1-SNAPSHOT/velocity-0.1-20130206.084855-1.jar.md5
D com/arcanio/maven/plugin/velocity/0.1-SNAPSHOT/velocity-0.1-20130206.084855-1.jar.sha1
D com/arcanio/maven/plugin/velocity/0.1-SNAPSHOT/velocity-0.1-20130206.084855-1.pom
D com/arcanio/maven/plugin/velocity/0.1-SNAPSHOT/velocity-0.1-20130206.084855-1.pom.md5
D com/arcanio/maven/plugin/velocity/0.1-SNAPSHOT/velocity-0.1-20130206.084855-1.pom.sha1
D com/arcanio/maven/plugin/velocity/maven-metadata.xml
D com/arcanio/maven/plugin/velocity/maven-metadata.xml.md5
D com/arcanio/maven/plugin/velocity/maven-metadata.xml.sha1
?? com/arcanio/maven/plugin/maven-metadata-local.xml
?? com/arcanio/maven/plugin/velocity/0.1-SNAPSHOT/_maven.repositories
?? com/arcanio/maven/plugin/velocity/0.1-SNAPSHOT/maven-metadata-local.xml
?? com/arcanio/maven/plugin/velocity/0.1-SNAPSHOT/velocity-0.1-SNAPSHOT.jar
?? com/arcanio/maven/plugin/velocity/0.1-SNAPSHOT/velocity-0.1-SNAPSHOT.pom
?? com/arcanio/maven/plugin/velocity/maven-metadata-local.xml
So mvn install really does generate something different. Does anyone know why? I thought mvn install was basically the same as mvn deploy but to the local repository.
EDIT
Using mvn -DuniqueVersion=false fixes the unique versions. See here.
EDIT
uniqueVersion=false does not work anymore with maven 3. So the problem stays the same, the timestamped versions are not found in the repository. I think I might be missing a artifactid-snapshot pom.
SOLUTION
Solution found here:
The repository must be added as pluginRepository:
<pluginRepositories>
<pluginRepository>
<id>rweng-plugins</id>
<url>https://github.com/rweng/mvn-repo/raw/master</url>
<snapshots>
<enabled>true</enabled>
<updatePolicy>always</updatePolicy>
</snapshots>
</pluginRepository>
</pluginRepositories>
The best solution for such things is to install a repository manager in your company.

Resources