Maven build on AWS wrongly using SNAPSHOT private repo - maven

We have a private maven repo (artifactory) and we have a release jar (cryptolib-0.9.7.jar) in the libs-release folder.
Building locally or on Jenkins (hosted on Azure), all works fine. Building on codebuild on AWS, maven tries (and fails) to download from libs-snapshot instead.
Jenkins & local output (v3.3.9)...
Downloading: http://[our-repo]/artifactory/libs-release/org/openpseudonymiser/cryptolib/0.9.7/cryptolib-0.9.7.pom
Downloaded: http://[our-repo]/artifactory/libs-release/org/openpseudonymiser/cryptolib/0.9.7/cryptolib-0.9.7.pom (457 B at 3.7 KB/sec)
Downloading: http://[our-repo]/artifactory/libs-release/org/openpseudonymiser/cryptolib/0.9.7/cryptolib-0.9.7.jar
Downloaded: http://[our-repo]/artifactory/libs-release/org/openpseudonymiser/cryptolib/0.9.7/cryptolib-0.9.7.jar (13 KB at 234.8 KB/sec)
AWS/CodeBuild output (v.3.3.3)...
Downloading: http://[our-repo]/artifactory/libs-snapshot/org/openpseudonymiser/cryptolib/0.9.7/cryptolib-0.9.7.jar
Both are executing mvn -B install. I've tried all sorts of online suggestions and the the pom repository section currently looks like this (which now seems to ONLY attempt download from central)...
<repositories>
<repository>
<releases><enabled>false</enabled></releases>
<snapshots><enabled>true</enabled></snapshots>
<id>snapshots</id>
<name>libs-snapshot</name>
<url>http://[our-repo]/artifactory/libs-snapshot</url>
</repository>
<repository>
<releases><enabled>true</enabled></releases>
<snapshots><enabled>false</enabled></snapshots>
<id>central</id>
<name>libs-release</name>
<url>http://[our-repo]/artifactory/libs-release</url>
</repository>
</repositories>
What am I missing (apart from hair!!!)??

Doh! "When you use Maven with an AWS CodeBuild provided Java build environment, Maven pulls build and plugin dependencies from the secure central Maven repository at repo1.maven.org/maven2. This happens even if your build project's pom.xml file explicitly declares other locations to use instead." http://docs.aws.amazon.com/codebuild/latest/userguide/troubleshooting.html#troubleshooting-maven-repos
My solution was to delete the default settings.xml in the buildspec install phase.
phases:
install:
commands:
- rm /root/.m2/settings.xml

Related

maven goal using default maven repo and not the configured one

I am trying to integrate evosuite in my maven project using instructions from here and here.
I can build the project (clean install) without any issue.
I can execute evosuite:help or evosuite:prepare goal without any issue.
However I am having issue with evosuite:generate goal. There is problem downloading the transitive dependency.
We are using our own central repository which have all the jars that evosuite or evosuite plugin needs yet I can see in the logs that evosuite:generate goal tries to connect to default maven central repo (repo.maven.apache.org)
instead of the repo that I have configured.
For example;
I have following configuration in my paren pom :
<repositories>
<repository>
<id>central</id>
<name>bintray</name>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
<url>https://jcenter.bintray.com</url>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<snapshots>
<enabled>false</enabled>
</snapshots>
<id>central</id>
<name>bintray-plugins</name>
<url>http://jcenter.bintray.com</url>
</pluginRepository>
</pluginRepositories>
So when I execute clean install, it downloads jars from jcenter.bintray.com
Downloading: http://jcenter.bintray.com/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.pom
Downloaded: http://jcenter.bintray.com/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.pom (4 KB at 3.1 KB/sec)
Downloading: http://jcenter.bintray.com/org/apache/maven/plugins/maven-plugins/22/maven-plugins-22.pom
Downloaded: http://jcenter.bintray.com/org/apache/maven/plugins/maven-plugins/22/maven-plugins-22.pom (13 KB at 20.7 KB/sec)
But while execute evosuite:generate it downloads some artifacts from repo.maven
Why is that? How can I force it to refer our custom repo (jcenter in above example)
Here is complete logs for same.
Executing in debug mode, I saw a warning that
The pom for <name> is invalid, transitive dependencies (if any) will not be available
I have deleted the .m2 folder, checked the jars, downloaded the jars manually and refer them via local url in pom. But it did not solve the issue.
The goals executes fine when you execute from an environment where there are no proxies or other network restriction.
Hence I believe If I am able to use our repo instead of maven default one when goal evosuite:generate executes, I might be able to solve the issue.
Note: I dont think there's any issue with setting.xml
It has all the configuration for proxy and to use our own artifactory.

How do I force Maven to use my local repository rather than going out to remote repos to retrieve artifacts?

I’m using Maven 3.3.3 with Java 8 on Mac Yosemite. I have a multi-module project.
<modules>
<module>first-module</module>
<module>my-module</module>
…
</modules>
When I build my one of my child modules, for example, “my-module” from above, using “mvn clean install”, the build attempts to download the child module artifacts from a remote repository I have defined in my ~/.m2/settings.xml file. Output is below
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building my-module 87.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading: https://my.remoterepository.com/nexus/content/repositories/snapshots/org/mainco/subco/first-module/87.0.0-SNAPSHOT/maven-metadata.xml
Downloading: http://download.java.net/maven/2/org/mainco/subco/first-module/87.0.0-SNAPSHOT/maven-metadata.xml
Downloaded: https://my.remoterepository.com/nexus/content/repositories/snapshots/org/mainco/subco/first-module/87.0.0-SNAPSHOT/maven-metadata.xml (788 B at 0.9 KB/sec)
Downloading: https://my.remoterepository.com/nexus/content/repositories/snapshots/org/mainco/subco/first-module/87.0.0-SNAPSHOT/first-module-87.0.0-20151104.200545-4.pom
How do I force Maven to check my local ~/.m2/repository first before trying to download from the remote repositories? Below is where I have my remote repositories defined in my ~/.m2/settings.xml file …
<profile>
<id>releases</id>
<activation>
<property>
<name>!releases.off</name>
</property>
</activation>
<repositories>
<repository>
<id>releases</id>
<url>https://my.remoterepository.com/nexus/content/repositories/releases/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
</profile>
<profile>
<id>snapshots</id>
<activation>
<property>
<name>!snapshots.off</name>
</property>
</activation>
<repositories>
<repository>
<id>snapshots</id>
<url>https://my.remoterepository.com/nexus/content/repositories/snapshots/</url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
</profile>
Edit: In response to the answer saying that the download occurs when the artifact is not there, below is the terminal output in which I prove the file was there in my repo but Maven is trying to download it anyway ...
Daves-MacBook-Pro-2:my-module davea$ ls -al ~/.m2/repository/org/mainco/subco/first-module/87.0.0-SNAPSHOT/first-module-87.0.0-SNAPSHOT.jar
-rw-r--r-- 1 davea staff 10171 Nov 5 10:22 /Users/davea/.m2/repository/org/mainco/subco/first-module/87.0.0-SNAPSHOT/first-module-87.0.0-SNAPSHOT.jar
Daves-MacBook-Pro-2:my-module davea$ mvn clean install
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.mainco.subco:my-module:jar:87.0.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.apache.maven.plugins:maven-antrun-plugin # org.mainco.subco:my-module:[unknown-version], /Users/davea/Documents/sb_workspace/my-module/pom.xml, line 678, column 12
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building my-module 87.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading: https://my.remoterepository.com/nexus/content/repositories/snapshots/org/mainco/subco/first-module/87.0.0-SNAPSHOT/maven-metadata.xml
Downloading: http://download.java.net/maven/2/org/mainco/subco/first-module/87.0.0-SNAPSHOT/maven-metadata.xml
Downloaded: https://my.remoterepository.com/nexus/content/repositories/snapshots/org/mainco/subco/first-module/87.0.0-SNAPSHOT/maven-metadata.xml (788 B at 0.8 KB/sec)
Downloading: https://my.remoterepository.com/nexus/content/repositories/snapshots/org/mainco/subco/first-module/87.0.0-SNAPSHOT/first-module-87.0.0-20151106.043202-8.pom
Downloaded: https://my.remoterepository.com/nexus/content/repositories/snapshots/org/mainco/subco/first-module/87.0.0-SNAPSHOT/first- module-87.0.0-20151106.043202-8.pom (3 KB at 21.9 KB/sec)
Downloading: http://download.java.net/maven/2/org/mainco/subco/subco/87.0.0-SNAPSHOT/maven-metadata.xml
Downloading: https://my.remoterepository.com/nexus/content/repositories/snapshots/org/mainco/subco/subco/87.0.0-SNAPSHOT/maven-metadata.xml
The dependency has a snapshot version. For snapshots, Maven will check the local repository and if the artifact found in the local repository is too old, it will attempt to find an updated one in the remote repositories. That is probably what you are seeing.
Note that this behavior is controlled by the updatePolicy directive in the repository configuration (which is daily by default for snapshot repositories).
Use mvn --help and you can see the options list.
There is an option like -nsu,--no-snapshot-updates Suppress SNAPSHOT updates
So use command mvn install -nsu can force compile with local repository.
To truly force maven to only use your local repo, you can run with mvn <goals> -o. The -o tells maven to let you work "offline", and it will stay off the network.
Follow below steps:
Ensure to delete all the contents of the jar folder located in your local except the jar that you want to keep.
For example files like .repositories, .pom, .sha1, .lastUpdated etc.
Execute mvn clean install -o command
This will help to use local repository jar files rather than connecting to any repository.
Maven always checks your local repository first, however,your dependency needs to be installed in your repo for maven to find it.
Run mvn install in your dependency module first, and then build your dependent module.
In my case I had a multi module project just like you. I had to change a group Id of one of the external libraries my project was depending on as shown below.
From:
<dependencyManagement>
<dependency>
<groupId>org.thirdparty</groupId>
<artifactId>calculation-api</artifactId>
<version>2.0</version>
<type>jar</type>
<scope>provided</scope>
</dependency>
<dependencyManagement>
To:
<dependencyManagement>
<dependency>
<groupId>org.thirdparty.module</groupId>
<artifactId>calculation-api</artifactId>
<version>2.0</version>
<type>jar</type>
<scope>provided</scope>
</dependency>
<dependencyManagement>
Pay attention to the <groupId> section. It turned out that I was forgetting to modifiy the corresponding section of the submodules that define this dependency in their pom files.
It drove me very crazy because the module was available locally.
Even when considering all answers above you might still run into issues that will terminate your maven offline build with an error. Especially, you may experience a warning as follwos:
[WARNING] The POM for org.apache.maven.plugins:maven-resources-plugin:jar:2.6 is missing, no dependency information available
The warning will be immediately followed by further errors and maven will terminate.
For us the safest way to build offline with a maven offline cache created following the hints above is to use following maven offline parameters:
mvn -o -llr -Dmaven.repo.local=<path_to_your_offline_cache> ...
Especially, option -llr prevents you from having to tune your local cache as proposed in answer #4.
Also take care that parameter <localRepositoryin> in your settings.xml points to the correct folder of your local Maven repository.
The -o option didn't work for me because the artifact is still in development and not yet uploaded and maven (3.5.x) still tries to download it from the remote repository because it's the first time, according to the error I get.
However this fixed it for me: https://maven.apache.org/general.html#importing-jars
After this manual install there's no need to use the offline option either.
UPDATE
I've just rebuilt the dependency and I had to re-import it: the regular mvn clean install was not sufficient for me
I had the exact same problem. Running mvn clean install instead of mvn clean compile resolved it.
The difference only occurs when using multi-maven-project since the project dependencies are uploaded to the local repository by using install.
use <classpathentry kind="var" path="M2_REPO/your jar location
without creating any environment veriable
just to give my 2 cents. For me, it was only necessary to find the jar inside the ~/.m2/repository directory and use its version. So, after using
cd local-dependency/
mvn install
the local-dependency jar will be in the .m2/repository/com/your-organization/local-dependency/
~$ tree ~/.m2/repository/com/your-organization/local-dependency/
/home/felipe/.m2/repository/com/your-organization/local-dependency/
├── 0.1-SNAPSHOT
│   ├── maven-metadata-local.xml
│   ├── _remote.repositories
│   ├── local-dependency-0.1-SNAPSHOT.jar
│   ├── local-dependency-0.1-SNAPSHOT.pom
│   └── resolver-status.properties
└── maven-metadata-local.xml
then use it as a local dependency
<dependency>
<groupId>com.your-organization</groupId>
<artifactId>local-dependency</artifactId>
<version>0.1-SNAPSHOT</version>
</dependency>

Local Nexus Repository Acting Like A Proxy For Some Maven Artifacts

I am using my own version of the Nexus web app repository installed on my local machine. I have Nexus configured with only one repository, the one where I store my snapshots:
http://localhost:8081/nexus/content/repositories/MySnapshots/
Note that after the Nexus installation I removed all the default repositories and added just my own. (Perhaps this was a bad idea?)
When I do a mvn clean install I noticed that some of the 3rd party artifacts are downloading straight from the remote repository. For example, here is one of the output lines from the build:
Downloading: http://repo.maven.apache.org/maven2/com/sun/org/apache/xml/internal/resolver/...
The strange thing is that I see other artifacts are going through my local Nexus to ultimately get to the artifact:
Downloading: http://localhost:8081/nexus/content/repositories/MySnapshots/org/apache/maven/wagon/wagon-provider-api...
Notice how the first part of the download url is my local repository but everything after MySnapshots is from apache.org.
It's almost like my Nexus repository is acting like a proxy to maven.apache.org for some artifact downloads but for others it goes straight to the source.
Can anyone tell me why this is happening?
I would't be bothered so much by this if all my builds succeeded all the time but sometimes, when I am compile large projects, I get build failures due to not being able to find an artifact.
For example, when I try to build another project that depends on eclipse jdt stuff I get the following error:
Downloading: http://localhost:8081/nexus/content/repositories/MySnapshots/eclipse/jdt/core/eclipse.jdt.core
Could not find artifact eclipse.jdt.core:eclipse.jdt.core
I am not sure if this means that my Nexus is not configured properly or if there really is no artifact eclipse.jdt.com. If the downloads were not going through my local Nexus repository I would then investigate the pom/settings.xml files. Instead this makes me wonder if it's due to my Nexus configuration.
If you would like to see my settings.xml for Maven and my pom file for the project I am building when I see this you can view them here:
settings.xml: http://pastebin.com/NvLr5bEA
pom.xml: http://pastebin.com/PJ0P3RaK
If you like to use the local nexus as a proxy as usual than you have to configure the settings.xml like this:
<mirrors>
<mirror>
<!--This sends everything else to /public -->
<id>nexus</id>
<mirrorOf>*</mirrorOf>
<url>http://localhost:8081/nexus/content/groups/public</url>
</mirror>
</mirrors>
<profiles>
<profile>
<id>nexus</id>
<!--Enable snapshots for the built in central repo to direct -->
<!--all requests to nexus via the mirror -->
<repositories>
<repository>
<id>central</id>
<url>http://central</url>
<releases><enabled>true</enabled></releases>
<snapshots><enabled>true</enabled></snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>central</id>
<url>http://central</url>
<releases><enabled>true</enabled></releases>
<snapshots><enabled>true</enabled></snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
</profiles>
<activeProfiles>
<!--make the profile active all the time -->
<activeProfile>nexus</activeProfile>
</activeProfiles>
The tricky thing is the mirror thing which reroutes every call to the configured nexus instance.
Things you mentioned like eclipse parts can be problematic, cause only a few artifacts are available via maven central. Furthermore you should leave the defaults like maven central, release repository and the snapshots repository unchanged, cause these are the repository you need.
I don't think its a proxy issue , upto my understanding for the first case when it is downloading from Maven Central Repo , it might be possible that same artifact is not available in your nexus repository , that's why it is going to Maven Central Repo.
In the second case it is available in your nexus so reactor didn't try to download it from Maven Central Repo.

Missing of added jars to maven builth classpath via static in-project repository solution

One of the best strategies recommended for adding of JARs files in maven's project with no public repository is using of static in-project repository. Nice blog : http://charlie.cu.cc/2012/06/how-add-external-libraries-maven/ or one of the most popular as I found StackOverflow posts Can I add jars to maven 2 build classpath without installing them?
Does someone have had next issue when make totally same steps for integrating of the solution (my particular case is adding of filters-1.0.jar file): "Missing artifact imageUtil:filters:jar:0.1:compile" or in other words, nevertheless the static in-project repository is set up in pom.xml at the end also could not be recognized ? Does something is missed by me ?
Here are my steps integrating of the solution:
create a "repo" folder in root of my maven project
in pom.xml I registered static in-repository :
<repository>
<id>repo</id>
<releases>
<enabled>true</enabled>
<checksumPolicy>ignore</checksumPolicy>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
<url>file://${project.basedir}/repo</url>
</repository>
Again in pom.xml added mention dependency :
<dependency>
<groupId>imageUtil</groupId>
<artifactId>filters</artifactId>
<version>0.1</version>
</dependency>
last step is copy-pacing of mention JAR in next sub-folder structure
/repo/imageUtil/filters/0.1/filters-0.1.jar
When I checked in buildpath for the project (I am using Eclipse) I also saw that mention JAR is looking in my local repository or
- /home/simeon/.m2/repository/imageUtil/filters/0.1 (missing)
Maybe the question here is "how to say to my maven project that this JAR should be searched in my static in-project repository" ?
Thanks in advance,
SImeon
Your settings look fine. Is it never taking your in-project repository or do you mean it works first time and after that it will look only in the /home/simeon/.m2/repository/imageUtil/filters/0.1
I tried something similar and saw that if I give the name of a standard library which is available in a Maven Central repository, it looks there first and downloads it from there.
So I took a standard library (saxon) and renamed it to something different (eg: saxonic) which I know will not be available in any external repo, and I can see it uses my defined "inprojrepo" and installs this into my own local maven repository.
After that it uses from the local maven repository for next build.
[INFO] ------------------------------------------------------------------------
Downloading: file://D:\mymavenproject\someproject/inprojrepo/net/sf/Saxo
nic/9.4/Saxonic-9.4.pom
Downloading: http://repo.maven.apache.org/maven2/net/sf/Saxonic/9.4/Saxonic-9.4.
pom
[WARNING] The POM for net.sf:Saxonic:jar:9.4 is missing, no dependency informati
on available
Downloading: file://D:\mymavenproject\someproject/inprojrepo/net/sf/Saxo
nic/9.4/Saxonic-9.4.jar
Downloaded: file://D:\mymavenproject\someproject/inprojrepo/net/sf/Saxon
ic/9.4/Saxonic-9.4.jar (9560 KB at 29234.1 KB/sec)
[INFO]
My POM
<repository>
<id>inprojrepo</id>
<releases>
<enabled>true</enabled>
<checksumPolicy>ignore</checksumPolicy>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
<url>file://${project.basedir}/inprojrepo</url>
</repository>

Deploying Java Application to Heroku Results in jackson-mapper-asl Maven Error

The exact maven error message when deploying to heroku is:
No versions available for org.codehaus.jackson:jackson-mapper-asl:jar:[1.9,1.9.9] within specified range
I believe the issue is related to this answer and this Jackson repository maven-metadata.xml bug.
The following entry in the maven deploy log indicates that jackson-mapper-asl 1.9.9 is being downloaded from a heroku maven repository:
Downloaded: http://s3pository.heroku.com/jvm/org/codehaus/jackson/jackson-mapper-asl/1.9.9/jackson-mapper-asl-1.9.9.pom (2 KB at 12.3 KB/sec)
The heroku versions of the jackson maven-metadata.xml files are using incorrect version numbers - causing the maven error when using version ranges.
Is there a way, at deployment, to tell heroku to use a different maven repository for this dependency? Better yet, is there a way to get the heroku jackson-mapper-asl maven-metadata.xml file fixed?
Could this suffice as a workaround? You could just take the repository configuration section and point to somewhere else or supplying the file yourself as you would do with custom items.
Anyway, you can do it. Defining a regular repository in pom.xml, here the central one:
<repositories>
<repository>
<id>central</id>
<name>Maven Repository Switchboard</name>
<layout>default</layout>
<url>http://repo1.maven.org/maven2</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
Defining a repository in pom.xml that is pointing to a local "repo" item, which exists only within the application deployment:
<repositories>
<!--other repositories if any-->
<repository>
<id>project.local</id>
<name>project</name>
<url>file:${project.basedir}/repo</url>
</repository>
</repositories>
You should contact Heroku support about this in any case.
Follow Up I contacted Heroku support (super helpful, BTW) and they discovered the Heroku Maven cache was invalid for more than just Jackson. Heroku support kicked off an update process that helped resolve the problem but did not fix it completely. To get this issue completely resolved I had to add an explicit dependency to jackson-mapper-asl 1.9.9 in my pom.xml before I could deploy my app to Heroku.

Resources