We are developing offline due to limited internet resources and would like to run once every several months a whole grab of an external repository (e.g repo1.maven.org/maven2 - Disk space isn't an issue).
Today I'm using a simple POM that contains a lot of common dependencies that we are using, I've set my local maven to use a mirror to proxy thru a local nexus repository to cache locally and this is how I'm grabbing for offline use - but that isn't very effective.
I'm now looking for a command line tool that allow me to run searches on maven repositories so that I can write a script that grab them all to my local nexus installation and would like to hear if there is any or if there is another way to achieve that.
Thanks
Not a whole solution (yet) but I'm using httrack to grab the whole content of repo1.maven.org/maven2 - That is already better than nothing :)
In general, there is a goal in Maven dependency plugin called "go-offline"
So it allows to grab all the project dependencies and to store them in local .m2 repo.
You can find more information here.
If you want to run Maven and tell it to behave like the network does not exist you can run it with "-o" option (offline mode). So that if there is no dependency installed locally, Maven won't even try to go to network and bring it - but will fail the build.
On the opposite, if you want to force Maven to check and bring new versions (otherwise they already should be in your repo), you can use "-U" option.
I'm not really sure I've got the point about general search-and-download use case. Usually people install Nexus or Artifactory once in a network so that each dependency will be downloaded only once. In local development machines people usually just work with filesystem and don't maintain tools like this.
Now if you want to copy the whole repository from internet (for copying it later to some other network or something) you can just use crawlers like Apache Nutch for example or craft your own script that will recursively download all the files.
Related
We currently use SilkCentral Test Manager (SCTM) integrated with our source control system via SCTM source control profiles. However, we would like to explore integrating with build artifacts checked into Maven's remote Nexus repository instead.
The idea being that the application-under-test is built and checked into Nexus along with the automated tests only if the build and the tests pass. Therefore, when QA is ready to run tests from SCTM (manual or automated), there is a well-defined combination of application build artifacts and test build artifacts in Nexus that present a more reliable target for SCTM as compared to getting the latest code from the source control system.
All of this is more relevant during active development when the code and the tests and changing daily and the builds are snapshot builds rather than formal builds with tags in the source control system that SCTM could use.
SCTM apparently has support for both universal naming convention (UNC) and Apache virtual file system (VFS) and either of these should potentially be utilizable to point the SCTM source control profiles to Nexus artifacts rather than raw source code. However, I wanted to check with the community to see if there's a simpler approach. (For example, I noted the existence of a Hudson SCTM plugin.) Also, I welcome alternative thoughts and ideas.
There are probably many solutions for solving this, I'd try the following:
Manage the build/first test/publishing steps in Hudson/Jenkins.
For example by modelling it with dependent jobs, the publish job is only triggered if the tests pass. There are also more advanced gatekeeper plugins available (for example a Downstream Ext plugin) which might solve this even more comfortable.
Once the publishing is done, use the Hudson/Jenkins-Silk Central plugin to trigger the executions on Silk Central. There, instead of using UNC or VFS, I'd rather use a setup script which pulls the artifacts from the repository and prepares everything for the tests. This would allow you to use something Maven/Nexus aware to pull the correct artifacts from the repository, instead of somehow trying to make it accessible via UNC or VFS.
Well, this is kind of embarrassing. I am in the process of mavenizing our build processes and just don't know how the access the result of a build. I build, let's say, a jar file and mvn deploy it. So it ends up as some blah-0.1.2.jar in our company maven repository, which is just a webdav share. Now how would you pass that on to someone else to use? Just pry it from target/blah-0.1.2.jar can't be the answer. I found several suggestions to use variants of mvn dependency:get but they were all just close and didn't feel right. There must be a way to use all those nice versions of blah-*.jar that end up in the repository for purposes other than a maven dependency. Preferably from the command line and maybe even without maven. Hm, a webdav client doesn't look too bad except for snapshots. What would you suggest?
Creating a script that makes a dependency:get call is probably going to be closest to your desired outcome. You can specify the destination of your downloaded jar this way.
If you are looking for an easy way to share builds between people in/outside of your company then you can look into setting up some automated build software like Bamboo or something similar. A new build gets triggered any time a commit is made to the section where your project resides in whatever version control system you use. An artifact is then made available for each successful build and are available via Bamboo's web interface. Bamboo can be configured to run with your maven pom's.
While they can bit a bit of pain to set up, going the automated build route will take a lot of the sting out of sharing your builds in the future.
As a consultant, I have multiple clients that I'm doing work for. Each client utilizes their own internal Maven repository that is also set up to mirror Central and other external repositories. I need to configure my maven installation on my laptop so that when I'm doing work for one client, it utilizes their internal repository for everything.
I had thought I would be able to utilize profiles to handle this, but mirror settings cannot be changed per-profile.
Does anybody have suggestions on how to approach this maven configuration?
Note: A similar question is here: How do I configure maven to access maven central if nexus server is down?, but that question deals with switching between Central coming from a mirror or not. What I need is for Central (and others) to come from one mirror or a different one based on some property/setting/variable etc.
Create two shell aliases:
alias build_at_home="mvn -s $HOME/.m2/home_settings.xml"
alias build_at_work="mvn -s $HOME/.m2/work_settings.xml"
The "-s" option is handy for explicitly stating which environment settings file to use.
We use this mechanism on our shared build server to ensure each project build is isolated.
Obviously on windows you could create a set of batch files.
I have a very similar requirement in my project too. I created two separate settings.xml files, named them as settings_one.xml and settings_two.xml and saved them in the MAVEN_HOME. Depending on which file I need, I have a small script (a bat file on windows) which overwrites the existing settings.xml with one of the two settings files.
del C:\Users\<username>\.m2\settings.xml
copy C:\Users\<username>\.m2\settings_one.xml C:\Users\tadigotl\.m2\settings.xml
The simplest solution i can suggest is to install git and commit you .m2/settings.xml (of course ignore the repository itself via .gitignore) into git and make appropriate branches for the customers. Change the settings will be done by:
git checkout CUSTOMER_BRANCH
and furthermore any change is tracked by a SCM.
I want to deploy a generated Maven AppAssembler assembler/ directory to somewhere in a file system, SSH, or whatnot. Can Cargo do that for me, or is there an equivalent deployment tool that will let me glob a bunch of files (in this case the target/appassembler/ directory) and deploy them to a destination?
I have a couple command-line applications that run as scheduled tasks (via cron or Windows Scheduler), and I want to deploy them out to these remote locations (in one case via SSH, and another a network share \\servername\C$\whatever\). I don't know how I can accomplish that, since all of the deployment plugins I have been looking at cater to web applications and app containers or Remote repos like Nexus.
Try maven copy plugin - it has excellent networking support (scp,FTP,HTTP).
You might also find useful maven sshexec plugin.
I know this question is quite old, but since someone else might also be interested in this:
I don't have a complete/concrete example for this, since I never tried it, but maybe the maven assembly plugin could be used for this, with the dir assembly format?
We have to setup new build environments regularily, and the process seems not so simple. Today I have got a new build machine, and the first Maven build was so slow, that I wanted to clarify why the performance was so bad. But how to do that?
Our context is:
We use multiple build machines, each project gets its own.
Each build machine has a similar setup, so that projects can start immediately and don't have to configure a lot.
We have the following tools preconfigured:
Hudson (currently 2.1.1, but will change)
Artifactory 2.3.3.1
Sonar
Hudson, Artifactory and Sonar have their own Tomcat configured
Maven 2.2.1 and Maven 3.0.3 (with no user configuration, only the installation has a settings.xml)
Ant 1.7.1 and Ant 1.8.2 (not relevant here)
Subversion 1.6 client
All tools should work together, especially the repository chain should be:
Build machine Maven repository
Build machine Artifactory
Central company Artifactory (is working as mirror and cache for the world)
Maven central (and other repository)
So when the Maven build needs a dependency resolved, it will be first looked-up in the local Maven repo, from there in the local Artifactory repo, then in the central Artifactory repo and only then on the internet.
We normally have to use proxies to connect to the internet, we don't need it in our intranet.
The first build (Maven Hello World) was built in around 45 minutes. In that time, all bootstrapping was happening, but I would have thought by using our chain of repositories (where the central repository is well filled), the build would be much faster. So I think the focus of the debugging will be the network, the local build is not the problem. So configuration and interaction of Maven and Artifactory is under consideration.
How do you debug such an environment? I have access to the build machine (as sudo) and to the central repository, but I do not know how to start, what to prove, where to look. So what is your experience, what are the tips and tricks you would like to share?
Here are a few things I have done up to now. If you have additional advice, you are welcome!
I suspected the chain of repositories to be the source of evil, so I addressed that first. The reasons are:
The real build on the local machine (of a hello world program) may differ in milliseconds, but not minutes.
Network makes a difference, so attack that first.
The chain of repository is interesting, if something is not found locally. Here are the steps to ensure that that is the case:
For Maven, delete the contents of the local cache. If the local cache is filled, you don't know if a resource if found in the local cache or elsewhere. (Do that at least at the end, if everything else is working again.)
For Artifactory, find that cache as well, and clean it by deleting its contents. It is only a cache, so it will be filled a new.
If you use a clever browser for measuring the lookup, ensure that what you asked for is not in the cache of the browser.
Else use a tool like wget to ask for a resource.
Try to minimize the sources for failure. So try to divide the long distance of your lookup in smaller segments that you control.
Don't use Maven for doing the lookup, start first with the Artifactory repository (only), and later then with Maven.
This led to the following tests I wanted to do. Every time I ensured that the previous prerequisits were met:
Ask for https://<my-project-artifactory>/repo/<my-pom>. Expectation:
Local lookup will fail, so has to find the resource in a remote repository in the central company Artifactory.
Possible effects could come from proxy, artifactory lookup.
Result: Lookup for a simple POM needed ~ 30 seconds. That is too much.
Remove the proxy. With wget, there is an option --no-proxy which does just that. Expection:
Faster lookup.
Result: No change at all, so proxy was not the reason.
Ask for https://<my-project-artifactory>/libs-snapshots-company/<my-pom>. So change the virtual repository to a real remote repository. Expectation:
Artifactory knows where to do the lookup, so it will be much faster.
Result: POM was found immediately, so the 30 seconds are Artifactory doing lookup. But what could be the reason for that?
Removed in Artifactory all remote and virtual repositories (only left our companies ones and the cached Maven central). But use again https://<my-project-artifactory>/repo/<my-pom>. Expectation:
Artifactory will find the repository much faster.
Result: POM came in an instant, not measurable.
I was then courageous and just started the build (with empty cache locally). The build needed then 5 seconds (instead of 15 minutes the same morning).
So I think I have now better understood what can go wrong, a lot of questions are remaining. Please add your ideas as answers, you will get reputation for them!