We are using SemaphoreCI as continuous-integration. Some of our maven dependencies are pretty big (like order of hundreds of megabytes).
A behavior that we are observing is that, our builds fails often due to failures in downloading the dependencies; we have to restart it once or twice to make it pass, in order to move on, which is really annoying.
I wonder if anyone has any suggestions/experiences regarding this problems.
Milana from SemaphoreCI here.
It would be good to cache these dependencies between builds. If your project is recognized as a Maven project on Semaphore, related dependencies will be cached by default. For such a project Semaphore caches ~/.m2 directory. To read more about this please refer to this page.
If you're still experiencing similar issue or if you would like to learn more about SemaphoreCI, please send us an email at support#semaphoreci.com and we'll do our best to help you out!
Related
I am suffering the slow download of maven.
I use the default configuration for Spring Boot starter project.
Group: com.example
Artifact: Demo
STS takes too much time to sync the content. is there a way to speed this up? for example, download the whole thing separately that the project needs, and then import them locally.
If you look carefully at the screenshot you'll notice that it says that Maven download operation is blocked by user operation.
The comments saying the build is slower under IDE (particularly Eclipse) are wrong in the sense that they are sort of a mental shortcuts. They are based on observation (it may indeed take longer time to achieve the end result) but that does not mean the build/download itself is slower. The thing is that Eclipse performs way more operations than just the build alone and sometimes those end up waiting on one another (as your screenshot clearly indicates).
With that in mind, if you run your build on the command line it may complete a lot faster as most likely it will not compete for resources with other tasks. But keep in mind that will keep Eclipse out of sync with what is actually on the file system. Eventually Eclipse will figure that out and try to sync. Sometimes it may not and you would have to do it manually. In both cases, depending on the size and amount of the projects and the number and complexity of changes made, it may take significant time to sync.
To summarize, it's not "slow download of maven" what you experience but multiple tasks competing for resources and waiting on each other. There is no point to pre-download all dependencies as this is not a recurring operation. Maven ONLY downloads missing dependencies. Once they are in local repo it will not try to download them again (unless you force it to).
Regardless of build tool, I have seen people doing a clean task/phase before every time they do package/compile or ... is it really necessary?
Dose build tools use reuse artifacts of previous builds?
Most of the time you see clean install as the default command, but I would encourage everybody to use verify instead.
When executing clean the target-folder is removed which makes it impossible to do incremental builds. Plugins have enough information to detect if they should do their action. For instance: the maven-compiler-plugin compares the java sourcesfiles and the compiled classfiles (and other things) to see if files needs to be (re)compiled. If you think that a plugin is not working correctly with incremental builds, please file an issue for that plugin.
The install was often required with multimodules in Maven2, but Maven3 is capable to resolve these inner module dependency references. The only thing 'install' does is copying artifacts to the local repository (=IO=expensive). And it'll make your local repo look different compared to your coworkers, which might give different results during builds. Better to let a buildserver push those artifacts to the shared remote repositories and let every pull those SNAPSHOTs from there. Only in rare cases calling install is valid (experienced Maven users know when :) ), so instead please use verify.
I work with a small team that manages a large number of very small applications (~100 Portlets). Each portlet has its own git repository. During some code I was reviewing today, someone made a small edit, and then updated their pom.xml version from 1.88-SNAPSHOT to 1.89-SNAPSHOT. I added a comment asking if this is the best way to do releases, but I don't really know the negative consequences of doing this.
Why not do this? I know snapshots are not supposed to be releases, but why not? What are the consequences of using only snapshots? I know maven will not cache snapshots the same as non-snapshots, and so it may download the artifact every time, but let's pretend the caching doesn't matter. From a release-management perspective, why is using a SNAPSHOT version every time and just bumping the number a bad idea?
UPDATE:
Each of these projects results in a war file that will never be available on a maven repo outside of our team, so there are no downstream users.
The main reason for not wanting to do this is that the whole Maven eco-system relies on a specific definition of what a snapshot version is. And this definition is not the one you're setting in your question: it is only supposed to represent a version currently in active development, and it is not suppose to be a stable version. The consequence is that a lot of the tools built around Maven assumes this definition by default:
The maven-release-plugin will not let you prepare a release with a snapshot version as released version. So you'll need to resort to tagging by hand on your version control, or make your own scripts. This also means that the users of those libraries won't be able to use this plugin with default configuration, they'll need to set allowTimestampedSnapshots.
The versions-maven-plugin which can be used to automatically update to the latest release version won't work properly as well, so your users won't be able to use it without configuration pain.
Repository managers, like Artifactory or Nexus, comes built-in with a clear distinction of repositories hosting snapshot dependencies and release dependencies. For example, if you use shared Nexus company-wide, it could be configured to purge old snapshots so this would break things for you... Imagine someone depends on 1.88-SNAPSHOT and it is completely removed: you'll have to go back in time and redeploy it, until the next removal... Also, certain Artifactory internal repositories can be configured not to accept any snapshots, so you won't be able to deploy it there; the users will be forced, again, to add more repository configuration to point at those that do allow snapshots, which they may not want to do.
Maven is about convention before configuration, meaning that all Maven projects should try to share the same semantics (directory layout, versioning...). New developers that would access your project will be confused and lose time trying to understand why your project is build the way it is.
In the end, doing this will just cause more pain on the users and will not simplify a single thing for you. Probably, you could make it somewhat work, but when something is going to break (because of company policy, or some other future change), don't act surprised...
Tunaki gave a lot of reasonable points why you break Maven best practices, and I fully support that view. But even if you don't care about "conventions of other companies", there are reasons:
If you are not doing CI (and consider every build as potential release), you need to distinguish between versions which should go productive and those who are just for testing. If everything is SNAPSHOT, this is hard to do.
If someone (accidentally) deploys a second 1.88-SNAPSHOT, it will be the new 1.88-SNAPSHOT, hiding the old one (which is available by a concrete timestamp, but this is messy). Release versions cannot be deployed twice.
By default, Go pulls imported dependencies by grabbing the latest version in master (github) or default (mercurial) if it cannot find the dependency on your GOPATH. And while this workflow is quite simple to grasp, it has become somewhat difficult to tightly control. Because all software change incurs some risk, I'd like to reduce the risk of this potential change in a manageable and repeatable way and avoid inadvertently picking up changes of a dependency, especially when running clean builds via CI server or preparing to deploy.
What is the most effective way I can pin (i.e. lock down or capture) a package dependency so I don't find myself unable to reproduce an old package, or even worse, unexpectedly broken when I'm about to release?
---- Update ----
Additional info on the Current State of Go Packaging. While I ended up (as of 7.20.13) capturing dependencies in a 3rd party folder and managing updates (ala Camlistore), I'm still looking for a better way...
Here is a great list of options.
Also, be sure to see the go 1.5 vendor/ experiment to learn about how go might deal with the problem in future versions.
You might find the way Camlistore does it interesting.
See the third party directory and in particular the update.pl and rewrite-imports.sh script. These scripts update the external repositories, change imports if necessary and make sure that a static version of external repositories is checked in with the rest of the camlistore code.
This means that camlistore has a completely repeatable build as it is self contained, but the third party components can be updated under the control of the camlistore developers.
There is a project to help you in managing your dependencies. Check gopack
godep
I started using godep early last year (2014) and have been very happy with it (it met the concerns I mentioned in my original question). I am no longer using custom scripts to manage the vendoring of dependencies as godep just takes care of it. It has been excellent for ensuring that no drift is introduced regardless of timing or a machine's package state. It works with the existing mechanism of go get and introduces the ability to pin (godep save) and restore (godep restore) based on Godeps/godeps.json.
Check it out:
https://github.com/tools/godep
There is no built in tooling for this in go. However you can fork the dependencies yourself either on local disk or in a cloud service and only merge in upstream changes once you've vetted them.
The 3rd party repositories are completely under your control. 'go get' clones tip, you're right, but you're free to checkout any revision of the cloned-by-go-get or cloned-by-you repository. As long as you don't do 'go get -u', nothing touches your 3rd party repositories already sitting at your hard disk.
Effectively, your external, locally cloned, dependencies are always locked down by default.
It is around 10 months now that Jenkins split off from Hudson.
When looking at the project homepages I am wondering what the differences between Hudson and Jenkins in the meantime really are. From the changelog I do not realy learn much. There are a bunch of changes and the major difference seems to be that Jenkins releases more often with less changes and Hudson less frequently, but then with more changes in a release.
Are there any notable differences yet?
So are there things that make me as a developer needing a CI system more productive rather with the one or the other?
Is one of them more stable than the other?
Is there any difference yet that has nothing to do with politics around Oracle?
What is the most notable difference from your point of view?
One notable difference is that a big number of plugins moved to Jenkins. While you would still be able to use the old versions with Hudson, the newer versions depend on Jenkins already. Also new plugins are mostly created with dependencies on quite recent Jenkins versions, so you probably won't be able to use them without hassle on Hudson.
This will probably differ from plugin to plugin, some might be more compatible with Hudson than others, while still others provide versions for both tools. But if something does not work well with a plugin you will receive help easier if you use Jenkins.
EDIT: Here is an interesting link I found, not only providing some solid numbers on the different paths Jenkins and Hudson have taken, but also addressing the (non-)issue of IP that was mentioned in the other post here...
check out the work being done on cleaning up the code and the IP checks that are needed to belong to Eclipse Foundation. This is one of the big differentiators if you care about clean IP.
How many plugins are you using? Hudson supports many of the most important plugins independently and is working with plugin owners to keep compatibility with those that are still maintained by their owners at Jenkins.
See the JavaOne presentations that show how Hudson is being maintained and new features added.
https://oracleus.wingateweb.com/scheduler/eventcatalog/eventCatalogJavaOne.do (search for Hudson)
Also check out the Hudson project at Eclipse http://www.eclipse.org/hudson/