I am getting ready to embark on a project mainly for experimenting with languages, but also with a hint of usefulness. It will consist of a server-application, written in Erlang, and client-libraries in a number of languages. Initially I will want to write clients in Java, Ruby and Python. The actual protocol for communication will be Thrift.
I'm looking for a build system that will allow me to build the server and all the client libraries in one go, running unit-tests in each language, then packaging up a releasable artifact of some sort in whatever way is the "standard" for each language.
That means a Jar for Java, a RubyGem and a distribute/setuptools tarball for Python. Erlang probably has something too, but I'm not yet familiar with that. It should also be able to run the Thrift compiler to generate the various Thrift-stubs in each language.
On the pad at the start is Maven. I'm fairly certain Maven can do all I need, but I fear it's too Java-centric, and leaves me with a ton of work for every new language I need to add.
well one should know what the requirements are for every language to create a deliverable artifact.
if copy files from here to there and zip it is enough maven could cover most of the scripting languages.
But you may end up writing plug-ins to support a custom packaging (which is not that complicated, so if there is no build system that may be a good choice).
It might not suit the build systems for every language to force maven upon them. So Maybe use the specific build tools available and wrap them in simple script and execute them using a continuous integration server (like bamboo, jenkings/hudson, teamcity, ...) to have them build in a specific order (to 'fake' dependencies)?
I'm not aware of a cross language system.
Gradle might be more flexible as its approach is more script oriented.
And there is http://eclipse.org/buckminster/ - just for completeness (havent had a look for quite some time)
regards
werner
"It will consist of a server-application, written in Erlang,
and client-libraries in a number of languages.
Initially I will want to write clients in Java, Ruby and Python. "
Maven is good if you follow its way. It is actually more of my way or high way.
See: http://community.jboss.org/wiki/MavenVsGradle
For a lot of standard java projects it is actually very good. But if you need to use other things it becomes fairly complicated very quickly.
From your description it is on its way to become complicated very quickly.
I suggest to look gant (groovy + ant) and gradle. You can call other scripts from gant and/or gradle. JPython, JRuby will be your friend. Ant has a lot of tasks which will be very useful.
I have successfully implemented complicated Java/C++/C build project using Gant. Groovy scripting is powerful and easy to use. Gradle is similar and in some ways more powerful than gant.
Related
My case is about having too much complex project configuration logic inside Jenkins jobs definition and in time this becoming harder and hard to deal with. This also prevents you from easily execute build jobs under other build/CI tools.
If those projects would be Java based anyone would probably tell me to use maven as I could put most of the things inside the pom.xml files and have them with the project. Still, in my case is more about C/C++ or even .NET projects for which the all the build scripts are usually in bash (cygwin being a dependency on Windows).
I do know that theoretically I could code the parts that are now inside jenkins job configuration in those bash files but this would clearly require significant effort and would be really hard to tune them to allow to enable and disable different steps based on external conditions.
So, what I am trying here is to achieve a high level of independency regarding the build system, so if I want I could switch it in the long future.
What would you recommend as a solution for that? Obviously I need something that can be used multiplatform and not tightened to a specific build system.
Does it make sense to use maven for that, even if those projects are not Java ones? Personally I am not a big fan of XML configuration files, YAML, JSON and INI being seen as more friendly.
What kind of extra logic existing in Jenkins configuration are we talking about?
One would deployment, as I want to be able to deploy to Nexus or similar repositories, executing tests, code coverage and maybe posting the results somewhere.
As a sidenote, looking at Travis configuration files makes me wonder why Jenkins didn't go for such approach.
Look at Groovy. Jenkins allows direct Groovy code manipulating almost everything. A Groovy script could be used to take care of everything from project specific configuration, and it could even be checked in together with the source code. Then in the Jenkins job, you just have a single build step to call the Groovy script.
The above suggestion, however, is very Jenkins dependent.
Another possibility is an Ant script. The AntExec plugin allows to execute Ant script, along with ant-contrib if needed, using the same tools installation process that the rest of Jenkins use. Therefore, you don't need to worry about Ant being installed on the node: Jenkins will take care of it on demand.
The benefits of the Ant script is that it's not tied to Java concepts as Maven is, it's cross platform (Windows and Linux), and just like the Groovy script example above, it can be checked in along with the rest of the source code.
I'm tasked with migrating our product's build/dependency-management technology from Buckminster to Maven. I'm quite decent with Maven, but before I drill down into Buckminster's inner-workings (which I'm not currently very knowledgeable on), I thought I'd see if there was an available tool to automate this, as it would be a rather lengthy task if done manually. If not, I guess I'll write one myself.
Any input would be much appreciated! Thanks!
I am not sure if the title of the question is correct, but please read the question.
I have been working on C/C++ for most part of my work life (close to 11 years). we only had C/C++ source/header files and all dependencies were managed by Makefiles. things were simple and manageable.
for the last 1.5 years i have shifted to Java domain. and i feel extremely irritated that most difficult aspect of working with anything new is the dependency managers. e.g. maven, leiningen, builder, sbt, etc etc etc.
whenever i download anything new from the open source world, there is a significant amount of time to be spent to just to setup the compilation, build, run environment. that too when i am using eclipse. why can't all the dependencies be placed along with the software to be downloaded?? why the tools like maven,leiningen, etc must make a separate internet connection to download the dependencies. i know that maven forms a local repository and should be able to find the dependency locally as it downloads whole internet anyway, but why is this model used. I am behind a firewall and not everything is accessible, and the tools fail to download dependencies. i am sure the same situation is there in most work environments.
recently i started with clojure, and boy it has been a pain to get eclipse configured for clojure. leiningen is supposed to be some magic which must be used with any clojure development. sometimes it feels learning leiningen is more important than learning concepts of clojure. i downloaded so called 'standalone' jar file for leiningen as 'self-install' was not working for me. but it fooled me. as soon as i run 'lein' command it is making an internet connection and trying to download somethings. WHY? it wont even print the help menu without connecting with the internet. WHY? there is no way i can fulfill its demands without bypassing my internet firewall, as i dont know, and no one can tell me what all things this guy wants. there is simply no other way.
And every one seems to be inventing their own. Java had ant which was simple, and went to Maven, some project uses Ruby based Builder, Clojure has leiningen, Scala has sbt. Go has something else. WHY? Why we need this added complexity in a world already full of complexity. why cant there be just one tool.
All you experts in Java technology please excuse my rant. I am sure this question will be downvoted and closed as from someone who is not trying hard enough to understand the things. But please believe me i have spend enough hours battling with this unnecessary complexity.
I just want to know how others get around this, or am i the only unfortunate one facing these issues.
I guess this question cannot accept an answer. I humbly can provide you with elements, hopefully they will help you get some perspective on the problem.
There are mainly 2 problems I identify with Java build systems:
some of them are declarative while others are using scripts
the fragmentation of the Java tools for building and exercising control is tied to people and Java stewardship of the space, not so much the technological choices.
Maven is the paramount of a method of defining your build using a formal grammar in a standard manner. Your pom.xml file contains a lot more than just your build : it is the identity of your artifacts, the project metadata, the modules and the plugins brought in. It treats with particular attention of the declaration of the dependencies and repositories.
Maven is declarative.
For a certain population of programmers, this is great, and they don't create new projects very often. It works well over time, it consolidates the build nicely.
Ant is a different system where you define tasks that will execute, chained in a particular order. All the definitions are made using XML and in effect, you are writing scripts and declaring how they will be stitched together.
Buildr (full disclosure: I am a committer there) is a build system which was created off the frustration of dealing with the inefficiency of the declarative approach for cases where the build needed to do additional steps and complex testing and the rigidity of using XML for a build. It is script-based, enforcing conventions over configuration (expecting a few good defaults, but letting you drive if you need to change things).
I am not familiar with Gradle and SBT but I think they extend and build on this approach, from what I heard.
So this gives you I hope a better picture of the landscape in terms of build tools.
The reason why no standard build tool emerged is probably tied to the fact Sun didn't push one with Java. Eventually, I think they adopted Ant (I have some most JSR jars being built with it). There also has been some products built in this space over extending some of those build systems ; there is always going to be a huge difference between people being paid to maintain code rather than doing it on the side.
And well, people argue. Build systems are a great way to start a flame war. We have a hard time agreeing on a standard though some of the common elements are now settling well around the Maven artifacts.
As for the need to download the Internet over and over again, it's a rather long story but here are a few things that may trigger the need for an unnecessary download:
any of the dependencies using SNAPSHOT will try to get the latest snapshot. This is a great scheme but it takes its toll. You might depend on something that depends on a snapshot, and get a download because of that.
Maven doesn't redownload the artifacts but sometimes checks md5. This is easy to fix, just use the -O option from the command line.
Tools like Buildr were built around fixing this issue once for all. First off, you only download what you said you would. Second, no connection is made again unless you asked for it. By default, Buildr doesn't play the transitive dependencies game though you can ask for it, but you have to do it explicitly.
I hope this was informative and that your journey in Java land becomes less painful going forward.
at my work we are presently looking for a automated build tool. We have googled quite a bit, but there are so many possibilities we are not sure which tool is the best option, to be honest.
We are not even sure there is one that fits our needs. It should have as many as possible from following features:
Integration with Subversion, meaning it should be able to do both updates and commits.
Diff tool or ability to integrate with one, and capability of auto-merging files (well, at least not the problematic ones).
Delphi build automation, using the command line compiler.
Integration with Oracle, meaning having the ability to run SQL scripts and compiling PL/SQL packages, functions, etc.
Sending of e-mails with build status (I guess this is something all of them have).
Additional functionalities would be welcome but are not needed at the moment. Anything in the line of getting at least points 1 and 2 would be great.
If anyone could point us to some particular tool we would be very grateful.
Thanks in advance and best regards
For my Delphi projects, I was using FinalBuilder (itself a Delphi app) way back when. I was always very happy with the product, and it worked very well.
In the meantime, they also launched a Server version of the build tool, so you can set up a full build server and have your code built there.
Another worthy contender back in those days used to be Visual Build - feature by feature the two almost match up, but the "feel" of the app, the ease of use, was with Finalbuilder, in my personal opinion.
I'm not sure if any of the two has direct and specific Oracle support, but at least Finalbuilder can run anything that can be run from the command line, so that should give you plenty of options, really.
I've been using CruiseControl.NET as a Continuous Integration server for our Delphi builds, which integrates quite nicely with NAnt as a build scripting language.
Forgetting about CC.NET, simply using NAnt would provide most/all of what you need, I think.
Combined with the extra NAnt-Contrib library of extra tasks, I have implemented everything you mention with the exception of the Oracle integration. However, you can easily invoke commandline stuff from within NAnt scripts, or write your own NAnt tasks using any .NET language.
NAnt has tasks for integrating with Subversion, or you can simply invoke the svn command line client.
I have invoked BeyondCompare as a diff tool from within my NAnt scripts, used NAnt to send customised HTML emails and execute Installshield builds, along with obviously executing the commmand line Delphi compiler and resource compiler.
NAnt is free and open-source, it's worth checking out.
Check also Automated Build Studio from AutomatedQA, it has many of the features you ask for.
We use FinalBuilder to handle requirements 1, 3, 4 and 5 and Beyond Compare to handle requirement 2 and have been very happy with both of these tools.
Apache Ant is open source and contains many tasks out of the box, and a very feature complete scripting and dependency management environment.
Subversion integration add-on: subclipse
SQL core task is included
Apply task can run any executable on any file pattern (like *.dpr)
Mail task included
Diff Patch task included
Also available: zip/unzip, tar/untar, xmlvalidate, xslt, ftp, telnet, scp, sshexec, cab, ...
I'm surprised no one has mentioned Hudson. Most build tools seem to target a specific language or platform. Hudson takes a different approach. It provides a plugin architecture so you can create a build server that fits your needs. The project itself is written in Java but is entirely self contained (with the exception that it depends on the JRE of course). It uses its own built-in web server to provide a user interface.
Embarcadero uses Hudson extensively and there are several great how to articles to get you started
http://christerblog.wordpress.com/2010/04/25/using-hudson-to-build-delphi-projects/
http://www.nickhodges.com/post/Getting-Hudson-set-up-to-compile-Delphi-Projects.aspx
And if you need commercial support the original author has started a company that provides a custom version as well as technical support.
Right now a project I'm working on has reached a level of complexity that requires more than a few steps (actually its become arcane!) to produce a complete/usable product. And unfortunately we didn't start out with a Continuos Integration mindset, so as you can imagine its kind of painful at times, and at others I can easily waste half a day trying to get a clean/tested build.
Anyways as any HUGE project it consists of many components in many different languages (not only enterprise style Java or C# for example), as well as many graphical, and textual resources. Now the problem is that when I look for Continuos Integration, I always find best practices and techniques that assume one is starting a new project, from the ground up. However this isn't a new project, so I was wondering what are some good resources to proactively start migrating from Arcane Integration towards Continuos Integration :)
Thanks in advance!
Here it is in two simple (hah) steps.
Go for the repeatable build:
Use source control, get all code checked in.
Establish and document all tools used to build (mainly, which compiler version). Have a repeatable deployment and set up process for these tools.
Establish and document clearly any resources which are necessary to build, but are not checked in (third party installations, service packs, etc). Have a repeatable deployment and set up process for these dependencies.
Before commiting to source control, developers must
update their working copy
successfully build
run and pass automated tests
These steps can be done 1 at a time, sort of a path to follow. You'll get benefits at each stage. For example, if you aren't using source control at all, just getting the code into source control (without anything else) is a big step forward. Also, if there are no automated tests, then developers can't run them - but they can still get the prior commits and get the compiler to check their work.
If you can do all of these, you'll get to a nice sane place.
The goals are repeatable build processes and developers that are plugged in to how their changes affect the build and other developers.
Then you can reap the bonuses by establishing higher compliance:
Developers establish a frequent commit habit. Code that is in the working copy should never be more than 1 day old.
Automated build process monitors source control for check-ins and gets the results to a place where the users can accept them (such as a test environment, a preview website, or even simply placing an .exe where the user can find it).
The same way you eat an elephant (one bite at a time) ;-) Continuous integration requires an automated build. Start with that. Automate the building of each piece. Ant or NAnt is a great way to do this. Have each component's construction be a NAnt task. Then your entire system build can aggregate those individual tasks.
From there, you can add tasks for deployment, for unit testing, etc. If you want to use a CI technology, you can wire it up to your NAnt build.
I would start by first writing down all the steps it takes you to do the build and test manually. After that you at least have a guide for doing it the old way, and writing things down gives you the chance to look at it as a complete process.
Then look for parts to script.
Ideally you want to trigger a build and test from a code commit and only rebuild and retest the changed parts, with perhaps a full build and test nightly or weekly. You'll need log files or database entries and reports on the build success or lack of it.
You'll want to search out and evaluate pre-built products and open-source build-your-own kits. You can certainly write all the scripting and reporting yourself, but it will take a while and you'll probably end up with a just barely good enough reporting system since your job is coding the product, not coding the build system. :-)
I would guess that migrating isn't really an option--Half-ass solutions will only make it worse.
My approach would be to take one creative engineer who understands the build process, sit him down and say "Fix this". Give him a week or two.
The end goal would be a process that runs beginning to end with a single make command.
I also recommend an automated "Setup" procedure where you simply do a checkout and run a batch file from a network share to install and build all your tools. The amount of time this will save overall is staggering if you bring in new programmers. Most projects take one to three days to get set up on a new computer--and it's always the "new" programmer who doesn't know what's going on doing the installs on his own system...
In short: Incrementally
Choose a framework that will work across the diverse range of projects.
One by one, add components to the framework.
If you are not familiar with the framework, tackle a couple of the easier components first, to reduce risk of screwing up.
If you do understand the framework, tackle some of the more difficult and/or commonly built components first, so your team (and management) will appreciate the benefits early, and support the effort more.
Be sure to have a plan to include all of your components, because that's when the full benefit will be realized.
Bring your team with you; make sure you have consensus that this is going to be valuable, or people won't maintain it as the components change.