Our development team has been fairly small and, until now, all working on a single Visual Studio 2012 solution. We are growing and wanting to create better separation with multiple solutions for different project teams.
However, there are occasions where the code in one solution will want to utilize code from another. We have decided using internal (i.e. private) NuGet packages will be a good way to manage these dependencies.
However, the question has come up on how to deal with multiple versions of the same package that are in different SDLC stages (e.g. Development, QA, Staging, Production, etc.)
Example: If we have these three solutions...
CoreStuff
CoolProject1
CoolProject2
If working in CoolProject1, and we need to utilize code from CoreStuff, we can add the NuGet package. Presumably this package will be the latest Production (stable) version of CoreStuff.
However, what if a developer working on CoolProject2 is aware of some changes in CoreStuff that are currently in Development and wants to utilize that version?
Not sure if the best approach is to create separate packages for each (seems to require changing your package references back and forth depending on what stage the solution is in) or somehow utilize multiple versions of the same package (not sure if that's easy to manage with NuGet).
Anyone tackle something like this?
The first thing to remember is that NuGet will not automatically update your package references, so if you have already 'linked' your solution to the latest stable package of CoreStuff (say 1.2.2) then there won't be any problems if a newer (unstable) version is provided (assuming that the package you're using doesn't disappear from the package repository). Obviously if you upgrade your package reference then you will get the unstable package.
So the simplest solution is to make sure that you 'link' your project to the stable package by getting it via the NuGet package manager before the other package is released. While the UI only allows you to get the latest version, the Package Manager Console can get any version of a package so you could use that to explicitly provide the version number, e.g.:
Install-Package CoreStuff -Version 1.2.2 -Project CoolProject1
If that is not a solution then there are several other options to tackle this problem:
Give the development version a different semantic version that indicates it is a unstable version, e.g. 1.2.3-alpha. In this case CoolProject1 could pull in package CoreStuff.1.2.2 (which should be latest stable version in your repository) and CoolProject2 could pull in CoreStuff.1.2.3-alpha (which would be the latest unstable version).
Have multiple repositories, e.g. one for stable (released) packages and one for unstable (development) versions. Then you can select your packages from the desired repositories. If you wanted to you could make it so that only your release process can push packages up to the stable repository and your CI build pushes up to the unstable one (so that you always have the latest packages available)
If the developer of CoolProject2 just wants to develop against the latest version (but will wait to release CoolProject2 until after CoreStuff v.next has been released) then he could potentially create a local package repository (i.e. a directory on his drive) and put the new package of core stuff there. That way other developers won't even see the package.
The most important thing will be to make sure that you don't get CoreStuff.1.2.2 and CoreStuff.v-next in the same repository if CoreStuff.v-next simply has a higher version number, because in that case the NuGet UI won't let you pick v1.2.2 (but the Package Manager Console does!).
If you would want to switch from one package type to another you'd have to do a manual update (which you always have to do when changing to the next package version anyway), but that's not a bad thing given that this forces a developer to at least check that the update of the package doesn't break anything.
Related
I am looking for a tool to manage the collection of binary files (input components) that make up a software release. This is a software product and we have released multiple versions each year for the last 20 years. The details and types of files may vary, but this is something many software teams need to manage.
What's a Software Release made of?
A mixture of files go into our software releases, including:
Windows executables/binaries (40 DLLs and 30+ EXE files).
Scripts used by the installer to create a database
API assemblies for various platforms (.NET, ActiveX, and Java)
Documentation files (HTML, PDF, CHM)
Source code for example applications
The full collected files for a single version of the release are about 90MB. Most are built from source code, but some are 3rd party.
Manual Process
Long ago we managed this manually.
When starting each new release the files used to build the last release would be copied to a new folder on a shared drive.
The developers would manually add or update files in this folder (hoping nothing was lost or deleted accidentally).
The software installer script would be compiled using the files in this folder to produce a SETUP.EXE (output).
Iterate steps 2 and 3 during validation & testing until release.
Automatic Process
Some years ago we adopted CI (building our binaries nightly or on-demand).
We resorted to putting 3rd party binaries under version control since they usually don't change as often.
Then we automated the process of collecting & updating files for a release based on the CI build outputs. Finally we were able to automate the construction of our SETUP.EXE.
Remaining Gaps
Great so far, but this leaves us with two problems:
Rebuilding Assemblies The CI mostly builds projects when something has changed, but when forced it will re-compile a binary that doesn't have any code change. The output is a fresh build of a binary we've previously tested (hint: should we always trust these are equivalent?).
Latest vs Stable Mostly our CI machine builds the latest versions of each project. In some cases this is ok, but often we want to release an older tested or stable version. To do this we have separate CI projects for the latest and stable builds - this works but is clumsy.
Thanks for your patience if you've got this far :-)
I Still Haven't Found What I'm Looking For
After some time searching for solutions it seems it might be easier to build our own solution, but surely someone else has solved these problems before!?
What we want is a way to store and manage binary files (either outputs from CI, or 3rd party files) such that each is tagged with a version (v1.2.3.4) that allows:
The CI to publish new versions of each binary (but reject rebuilt versions that already exist).
The development team to make a recipe for a software release (kinda like NuGet packages.config) that specifies components to include:
package name
version
path/destination in the release folder
The Automatic package script to use the recipe collect the required files, and compile the install package (e.g. SETUP.EXE).
I am aware of past debates about storing binaries in a VCS. For now I am looking for a better solution. That approach does not appear ideal for long-term ongoing use (e.g. how to prune old binaries)... amongst other issues.
I have tried some artifact repositories currently available. From my investigation these provide a solution for component/artifact storage and version control. However they do not provide tools for managing a list of components/artifacts to include in a software release.
Does anybody out there know of tools for this?
Have you found a way to get your CI infrastructure to address these remaining issues?
If you're using an artifact repository to solve this problem, how do you manage and automate the process?
This is a very broad topic, but it sounds like you want a release management tool (e.g. BuildMaster, developed by my company Inedo), possibly in conjunction with a package management server like ProGet (which you tagged, and is how I discovered this question).
To address some specific questions you have, I'll associate it with a feature that would solve the problem:
A mixture of files go into our software releases, including...
This is handled in BuildMaster with artifacts. This video gives a basic overview of how they are manually added to releases and deployed to a file system: https://inedo.com/support/tutorials/buildmaster/deployments/deploying-a-simple-web-app-to-iis
Of course, once that works to satisfaction, you can automate the import of artifacts from your existing CI tool, create them from a BuildMaster deployment plan itself, pull them from your package server, whatever. Down the line you can also have your CI tool call the BuildMaster release management API to create a release and automatically have it include all the artifacts and components you want (this is what most of our customers do now, i.e. have a build step in TeamCity create a release from a template).
Rebuilding Assemblies ... The output is a fresh build of a binary we've previously tested (hint: should we always trust these are equivalent?)
You can mostly assume they are equivalent functionally, but it's only the times that they are not that problems arise. This is especially true with package managers that do not lock dependencies to specific version numbers (i.e. NuGet, npm). You should be releasing exactly the same binary that was tested in previous environments.
[we want] the development team to make a recipe for a software release (kinda like NuGet packages.config) that specifies components to include:
This is handled with releases. A developer can choose its name, dates, etc., and associate it with a pipeline (i.e. a set of testing stages that the artifacts are deployed to), then can "click the deploy button" and have the automation do all the work.
Releases are grouped by "application", similar to a project in TeamCity. As a more advanced use case, you can use deployables. Deployables are essentially individual components of an application you include in a release; in your case the "Documentation" could be a deployable, and maybe contain an artifact of the .pdf and .docx files. Deployables from other applications (maybe a different team is responsible for them, or whatever) can then be referenced and "included" in a release, or you can reference ones from a past release.
Hopefully that provides some overview and fits your needs. Getting into this space is a bit overwhelming because there are so many terms, technologies, and methodologies, but my advice is to start simple and then slowly build upon it, e.g.:
deploy a single, manually uploaded component through BuildMaster to a share drive, then manually deploy it from there
add a deployment plan that imports the component
add a second plan and associate it with the 2nd stage that takes the uploaded artifact and deploys it to the target, bypassing the need for the share drive
add more deployment plans and associate them with pipeline stages and promote through them all to "close out" a release
add an agent and deploy to that instead of the default localhost server
add more components and segregate their deployment with deployables
add event listeners to email team members at points in the process
start adding approvals if you require gated "sign-offs"
and so on.
We have a large solution with many projects, and many developers working together.
We need a way to validate Nuget package versions to insure no developer accidentally breaks the build with a package update.
Ideally, is there a way to validate and interupt/stop during Nuget package installation if it is a known incompatible package? We know we can do a validation at build time, but Ideally I'd like to actually be able to stop/inform the developer the newer version is not supported in the build to prevent them from going off and building with the newer package, only at build time to discover they've wasted time on a new package that might have differing calls etc.
If this is not possible, what would be the easiest way to do this at build time? I'm thinking a pre-build script but interested in other ideas. Effectively, the script would look across other project package versions to compare and inform if an incorrect version (and stop a publish to a shared location during build).
I've read a whole bunch of articles and SO questions on importing 3rd party go packages which all seems straight forward, but what I don't understand is that none that I have read make any references to versioning. In Dartlang there's the pubspec file that defines your package including its version and its dependencies including their required versions. What if I do a go get github.com/gorilla/sessions and write my app then 6 months later I have to clear my directories and re get everything again, in which time that package has been update and broken backwards compatibility with my code that was using the older version?
The official version, from the GO FAQ:
If you're using an externally supplied package and worry that it might change in unexpected ways, the simplest solution is to copy it to your local repository. (This is the approach Google takes internally.) Store the copy under a new import path that identifies it as a local copy.
There are many alternative to that approach, mainly based on declaring the exact version of those projects you are using.
See for instance "Dead Simple Dependencies in Go -- Keep it simple and keep your sanity." (based on emil2k/vend)
The main different options for Go Dependency Management are listed at:
"Go Package Management -- A summary of dependency management in Go"
(And its associate GOPM mailing list)
Update July 2015:
the official vendoring approach from Go team is discussed here.
an alternative go build tool called "gb" is proposed at getgb.io by Dave Cheney.
Update Q4 2017: as mentioned below, go dep is the official tool for pinning version of dependencies (even though that pinning approach is not without criticism: see "The cargo cult of versioning").
It should be merged into the toolchain when Go 1.10 development begins, according to its roadmap.
Update Q2 2018: go dep has been replaced by go mod (modules) in Go 1.11, following works on vgo.
I use dep as a dependency management tool for golang project. Please use the following link dep tool for more info.
dep is a dependency management tool for Go. It requires Go 1.9 or newer to compile.
dep was the "official experiment." The Go toolchain, as of 1.11, has (experimentally) adopted an approach that sharply diverges from dep. As a result, we are continuing development of dep, but gearing work primarily towards the development of an alternative prototype for versioning behavior in the toolchain.
Current status: January 2019
dep is safe for production use.
First, I will describe our setup (it is an extremely simplified version of the corporate development pipeline). We develop a .NET application, which is dependent on some shared components. The shared components typically develop together with the application. We want turn these shared component to NuGet packages.
So, I publish version 1.0.0.4 of package A and want to consume it in my application. My packages.config looks like this.
<package id="A" version="1.0.0.4"/>
So far so good. I probably commit packages.config to version control at this point to remember my dependencies. Now I run the next build of package A, which publishes the version 1.0.0.5. Now I want my application to automatically update package A to 1.0.0.5 before the build.
However, my packages config now contains an exact verion 1.0.0.4 and it's version-controlled. My questions are:
Can I somehow specify that I want version 1.0 or higher in packages config? In other words, can I avoid change of packages.config with every new build of every package?
Can I somehow update the dependencies before build to the the newest version? Can I do it automatically using a script?
I am quite new to NuGet and I come from the ant+ivy/maven world, where this feature is kinid of automatic, so I am still hoping I am missing something obvious in NuGet, although scanning through the discussions on stackoverflow doesn't sound too encouraging.
I've found Using Nuget in development environment - best practices / how to and How to automatically update NuGet packages to latest available version and NuGet issues with packages.config, project references and the solutionwide packages folder, which do not give clear answers.
You can use the allowedVersions attribute to automatically use newer package versions within a range, like this:
<packages>
<package id="SomePackage" version="2.1.0" allowedVersions="[2,3)" />
</packages>
In this example, any package from the 2.x series will be used, but not from the 3.x series. You can be more specific with the contraint e.g. `allowedVersions="[2.1,2.4)" if you wanted to take any package from 2.1, 2.2, and 2.3 but not 2.4.
See http://docs.nuget.org/docs/reference/versioning for more details
I don't think you can change the packages.config to implicitly allow a higher version. Also, the project files have references to the packages folder which include the package version in the path, so those need to be changed after a package update.
You can perform an update of the packages.config and the projects files by running nuget update.
You could automate this in the nightly builds and checking in the resulting changes as well. However, it's debatable whether that's a good idea. At the minimum, you should gate the update operations by running a build to make sure the new version doesn't cause a build break.
I understand that there is currently a feature request for something like this, but I'm hoping that there is some sort of workaround using the current version (1.6)
We build for our dev and test environments from a dev branch in TFS, and build for our QA and Production environments from a release branch in TFS. Since these produce distinct nuget packages, I can't use the same package feed. The deploy package step doesn't seem to have a environment scope option. Is there some other way to say "Deploy package X version Y for Dev/Test, and package A version B for QA/Production?"
You can use the same feed, with the following caveats:
The nugets built from the two branches obviously have to have different (non-colliding) versions. I add the 'dev' suffix to the package built from the 'dev' branch (eg 1.2.3.4-dev), and leave my 'stable' branch bare.
You have to be explicit when you 'create release', because by default Octopus will pick up the highest version nuget available, and that might not be the release you want (stable presumably lags dev). Pick the version of the package you want (and set the deployment version appropriately). If you are creating your release via TeamCity, ensure you use the --packageVersion argument as well as set the release number.
Since the above only works if there's only one package in the release, your deployment process does (unfortunately) have to produce one monolithic nuget, or you will get version-mismatches.
The advantage of this kind of arrangement, of course, is that at a pinch you could push your dev build to QA (or Prod) if the need ever arose.
All this assumes both branches build as the same package of course. You could build different packages between the dev and stable branches (but I don't think I'd recommend this because of duplicating all the Octopus config).
Update: apparently you can use Octo.exe to specify different version numbers across different packages - see https://github.com/OctopusDeploy/Octopus-Tools.