I have been trying to find an "elegant" way to integrate non-NuGet package with my Bamboo builds.
There's a plethora of stacks on the topic of adding non-NuGet packages into NuGet bundles:
managing non-nuget dlls along with nuget packages
Creating NuGet package with reference to a non-NuGet reference
Trying to add non-.NET libraries to NuGet package
and the list goes on. There's also many a stack about using NuGet in Bamboo and that part works smoothly.
None of these deal with the situation of having an automated build environment, which may be sitting on some other remote server, running Bamboo.
Specifically, I'm trying to automate Xamarin.iOS deployments to HockeyApp.
The steps are:
Coding and local testing in VS2015 on Windows and with Mac for iPhoneSimulator
Merge into deployment branch and push to Bitbucket server
Bamboo picks up the push and kicks off build
Build checks out deployment branch
Runs NuGet downloads
Starts compile for Ad-Hoc/iPhone environment, creating IPA
Kicks off the HockeyApp deployment (there's a free addon for that)
Nearly all the steps are in place, except for the one where I have 2 dependencies which the commercial vendor (Syncfusion) has for unknown reasons decided to bundle into their "Studio" product, from where my Visual Studio project/solution has to refer to them by location outside of my project directory.
As a result, my Bamboo build fails with not-found DLLs, because they're missing as they would have to come in somewhere between 5. and 6. above.
I don't want to copy the binaries and then check/commit/push them into my repository, as that's considered a Bad Thing. My Bamboo Plan already successfully grabs NuGet packages before the actual build without having to drag binaries along.
Simply copying the DLLs on to the Bamboo build machine (i.e. where the remote agent is running) was one idea, but the problem is that the VS/MSBuild project file now has hard-wired directories - so, I'd have to install the whole Syncfusion Studio, or emulate their directory structure just for those 2 DLLs.
So I would need to adjust the .csproj references in an automated fashion. Not sure how I would do that, except with Yet Another Task and Script.
Apart from yelling at Syncfusion (which I've already done) about making all their DLLs available through NuGet (because some are, and those I'm successfully receiving in step 5. above), does anybody have a suggestion how to get this missing step to work?
For expediency sake, I have now added the libraries to the repository in a separate sub-directory.
It's not how I wanted to do it, but as the 2 libraries were a mere 200-300KB each and as there just didn't seem to be a simpler solution it solved the issue for now.
Specifically:
Leave .dll in .gitignore
Copy libraries you need into local sub-directory, e.g. LocalLibs/
add specific libraries with git add -f LocalLibs/speciallib.dll so that only these become a part of the repo
Change the project reference in Visual Studio to point to the local libraries, instead of their main install location
Verify that builds still work from within Visual Studio but also with MSBuild
I may revisit this and update if a better way comes along, especially if the libraries are significantly larger, such that you definitely would not want to add them to your repository.
Related
We're just in the process of transitioning from VS2013&15/TFS2013 to VS2017/TFS2017 (on-site TFS, not VSTS) and the first test solution is a dotNet Core 1.1 based one (a multi-project web service).
The solution builds fine on the original developer's box and I've got it out of TFS and it builds fine on mine too. In keeping with our previous methodology the contents of the packages folder are checked in with the projects as this makes the packages locally available on the build box (no internet).
Building the solution on the build server is a different story, however, as I get multiple errors of the form...
..\obj\project.assets.json' not found. Run a NuGet package restore to generate this file.
I get the errors both when I run the TFS build definition and when I remote to the box and build directly through the VS on the box itself.
This whole project.assets.json not found issue seems to be causing headaches all over. In my case the issue is that I'm trying to resolve it on our TFS 2017 Build Server, which does not and never will have internet access ('cos it's a server!).
All the solutions I've seen thus far seem to suggest running the Nuget Restore command but that can't work since the server cannot get to nuget.
This is nothing fancy yet, just a simple TFS 2017 Build definition with a Get sources and a Build solution step. I can't understand how something so simple has become so difficult.
Changing the Nuget Package Restore options makes no difference.
Since the project.assets.json files are generated on the fly in the obj folder, I can't even check them in to reuse. Can anyone please suggest a workaround, at the moment the test project is dead in the water.
Edit: trying the same process with a 4.6.1 web project created with VS2015 had similar results of unresolved references (e.g. System.Web) but didn't raise the same error, probably due to being an older, non-Core project.
According to I get the errors both when I run the TFS build definition and when I remote to the box and build directly through the VS on the box itself.
The issue seems not related to TFS build side since it also not work with local build through VS in the build agent machine.
Since this is a dotnet project. So, you could try to use “dotnet restore” and not “nuget restore”. Try using the dotnet core template (which uses dotnet restore).
If you are using authenticated nuget feeds, then you can use nuget restore but you also need to use nuget installer task. See https://github.com/Microsoft/vsts-tasks/issues/3762 for a discussion on that.
The Nuget version should be higher than 4.0.
Without dotnet restore and Nuget restore and only use get source/Visual Studio Build will not be able to build the dotnet core project. If your server do not have internet access, as a workaround you should use Local feeds.
I have just setup TeamCity to automate our builds, our current solution has both a dev and main branch. What I am trying to achieve is to have the development branch build and publish to a development NuGet feed on our ProGet installation, and then have the main branch publish to our Main NuGet feed on ProGet server.
We are using octopus deploy to deploy the packages, within TeamCity we have the octopus deploy plugin installed and if I tick the box to run OctoPack it builds the packages and they appear as artifacts when the build completes. If I try to use the NuGet Pack build step in TeamCity I get the following error for one of our projects:
[08:33:49] : [pack] Attempting to build package from 'xxx.csproj'.
[08:33:50]W: [pack] Unable to find 'xxx.exe'. Make sure the project has been built.
The project has been built and it works with OctoPack so why isn't it working with the NuGet Pack? Wwe have five projects being built and the first four run fine, one is a console app, one is an mvc website, and two are class libraries. The one that doesn't work is a windows service.
The end goal here is to publish these packages to a private feed on ProGet. I don't mind using OctoPack but in my head wanted to remove that dependency from TeamCity but I can live with it. However when I try to use the NuGet Publish runner type how do I select to publish any NuGet artifacts that have been created?
I have been googling like mad and I cannot find any helpful links that describe what you are supposed to enter, I would really appreciate any helpful comments/answers.
We are using version 8.15 of TeamCity.
Hopefully the following will help with at least part of your question; mainly the bit relating to how to publish packaged artifacts.
NuSpec Approach
When using the NuGet Pack build step, you can specify the Output Directory, which will determine the output location of the packages. You can specify this as a relative path to the checkout directory, probably best to define it as a build parameter, such as %system.PackageDeployOutput% as you'll be using it in the next step...:
Next, specify a NuGet Publish build step, fill in the Package Source / API key etc, and specify the Packages to upload as
%system.PackageDeployOutput%\*.nupkg
This will pick up the packages output in the previous step. I've used this quite effectively, and the parameterisation approach encourages conventions across all your builds.
OctoPack Support
If you're using the MsBuild build step with OctoPack, you can use a similar approach by declaring a system parameter called
system.OctoPackPublishPackageToFileShare = %teamcity.build.checkoutDir%\%system.PackageDeployOutput% (note the same parameter as above)
You can declare these as root project parameters, so you get the best of both worlds. My preferred approach to packaging is currently to use nuspec files for deployable endpoints. I've found OctoPack to be a bit of an overhead when it comes to more complex deployments (it's fine for basic MsBuild projects).
Normally "continuous package integration" involves source control, a build server, and participating teams fetching updated packages as often as they like. But I'm looking for a more extreme version of this story - without CM - that happens entirely on a developer's machine, all in one swoop. A more detailed description of what I want goes like this...
Using Visual Studio 2010 or 2012, assume a "foo.csproj" application that implements a plugin system. Each plugin represents a nuget package and has a corresponding Visual Studio project. Each of these projects is part of the same VS solution that contains the base application.
I want the following development story:
Change source code for a plugin.
Build solution, or perform a debug-launch, which causes msbuild to...
rebuild the changed plugin(s)
nuget then packages and uploads each plugin to a local repository (which can be just a subfolder of the VS solution)
rebuild the base application.
refresh the base application's nuget-plugin dependencies, which were just updated in prior steps. Notes:
This assumes MSBuild magically knows not to perform this last step until all plugins are built, packaged, and uploaded.
The "foo" application could itself use nuget.core to refresh the packages, but in this case I'm assuming that the VS build process did this step.
I would like to know if this story is common enough that there are "common" (msbuild?) scripts for this.
My own guess of how this should be handled is as follows:
All plugin projects are placed in a common "Plugins" folder somewhere in the VS solution folder structure.
The base application "project dependencies" are configured with references to all the plugin projects.
Note: I don't like the idea of managing these project dependencies manually.
The base application "foo.csproj" has a build step that scans the "foo.csproj" XML for dependencies it has in the "plugins" folder, and initiates the nuget packaging and deployment for each.
The base application then initiates the nuget "update all". Hopefully this is possible even though msbuild already mid-stride in execution.
In short, the base application is able to instantly consume plugins that have been altered. This is done without check-ins, a build-server, or manual and arbitrary requests to update plugin packages.
If pre-existing scripts do not already exist for this story, then I'll make my own. But I'd still like to know:
Can step 2, immediately above, be converted to something generic? That is, how can I convince msbuild not to build the "base application" until all projects in a particular folder have already been built? Remember, I'd like not to manage the project dependencies manually.
Is there anything flawed with this overall approach?
I would be particularly interested to know if there is an already existing nuget-visual-studio integration that assists with this story that I may have overlooked.
That's quite a long question to answer, so not sure I'm covering everything in this one; I'll do my best. First, your scenario is not uncommon. The first 2 steps of your planned approach seem OK to me (you're free to choose the location of the plug-in projects).
One thing's for sure: you'll have to manually define the build order, because your solution has no idea of knowing whether the projects consuming (NuGet) plug-ins have a dependency to the projects containing the source code for those plug-ins. Instead of using the built-in Build Order dialog in Visual Studio, take a look at this post on the MSDN blog for a correct way of doing this (or you might end up with something that works locally but not on the build server).
The key MSBuild elements in the referred post are the following:
<ProjectReference Include="... foo.csproj">
<ReferenceOutputAssembly>false</ReferenceOutputAssembly>
</ProjectReference>
Now, as for the packaging, deployment and consumption of those plug-ins:
Each plug-in project should trigger package creation and publication in a post-build step. This post on my blog contains ZIP-download with quite lot of MSBuild stuff you can use to get started. E.g. I version, package and publish the NuGet package for a class library in Release builds. I'm using the NuGet command line tool to pack (command reference) and push (command reference) the package.
The consuming application project(s) should run NuGet.exe update <packages.config> (command reference) in a pre-build step.
Also pay attention you're NOT running builds in parallel.
I'm creating an OpenSource project on BitBucket (using Mercurial) (in C# with Visual Studio 2010, if it's important). The project has some external dependancies (xUnit, StyleCop, Mono.Nat, BerkeleyDB, ...). I want to use AppHarbor to make test builds and run tests.
I don't want to use NuGet to download them (because some of them aren't in NuGet and so I would have to create a private NuGet repository, because the StyleCop/*MsBuild*/NuGet integration doesn't work very well. It requires two "rounds" of build to load correctly the StyleCop.targets file, but AppHarbor will only do one and finally because I think it's very bad to create an open source project that has an exe file inside that is automatically launched when the project is compiled. Would you launch a random exe contained in a solution you downloaded from Internet?).
I don't want to include the external dependancies in the main tree (I'm very much against it).
I want to have the external dependancies somewhere else (another BitBucket repository, for example).
I don't have a server that can do a cron job and pull/push from one repository to another, but if you can find a free service that can do it for me it's ok as a solution.
How can I do it? I have tried adding multiple repositories to AppHarbor, but they are compiled one at a time (from an empty folder)
We’re on a closed network without Internet access so we’re currently using a file share as a NuGet repository. We do a lot of merging between a development branch and a main branch and occasionally setup a one-off branch for a hotfix or large functionality that will extend beyond our normal release cycles.
What we’ve found is that when we add a NuGet package to a project it puts a file path in the .csproj file for where the package is located. This works fine until we merge into another branch in TFS and then kick off a build. The builds do not pull down the same files from source control (keeping dev and main completely separate in that regards) so the package path is not found and the build fails.
We’ve come up with one solution we know will work but is utterly painful and one proposed solution that we need to investigate further. One solution is to put the packages folder into a common location that every build will include in its workspace and manually modify each .csproj file to point to that location for its packages.
The solution we need to investigate is using NuGet without committing packages to source control. What we need to determine is if NuGet.exe will create the folder structure needed by the .csproj file to find the package.
Are there better solutions to using NuGet in a branching environment that uses automated builds?
What you could do is create a local NuGet repository that is reachable by http:// that can be easily used in both branch situations. Checkout how to create a local NuGet repository here