Normally "continuous package integration" involves source control, a build server, and participating teams fetching updated packages as often as they like. But I'm looking for a more extreme version of this story - without CM - that happens entirely on a developer's machine, all in one swoop. A more detailed description of what I want goes like this...
Using Visual Studio 2010 or 2012, assume a "foo.csproj" application that implements a plugin system. Each plugin represents a nuget package and has a corresponding Visual Studio project. Each of these projects is part of the same VS solution that contains the base application.
I want the following development story:
Change source code for a plugin.
Build solution, or perform a debug-launch, which causes msbuild to...
rebuild the changed plugin(s)
nuget then packages and uploads each plugin to a local repository (which can be just a subfolder of the VS solution)
rebuild the base application.
refresh the base application's nuget-plugin dependencies, which were just updated in prior steps. Notes:
This assumes MSBuild magically knows not to perform this last step until all plugins are built, packaged, and uploaded.
The "foo" application could itself use nuget.core to refresh the packages, but in this case I'm assuming that the VS build process did this step.
I would like to know if this story is common enough that there are "common" (msbuild?) scripts for this.
My own guess of how this should be handled is as follows:
All plugin projects are placed in a common "Plugins" folder somewhere in the VS solution folder structure.
The base application "project dependencies" are configured with references to all the plugin projects.
Note: I don't like the idea of managing these project dependencies manually.
The base application "foo.csproj" has a build step that scans the "foo.csproj" XML for dependencies it has in the "plugins" folder, and initiates the nuget packaging and deployment for each.
The base application then initiates the nuget "update all". Hopefully this is possible even though msbuild already mid-stride in execution.
In short, the base application is able to instantly consume plugins that have been altered. This is done without check-ins, a build-server, or manual and arbitrary requests to update plugin packages.
If pre-existing scripts do not already exist for this story, then I'll make my own. But I'd still like to know:
Can step 2, immediately above, be converted to something generic? That is, how can I convince msbuild not to build the "base application" until all projects in a particular folder have already been built? Remember, I'd like not to manage the project dependencies manually.
Is there anything flawed with this overall approach?
I would be particularly interested to know if there is an already existing nuget-visual-studio integration that assists with this story that I may have overlooked.
That's quite a long question to answer, so not sure I'm covering everything in this one; I'll do my best. First, your scenario is not uncommon. The first 2 steps of your planned approach seem OK to me (you're free to choose the location of the plug-in projects).
One thing's for sure: you'll have to manually define the build order, because your solution has no idea of knowing whether the projects consuming (NuGet) plug-ins have a dependency to the projects containing the source code for those plug-ins. Instead of using the built-in Build Order dialog in Visual Studio, take a look at this post on the MSDN blog for a correct way of doing this (or you might end up with something that works locally but not on the build server).
The key MSBuild elements in the referred post are the following:
<ProjectReference Include="... foo.csproj">
<ReferenceOutputAssembly>false</ReferenceOutputAssembly>
</ProjectReference>
Now, as for the packaging, deployment and consumption of those plug-ins:
Each plug-in project should trigger package creation and publication in a post-build step. This post on my blog contains ZIP-download with quite lot of MSBuild stuff you can use to get started. E.g. I version, package and publish the NuGet package for a class library in Release builds. I'm using the NuGet command line tool to pack (command reference) and push (command reference) the package.
The consuming application project(s) should run NuGet.exe update <packages.config> (command reference) in a pre-build step.
Also pay attention you're NOT running builds in parallel.
Related
I have been trying to find an "elegant" way to integrate non-NuGet package with my Bamboo builds.
There's a plethora of stacks on the topic of adding non-NuGet packages into NuGet bundles:
managing non-nuget dlls along with nuget packages
Creating NuGet package with reference to a non-NuGet reference
Trying to add non-.NET libraries to NuGet package
and the list goes on. There's also many a stack about using NuGet in Bamboo and that part works smoothly.
None of these deal with the situation of having an automated build environment, which may be sitting on some other remote server, running Bamboo.
Specifically, I'm trying to automate Xamarin.iOS deployments to HockeyApp.
The steps are:
Coding and local testing in VS2015 on Windows and with Mac for iPhoneSimulator
Merge into deployment branch and push to Bitbucket server
Bamboo picks up the push and kicks off build
Build checks out deployment branch
Runs NuGet downloads
Starts compile for Ad-Hoc/iPhone environment, creating IPA
Kicks off the HockeyApp deployment (there's a free addon for that)
Nearly all the steps are in place, except for the one where I have 2 dependencies which the commercial vendor (Syncfusion) has for unknown reasons decided to bundle into their "Studio" product, from where my Visual Studio project/solution has to refer to them by location outside of my project directory.
As a result, my Bamboo build fails with not-found DLLs, because they're missing as they would have to come in somewhere between 5. and 6. above.
I don't want to copy the binaries and then check/commit/push them into my repository, as that's considered a Bad Thing. My Bamboo Plan already successfully grabs NuGet packages before the actual build without having to drag binaries along.
Simply copying the DLLs on to the Bamboo build machine (i.e. where the remote agent is running) was one idea, but the problem is that the VS/MSBuild project file now has hard-wired directories - so, I'd have to install the whole Syncfusion Studio, or emulate their directory structure just for those 2 DLLs.
So I would need to adjust the .csproj references in an automated fashion. Not sure how I would do that, except with Yet Another Task and Script.
Apart from yelling at Syncfusion (which I've already done) about making all their DLLs available through NuGet (because some are, and those I'm successfully receiving in step 5. above), does anybody have a suggestion how to get this missing step to work?
For expediency sake, I have now added the libraries to the repository in a separate sub-directory.
It's not how I wanted to do it, but as the 2 libraries were a mere 200-300KB each and as there just didn't seem to be a simpler solution it solved the issue for now.
Specifically:
Leave .dll in .gitignore
Copy libraries you need into local sub-directory, e.g. LocalLibs/
add specific libraries with git add -f LocalLibs/speciallib.dll so that only these become a part of the repo
Change the project reference in Visual Studio to point to the local libraries, instead of their main install location
Verify that builds still work from within Visual Studio but also with MSBuild
I may revisit this and update if a better way comes along, especially if the libraries are significantly larger, such that you definitely would not want to add them to your repository.
We are currently migrating our source control to Visual Studio Online. A feature we had in our old system (SourceGear Vault) was to share projects between solutions. Although this created a folder for our Framework project in each solution it kept it up to date when changes were checked in.
This is useful to us as it allows us to work on the Framework code in all the Solutions that are using it. I know its better practice to just compile the dlls and reference them - at this point in development we want to continue having full code access and debugging in all the solutions using this core framework.
Any help very much appreciated.
You have a few equally valid options for handling shared projects:
Reference the same project from each solution that needs it.
This gives you full control over the source code of the shared project while you work on the consuming solution, and may allow for easier debugging.
The downside here is that maintenance and releases may become trickier if Solution A is being released on Thursday, but Solution B is being released in 3 months and is in the middle of a huge refactoring cycle that has significantly modified Shared Component X, and Shared Component X isn't stable enough to be released.
Reference shared components from an internal NuGet repo.
You set up your release pipeline to push the shared components into NuGet as part of your release process (ideally, using a purpose-built release management tool... Microsoft Release Management is what I have in mind here) -- you check the code in, project gets built. Release process packages it up and pushes it into NuGet as a "prerelease" version. You reference the latest version in anything that needs the latest version.
If you need to reference a known-good, stable version, you just make sure your project is configured to pull a specific version from NuGet.
When you're done, you've tested the shared thing, and you know everything is good, you approve the prerelease version, and the same binaries are repackaged into a "stable" version.
The downside here is that there are some additional software requirements, configuration, and training for your team. This would be my recommended approach.
Check binaries into source control.
I don't recommend this one -- you end up bloating your source control repo (and if you're using Git, it's an explicitly stated anti-pattern -- never put binaries into Git, it causes long-term severe performance problems), and it's never exactly clear which projects are using which versions of which assemblies. It's a maintenance nightmare.
(1) is the best approach if you're releasing everything in lockstep and don't have to worry about maintaining separate versions.
(2) is the best approach if #1 is false.
(3) is the best approach if #1 is false and you're a time traveler who is posting from 2006.
Have you considered implementing Symbol Server and Source Server indexing with the check-in binaries or NuGet repo approach? This allows you to easily get back to the source while debugging and it's coming from a single known location. Visual Studio Online and Team Foundation Server have built-in support for helping you out with getting this setup during your build process. Here's more information in a write-up here: Source Server and Symbol Server Support in Team Foundation Server and Visual Studio Online
Thanks for the responses. We actually found a solution that works well for us. We branch our framework project into the implementation projects when we want access to the code base. If not we just use the DLLs
If it is then altered and checked into the implementation project it can be merged back with the other Framework branches easily when ready.
This probably wouldn't work well if the Framework code was being developed heavily, as it is now its only undergoing small additions and tweaks so wont be plagued with merge issues.
I have to agree with the majority. I just ran into the same issue. I implemented the Nuget Gallery Site on the internal network. It was a pain to implement, but once implemented, it's easy to use. I created a class library project that implements ADO.Net and the EntityFramework. I bundled it into a NuGet package and uploaded it to the internal NuGet gallery. From there I was able to add a package source to the internal NuGet gallery and grab the package that I uploaded. Very simple and convenient.
I set up the NuGet Gallery with Visual Studios 2017. FYI: Make sure that building the project isn't part of the Publish. It will not render with a ViewHelper.cshtml error.
I created the projects with Visual Studios 2015. I ran the command prompt as administrator. I also had to copy the NuGet.exe file into the directory where the project file existed.
Check out the below links for more information.
NuGet Gallery
Hosting NuGet Gallery
Create and Publish Package
Creating a Package
Create .Net Standard Package
When opening a project you may have recently downloaded, Visual Studio 2013 briefly warns you to only open projects from trustworthy sources. What are some of the risks to opening projects? Can a project do any harm to your system before compilation, and what damage is possible? Are there any indicators you should be aware of before actually opening projects from "untrustworthy" sources? (Not that I can think of any)
Attack Surface within Visual Studio
There are many attack vectors within Visual Studio. All of them are by design. We developers want complete control over our systems within the build process. Unfortunately, when we say "I want to delete the contents of the cache directory on build", that also means malicious project files can delete just about anything from just about anywhere. Or worse. The compromise is the "Hey. We've given you the keys to the kingdom, but we don't recognize this project. Are you sure you want to open this? We're not responsible if it does something stupid" warning message that you mentioned.
Now consider that many developers run Visual Studio as an administrator.
Here are some of those attack vectors:
Pre- and Post-Build Events
In it's simplest form, untrusted projects could execute a Pre-Build event that would delete files. Or worse. Just about anything can be executed within a Build event. This is the 101 stuff that happens on Compile.
Executions on Project Open
Visual Studio project files are nothing more than big MSBuild configurations. There are a few MSBuild targets that Visual Studio executes when you open a project, all to support the tooling. These targets include Compile, ResolveAssemblyReferences, ResolveCOMReferences, GetFrameworkPaths, and CopyRunEnvironmentFiles. If any of these targets exist, the tasks within them are also executed. Delete files, or worse.
See: 'Design-Time Execution' at http://msdn.microsoft.com/en-us/library/ms171468.aspx
IntelliSense
Part of that tooling mentioned above includes IntelliSense, which executes the Compile task within MSBuild; the CSC/VBC must be executed to get all of the IntelliSense functionality. Because of the nature of IntelliSense, this task is repeatedly executed as you work, rather than the opportunities above that are just run once on open.
See: 'Design-Time IntelliSense' at http://msdn.microsoft.com/en-us/library/ms171468.aspx
Hidden Elsewhere in MSBuild
There is a sea of other routine MSBuild targets that you will manually execute throughout your day, including Build, Rebuild, Test, and Clean. Yes, keep in mind that even clean is a build target, so Clean could delete more than just your old \bin directories.
NuGet
Malicious projects may also expose systems via NuGet. Though Package Restore would not be an issue, the packages.config could specify a different Repository Source. Then, when you install a new package, such as install-package jquery, NuGet will retrieve the jQuery package from the untrusted alternate, rather than from nuget.org. This malicious jQuery package could have all sorts of other 'Goodies' within it that would be executed as a part of the package installation.
This is not a security vulnerability from NuGet, because "you" specified an alternate package source; this is by design, such as Corporations that have their own internal package repository.
What you can do?
At the end of the day, what can you do about this? The answer really is to just not open projects from untrusted sources. The project's packages.config file could be analyzed before you open it, but the big exposure is through MSBuild. Unless you are quite adept at reading through MSBuild schema, I would steer clear.
Can MSBuild capture a list of the projects it has built "so far" during a build?
I would love a hypothetical #(ProjectsBuilt) item list I could access, which names the csproj files that have finished being been built.
You might ask "Why this feature?" In the abstract, it pertains to updating or exporting dependencies - particularly in an incremental build setting. I'll give two examples:
A Visual Studio solution contains a number of "plugin" projects. When some of the plugins have been updated, and built, a general-purpose script "captures" which projects were built, and immediately kicks-off nuget packaging and repository uploads.
A Visual Studio solution contains "implementation" projects and "interface" projects. In other words, some of the resulting assemblies are meant to be exported for use in other solutions. After some code changes have been made, the "interface" assemblies resulting from building any and all projects are then exported to a special "common" folder to be consumed by other applications.
Certainly the problems described above could be solved by having each "exported" or "plugin" project contain a post-build script to apply the appropriate dependency-exportation step. But it is anti-DRY to require a post-build step be added into every participating VS project.
If MS build would simply keep a running-tally of what projects have been built "so far," then at an appropriate build-step all the necessary "dependency exportations" could be done in one swoop, and from a centralized msbuild location. The projects targeted by this capability would not require any per-project build-steps at all. Nice!
I'm making a build using FinalBuilder Pro 7.
I've an ASP.NET web site and I'm trying to use FinalBuilder's "Precompile Asp.net 2.0 Application" action. Well, It fails.
To build it successfully I need to run Visual Studio, open the web site and either build it manually from within VS or manually update all references. After that it works.
Now the question: How to force FinalBuilder to update those references? Even if I create web deployment project associated with the web site and try to build it with msbuild action it would fail for the same reason. Somehow neither action updates references automatically.
Update: OK. Maybe I need to force msbuild to update references. How to do that?
I found some properties that I can change at msbuild action.
On called ResolveAssemblyReferencesDependsOn I tried to put the value = true. Didn't help.
Any ideas?
There are different types of projects in .NET like library project, website project, web API project etc. As you mentioned in your question it is a website project, so I am going to give you a solution for website project.
You can build a project by two ways. Either you can build by visual studio or you can build by using MS Build. If you build your project by using VS, you can update references of your dependencies by executing the command "Update-Package -reinstall" in package manager console. It will reinstall all the packages automatically.
Please note that, all your dependencies are listed in packages.config file.
Secondly if you have to build your project by MS Build using cmd prompt, to load all the dependencies from nuget, you have to execute nuget.exe. By which all your dependencies will be loaded, but their references may not be updated. So in website project you do not have .proj file. So you can't have access to the references of your dependencies. Now problem is that how you can modify your dependencies references?
In website project reference of an assembly exists in its .refresh file. So you have to modify that .refresh file to update the reference of an assembly in website project.
Have you tried using the 'Build VS.NET Solution' or 'MSBuild Project' actions? Both should resolve your assembly references, provided the reference is set to the right location. This requires that you at least have a project file.
As I understand it, the Precompile action (which uses the MS aspnet_compile.exe - see http://msdn.microsoft.com/en-us/library/ms229863(VS.80).aspx) is designed to re-compile an asp.net application which has previously been built via VS or MSBuild. It does either an in-place compile to improve performance for the first user that hits the site, or creates a deployable application (removing source code etc). It's not meant as an alternative to VS/MSBuild.
I'm not 100% sure I understand the problem, but I believe you need to be able to correct some pathing on an assembly reference automatically.
I created a project to handle this (along with some other things): refswap.codeplex.com