Set up
I have a solution Commons.sln which contain common utils for other more specific projects. Let's take for future example one specific solution called Cars.sln just to make it easier to explore the case. All solutions have their own set of NuGet packages.
All projects are laying in the same VCS branch therefore folder structure is pretty strict.
What I want to achieve?
I would like to have references for common libraries in all specific projects. Accordingly, I have added projects from Commons.sln to my Cars.sln. However I met an issue with this setup because people from my team who never built Commons.sln do not have NuGet packages from this solution downloaded to their machines and when the try to build Cars.sln - they face an error saying that dlls for projects from Commons.sln are not found.
Long story short - I would like to execute nuget restore ../../Commons/Commons.sln before build, but do not know how to do it better on Cars.sln solution level on every build/rebuild in Visual Studio.
Specifics
I have done a little research on this and there are my bullet points:
I do not want to commit any of my NuGet packages in VCS
I so a solution with adding a .target file per solution but it works only when I build directly with msbuild.exe.
In other SO question. There is a recommendation to add one more project and resolve it by build order which seems to be hacky to me.
Specific projects are bound to CI server so this stucture should work there as well (currently I have solved by adding a build step that builds Commons.sln before everything else)
Any suggestions are appreciated.
Currently I have fixed it with adding a msbuild target to the project in Commons.sln which is a dependency to all projects including specific ones, like Cars.sln:
<Target Name="Test" BeforeTargets="BeforeBuild">
<Message Text="Restoring Common Nugets" Importance="high"></Message>
<Exec Command="nuget restore "$(ProjectDir)..\Avid.Common.sln"" />
</Target>
However if somebody has a better solution I would be glad to know about it.
Related
I have a large VStudio solution whose .csproj files I'm porting over to the new .NET SDK format. I have integrated the github project with AppVeyor. When I work on the solution, my project is built by VStudio; AppVeyor uses msbuild to build my project.
All of the class library project files in the solution have been modified so that they specify <TargetFrameworks>netStandard2.0</TargetFrameworks>. For simplicity, let's represent the class library project files with classlib.csproj.
The solution also contains a few command-line tools, which need to be built in advance of some of the other projects, because those later projects need to run the command-line tools as a preprocessor before building themselves. These command-line projects are all specifying <TargetFrameworks>net472;netcoreapp2.0</TargetFrameworks>. Let's represent the tool project files with tool.csproj.
In order to build correctly, tool.csproj needs to be built before classlib.csproj. In the solution I've used the Build dependencies > Project dependencies command in the VStudio solution explorer to indicate this. This works fine when I'm working inside of VStudio.
However, when I push my changes to github, AppVeyor kicks off an msbuild process to build the projects. From the error spew, it's clear that the tool.csproj didn't get built in time. It seems msbuild doesn't understand the project dependency information in the .sln file.
Reading what's posted on the intertubes (e.g. https://devblogs.microsoft.com/visualstudio/incorrect-solution-build-ordering-when-using-msbuild-exe/), I attempted to add the following to classlib.csproj:
<ProjectReference Include="..\tool\tool.csproj">
<ReferenceOutputAssembly>false</ReferenceOutputAssembly>
</ProjectReference>
But when I do this I get the error:
Project '..\tool\tool.csproj' targets 'net472;netcoreapp2.0'. It cannot be referenced by a project that targets '.NETStandard,Version=v2.0'
which clearly makes sense.
Some other solutions I've read suggest editing the .sln file so that tool.csproj gets built first, and hope that building will be completed before classlib.csproj starts building. However, this is a race condition since msbuild is not aware of the dependency.
How, then, do I express that classlib.csproj depends on tool.csproj so that msbuild understands it?
EDIT
It turns out the issue was caused by an interaction between the target I'd written to start my tool and the globbing operation MSBuild does to resolve the <Compile> item collection using a pattern similar to **\*.cs. The tool generated the file after the globbing operation. Since the generated file isn't present when the globbing happens, it is missing from the <Compile> item collection, and the build failed. I mistakenly interpreted the error "foo.cs not found" as a failure to execute the tool, when in fact, it's a timing issue.
To make the build order works outside VS IDE, we need to use add reference
instead of BuildDependencies=>ProjectDependencies.
Right-click classlib project and choose add=>reference, in Project node choose the Tool Project you want like this:
Click ok to add it and save changes. After that, open the classlib.csproj you can find sentence like:
<ItemGroup>
<ProjectReference Include="..\xxx\Tools.csproj">
<Project>{6eaa430f-9793-4639-a84b-6ab767d57147}</Project>
<Name>Tools.csproj</Name>
</ProjectReference>
</ItemGroup>
This what the msbuild can understand. And I think that's what you want.
After that,you can use a single msbuild tool or Appveyor to check the
build order.
And in Appveyor, we can also disable the paralell builds(settings=>build=>msbuild options) to make sure it won't build classlib.csproj until the tools.csproj build ends.
The final build order in Appveyor will be what we defined in VS. Hope it helps.
We are using TeamCity 2017.1 and have been using it for years with great joy. A long time ago, someone decided that all third-party binaires should be put into Subversion (our VCS of choice).
This has worked fine, but over time this repos has grown quite large, and combined with our being better and better at using TeamCity, we now have dozens of build configurations which all uses third-party binaries.
Our third-party folder is called Department and is around 2.6 GB in size. As such this is not so bad, but remember that this folder is used by pretty much every single project on the build server!
Now, I will agree with everyone that says that we should use Nugets, network shares etc., and that would work great with new projects. However, we have a lot of history and we cannot begin to change every single solution and branch.
A co-worker came up with the idea, that IF we made a single build project that in reality did nothing but keep a single folder updated with our Department stuff. Then we just need to find a way to reference this, without have to change all our projects and solutions.
My initial though is using Snapshot dependencies and then create a symbolic link as the first build step and remove it as the last, in order to achieve the same relative levels.
But is there a better way? What do other people do?
And keep in mind, that replacing with nugets or something else is not an option.
Let me follow the idea of your colleague and improve it. There would be a build configuration that monitors the Subversion repository and copies packages to a network share. That network share will be used by development teams as nuget repository. Projects that will convert their dependencies from Binary reference to nuget reference will enjoy faster build. When all the teams will use nuget repositories you may kill that Subversion.
We have two separate .NET solutions:
Running a build for the first solution produces our end product: a bunch of DLLs. These DLLs are delivered to our customers via a NuGet package.
The second solution serves as a product-test solution: the NuGet package is installed to it, and it is built and executed - thus it makes use of our product exactly the same way as our customers would do.
The challenge here is that there should be a way our latest NuGet package gets installed automatically to the product-test solution, preferably during the build of this product-test solution.
Based on the ideas from a similar question, I got this far with configuring the product-test solution:
First I enabled NuGet Package Restore. This lets me get rid of the "packages" directory completely from VCS since the package with the version defined in packages.config file would be downloaded automatically by NuGet before build.
Then I added the following pre-build event in Visual Studio: $(SolutionDir).nuget\nuget update -prerelease $(ProjectDir)packages.config. This lets me pull in the latest version of our NuGet package during build.
I currently use the above scenario to run local builds using Visual Studio and unattended builds using TeamCity. The solution seems to work for both scenarios on the first sight, but actually it does not produce the expected result: when the product-test solution is built, in the bin directory I don't get the latest version of the DLLs, only the latest-1 version.
The problem is that although the nuget update command updates everything as expected, including the packages.config and the .csproj file, their new content is not picked up by the build, therefore - as my guess goes - the HintPath settings from the .csproj file still reflect a "before build" state, therefore old DLLs are copied to the bin directory. I assume the .csproj file is processed only once: before the pre-build event is triggered, and the changes made by the pre-build event are ignored until the next build.
I considered the following solutions:
Apparently pre-build is not "pre" enough. If there was an even earlier point I could insert the nuget update command, my above solution would probably work.
I read that I could override the HintPath-s in the .csproj file by defining a ReferencePath. But I doubt I could easily figure out the right path or I could set it early enough so the build picks it up.
As a workaround I could run the builds twice: duplicate the build step for the product-test solution in TeamCity and I could always build the solution twice locally in Visual Studio.
Has someone figured out how to automatically update a NuGet package to the latest version during build?
Have a look at a blog post I just did regarding this process. Instead of configuring stuff on the server, I did it by augmenting the Nuget.Targets file that's put in place with the Nuget Package Restore option. Beauty of this approach is it executes locally AND on the server (so you get to see any possible side-effects before breaking the build)
Posted details here: http://netitude.bc3tech.net/2014/11/28/auto-update-your-nuget-packages-at-build-time/
I think that put automatic update to pre-build step it's not NuGet style. You can understand that you need this operation every time when you do it. Mostly because it can increase build time. For example when you use TDD and often rebuild project and run tests, you expect that it will be done quickly. Additionally it can update unexpected packages and break something, after that you can spend a lot of time to find the problem.
I suggest to do update as the separate step. On TeamCity you can use NuGet installer build step. For performing update just check two checkboxes in the bottom area of step configuration:
Additionally if you want keep result of update after successful build and passing test, you can add later build step which commits this changes to VCS (using cmd or PowerShell for example).
When you are working locally, I thing the better way run update packages once before you start working with project. You can use Package Manager Console for this, with command Update-Package -IncludePrerelease.
The MSBuild-solution I figured out is to override the BuildDependsOn property. Therefore, I created an UpdateNugetPackages.target which looks like this:
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="14.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<BuildDependsOn>
UpdateNugetPackages;
$(BuildDependsOn);
</BuildDependsOn>
<UpdateCommand>"$(SolutionDir)pathToYourNugetExe.exe" update "$(SolutionDir)NameOfYourSolution.sln"</UpdateCommand>
</PropertyGroup>
<Target Name="UpdateNugetPackages">
<Exec Command="$(UpdateCommand)"></Exec>
</Target>
</Project>
The UpdateCommand defines where, and with which arguments the nuget.exe gets called. Feel free to adopt this for your own needs.
This target has then be referenced in your .csproj File. As simple as that:
<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
...
<Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
<!-- After the initial decleration of the BuildDependsOn-Property. -->
<Import Project="UpdateNugetPackages.target" Condition="Exists('UpdateNugetPackages.target')" />
Keep in mind, that the import order matters. You have to import your target-file (UpdateNugetPackages.targets), with overrides (or actually decorates) the BuildDependsOn property, after the target-file Microsoft.Common.targets which defines it. Otherwise, the property will be redefined and will remove your changes, as the initial definition in Microsoft.Common.targets does not include any existing value in BuildDependsOn. Microsoft.Common.targets is imported by Microsoft.CSharp.targets for a C# project. Thus, your import has to go after the import of Microsoft.CSharp.targets.
We develop with Visual Studio 2010 (in C#) and migrated a while ago from SVN to GIT. Now we try to split up our repository (which is quite big - ~30.000 files) to many git repositories - one for each solution.
The solutions share some projects, mostly libraries we develop in-house and like to add to from all the solutions.
The new repositories have a flat layout. One subdirectory for each project (shared projects are submodules).
In the big old repo, the projects are in a tree structure.
The Problem occurs with external references in the submodules. In the new repos, the path to a referenced project may be "......libs\someproject", while in the new layout the correct path would be "..\someproject".
We already had some edit wars concerning this and are not keen on more.
Half-baked Solutions I could think of:
use "Reference Paths" in ...csproj.user and exclude this file from version control (has to be redone for each developer and after each reopsitory cleanup)
use branches for each situation and try to teach everyone where "real" commits should go and where "environment-change" commits should go (submodules are already not the simplest concept...)
embed binaries instead of the submodules (but what about developing changes to the submodules? what about different log4net versions?)
Does anyone know of a sane solution?
Since you are asking for a sane solution, I can only advise you to look into setting up your own NuGet service (look at http://www.MyGet.org for inspiration)
http://nuget.codeplex.com/
IF you go down the route of package management, consider OpenWrap. However, embedding the package management artefacts in source code is a bad idea. You can use such tools to update what is actually stored in submodules, but don't rely on them at build time. Expect the binaries to be there from the point of view of your build scripts.
So if I understand you correctly, the problem is with Visual Studio and not with Git? If that's the case, use the old tree structure that worked with Visual Studio. Make your submodules structure a tree structure too. So the top of the tree would be one super repo whose sub modules (the branches) would have submodules of their own, until you get down to the leaves of your tree. It would be a pain to setup at first, but it should just work.
Use one submodule to house all "common libraries". Just one level. But you should move the common libraries as services with well defined contracts. This way you can incrementally rollout new versions with no down time. This way you only have a submodule in each that holds the contracts. These could be interfaces or messages.
I have a similar problem using VS 2013.
I want to use git-svn instead of SVN directly. SVN has a gigantic set of directories. I could not create a single git-repository that would contain all of our trunk folder. Git-always exited with an error and the repository was corrupted. I worked around the problem by doing as follows:
Using git-svn, I cloned the subset of folders off SVN/trunk that I needed by creating one git-repository per folder.
Created a local parent git repository that contains all my git-svn-cloned folders.
Each git-repository was added as a sub-module to the parent git-repository.
The problem with Visual Studio is that it does not recognize the multiple projects outside the main project where I opened the solution. This solution is in a folder that contains the only files recognized by Visual Studio as being under git-source control.
I tried setting the git-preferences to use the upper level parent directory as the location of the git-repostitory without noticing any difference.
Suppose I have a project "MyFramework" that has some code, which is used across quite a few solutions. Each solution has its own source control management (SVN).
MyFramework is an internal product and doesn't have a formal release schedule, and same goes for the solutions.
I'd prefer not having to build and copy the DLLs to all 12 projects, i.e. new developers should to be able to just do a svn-checkout, and get to work.
What is the best way to share MyFramework across all these solutions?
Since you mention SVN, you could use externals to "import" the framework project into the working copy of each solution that uses it. This would lead to a layout like this:
C:\Projects
MyFramework
MyFramework.csproj
<MyFramework files>
SolutionA
SolutionA.sln
ProjectA1
<ProjectA1 files>
MyFramework <-- this is a svn:externals definition to "import" MyFramework
MyFramework.csproj
<MyFramework files>
With this solution, you have the source code of MyFramework available in each solution that uses it. The advantage is, that you can change the source code of MyFramework from within each of these solutions (without having to switch to a different project).
BUT: at the same time this is also a huge disadvantage, since it makes it very easy to break MyFramwork for some solutions when modifiying it for another.
For this reason, I have recently dropped that approach and am now treating our framework projects as a completely separate solution/product (with their own release-schedule). All other solutions then include a specific version of the binaries of the framework projects.
This ensures that a change made to the framework libraries does not break any solution that is reusing a library. For each solution, I can now decide when I want to update to a newer version of the framework libraries.
That sounds like a disaster... how do you cope with developers undoing/breaking the work of others...
If I were you, I'd put MyFrameWork in a completely seperate solution. When a developer wants to develop one of the 12 projects, he opens that project solution in one IDE & opens MyFrameWork in a seperate IDE.
If you strong name your MyFramework Assemby & GAC it, and reference it in your other projects, then the "Copying DLLs" won't be an issue.
You just Build MyFrameWork (and a PostBuild event can run GacUtil to put it in the asssembly cache) and then Build your other Project.
The "best way" will depend on your environment. I worked in a TFS-based, continuous integration environment, where the nightly build deployed the binaries to a share. All the dependent projects referred to the share. When this got slow, I built some tools to permit developers to have a local copy of the shared binaries, without changing the project files.
Does work in any of the 12 solutions regularly require changes to the "framework" code?
If so your framework is probably new and just being created, so I'd just include the framework project in all of the solutions. After all, if work dictates that you have to change the framework code, it should be easy to do so.
Since changes in the framework made from one solution will affect all the other solutions, breaks will happen, and you will have to deal with them.
Once you rarely have to change the framework as you work in the solutions (this should be your goal) then I'd include a reference to a framework dll instead, and update the dll in each solution only as needed.
svn:externals will take care of this nicely if you follow a few rules.
First, it's safer if you use relative URIs (starting with a ^ character) for svn:externals definitions and put the projects in the same repository if possible. This way the definitions will remain valid even if the subversion server is moved to a new URL.
Second, make sure you follow the following hint from the SVN book. Use PEG-REVs in your svn:externals definitions to avoid random breakage and unstable tags:
You should seriously consider using
explicit revision numbers in all of
your externals definitions. Doing so
means that you get to decide when to
pull down a different snapshot of
external information, and exactly
which snapshot to pull. Besides
avoiding the surprise of getting
changes to third-party repositories
that you might not have any control
over, using explicit revision numbers
also means that as you backdate your
working copy to a previous revision,
your externals definitions will also
revert to the way they looked in that
previous revision ...
I agree with another poster - that sounds like trouble. But if you can't want to do it the "right way" I can think of two other ways to do it. We used something similar to number 1 below. (for native C++ app)
a script or batch file or other process that is run that does a get and a build of the dependency. (just once) This is built/executed only if there are no changes in the repo. You will need to know what tag/branch/version to get. You can use a bat file as a prebuild step in your project files.
Keep the binaries in the repo (not a good idea). Even in this case the dependent projects have to do a get and have to know about what version to get.
Eventually what we tried to do for our project(s) was mimic how we use and refer to 3rd party libraries.
What you can do is create a release package for the dependency that sets up a path env variable to itself. I would allow multiple versions of it to exist on the machine and then the dependent projects link/reference specific versions.
Something like
$(PROJ_A_ROOT) = c:\mystuff\libraryA
$(PROJ_A_VER_X) = %PROJ_A_ROOT%\VER_X
and then reference the version you want in the dependent solutions either by specific name, or using the version env var.
Not pretty, but it works.
A scalable solution is to do svn-external on the solution directory so that your imported projects appear parallel to your other projects. Reasons for this are given below.
Using a separate sub-directory for "imported" projects, e.g. externals, via svn-external seems like a good idea until you have non-trivial dependencies between projects. For example, suppose project A depends on project on project B, and project B on project C. If you then have a solution S with project A, you'll end up with the following directory structure:
# BAD SOLUTION #
S
+---S.sln
+---A
| \---A.csproj
\---externals
+---B <--- A's dependency
| \---B.csproj
\---externals
\---C <--- B's dependency
\---C.csproj
Using this technique, you may even end up having multiple copies of a single project in your tree. This is clearly not what you want.
Furthermore, if your projects use NuGet dependencies, they normally get loaded within packages top-level directory. This means that NuGet references of projects within externals sub-directory will be broken.
Also, if you use Git in addition to SVN, a recommended way of tracking changes is to have a separate Git repository for each project, and then a separate Git repository for the solution that uses git submodule for the projects within. If a Git submodule is not an immediate sub-directory of the parent module, then Git submodule command will make a clone that is an immediate sub-directory.
Another benefit of having all projects on the same layer is that you can then create a "super-solution", which contains projects from all of your solutions (tracked via Git or svn-external), which in turn allows you to check with a single Solution-rebuild that any change you made to a single project is consistent with all other projects.
# GOOD SOLUTION #
S
+---S.sln
+---A
| \---A.csproj
+---B <--- A's dependency
| \---B.csproj
\---C <--- B's dependency
\---C.csproj