Can MSBuild capture a list of the projects it has built "so far" during a build?
I would love a hypothetical #(ProjectsBuilt) item list I could access, which names the csproj files that have finished being been built.
You might ask "Why this feature?" In the abstract, it pertains to updating or exporting dependencies - particularly in an incremental build setting. I'll give two examples:
A Visual Studio solution contains a number of "plugin" projects. When some of the plugins have been updated, and built, a general-purpose script "captures" which projects were built, and immediately kicks-off nuget packaging and repository uploads.
A Visual Studio solution contains "implementation" projects and "interface" projects. In other words, some of the resulting assemblies are meant to be exported for use in other solutions. After some code changes have been made, the "interface" assemblies resulting from building any and all projects are then exported to a special "common" folder to be consumed by other applications.
Certainly the problems described above could be solved by having each "exported" or "plugin" project contain a post-build script to apply the appropriate dependency-exportation step. But it is anti-DRY to require a post-build step be added into every participating VS project.
If MS build would simply keep a running-tally of what projects have been built "so far," then at an appropriate build-step all the necessary "dependency exportations" could be done in one swoop, and from a centralized msbuild location. The projects targeted by this capability would not require any per-project build-steps at all. Nice!
Related
I have a solution which I have a number of unloaded projects. When I build the solution locally the unloaded projects are obviously not built. In Configuration Manager, I can't even select configuration, platform, build and deploy options for the unloaded projects.
Now I have created an Azure Pipeline which also builds the solution. But the unloaded projects are still being built.
How can I exclude building projects in my pipeline?
Now I have created an Azure Pipeline which also builds the solution.
But the unloaded projects are still being built.
Both Azure Devops Pipeline and VS IDE would call msbuild.exe to build. When building a solution using msbuild, it actually is running command msbuild xx.sln /t:build ....
Check the content of xx.sln file (solution file) we can find it defines definitions about the projects within the solution. That's why if you build the solution, all these projects will be built. This is expected behavior when building whole solution. Sample .sln file:
And it's VS IDE's magic to exclude unloaded projects when building whole solution. The IDE will check the unloaded projects and skip them when building whole solution, it has advantage that the unloading project behavior in IDE won't modify your solution file. (It's important for solution under Git version control).
To sum up: It's expected that building solution will build all projects. When we unload the projects in VS, this action won't affect our source solution file or project file. VS will automatically skip the unloaded projects, which is the unique feature of IDE. (Unload is a UI action in VS, it has no effect on .sln/.xxproj file)
To exclude projects to build in Azure Devops:
1.Specify projects to build when building whole solution. (It doesn't change any file, so that it won't affect Version control)
You can check this answer and corresponding document. My devops task setting (no need to specify Build target):
2.Exclude projects from solution configuration (Not recommended) or create a new configuration that excludes specific projects (Recommended). The solution configuration is defined in solution file (sln), so this way would makes effect on Source control.
Exclude projects from solution configuration(Not recommended):
Create a new configuration that excludes specific projects:
Related document here. I suggest you can do the changes via VS IDE, and then push the commits to remote repo. It's complex if you're not familiar with msbuild and sln file, so I doesn't suggest doing that yourself. Let IDE do the job for you.
Once the remote repo get the changes:
We can build the solution in devops with customrelease configuration which excludes ProjectA and ProjectB instead of release configuration which contains all projects.
All these workarounds had been tested on my side before posting them.
When opening a project you may have recently downloaded, Visual Studio 2013 briefly warns you to only open projects from trustworthy sources. What are some of the risks to opening projects? Can a project do any harm to your system before compilation, and what damage is possible? Are there any indicators you should be aware of before actually opening projects from "untrustworthy" sources? (Not that I can think of any)
Attack Surface within Visual Studio
There are many attack vectors within Visual Studio. All of them are by design. We developers want complete control over our systems within the build process. Unfortunately, when we say "I want to delete the contents of the cache directory on build", that also means malicious project files can delete just about anything from just about anywhere. Or worse. The compromise is the "Hey. We've given you the keys to the kingdom, but we don't recognize this project. Are you sure you want to open this? We're not responsible if it does something stupid" warning message that you mentioned.
Now consider that many developers run Visual Studio as an administrator.
Here are some of those attack vectors:
Pre- and Post-Build Events
In it's simplest form, untrusted projects could execute a Pre-Build event that would delete files. Or worse. Just about anything can be executed within a Build event. This is the 101 stuff that happens on Compile.
Executions on Project Open
Visual Studio project files are nothing more than big MSBuild configurations. There are a few MSBuild targets that Visual Studio executes when you open a project, all to support the tooling. These targets include Compile, ResolveAssemblyReferences, ResolveCOMReferences, GetFrameworkPaths, and CopyRunEnvironmentFiles. If any of these targets exist, the tasks within them are also executed. Delete files, or worse.
See: 'Design-Time Execution' at http://msdn.microsoft.com/en-us/library/ms171468.aspx
IntelliSense
Part of that tooling mentioned above includes IntelliSense, which executes the Compile task within MSBuild; the CSC/VBC must be executed to get all of the IntelliSense functionality. Because of the nature of IntelliSense, this task is repeatedly executed as you work, rather than the opportunities above that are just run once on open.
See: 'Design-Time IntelliSense' at http://msdn.microsoft.com/en-us/library/ms171468.aspx
Hidden Elsewhere in MSBuild
There is a sea of other routine MSBuild targets that you will manually execute throughout your day, including Build, Rebuild, Test, and Clean. Yes, keep in mind that even clean is a build target, so Clean could delete more than just your old \bin directories.
NuGet
Malicious projects may also expose systems via NuGet. Though Package Restore would not be an issue, the packages.config could specify a different Repository Source. Then, when you install a new package, such as install-package jquery, NuGet will retrieve the jQuery package from the untrusted alternate, rather than from nuget.org. This malicious jQuery package could have all sorts of other 'Goodies' within it that would be executed as a part of the package installation.
This is not a security vulnerability from NuGet, because "you" specified an alternate package source; this is by design, such as Corporations that have their own internal package repository.
What you can do?
At the end of the day, what can you do about this? The answer really is to just not open projects from untrusted sources. The project's packages.config file could be analyzed before you open it, but the big exposure is through MSBuild. Unless you are quite adept at reading through MSBuild schema, I would steer clear.
Normally "continuous package integration" involves source control, a build server, and participating teams fetching updated packages as often as they like. But I'm looking for a more extreme version of this story - without CM - that happens entirely on a developer's machine, all in one swoop. A more detailed description of what I want goes like this...
Using Visual Studio 2010 or 2012, assume a "foo.csproj" application that implements a plugin system. Each plugin represents a nuget package and has a corresponding Visual Studio project. Each of these projects is part of the same VS solution that contains the base application.
I want the following development story:
Change source code for a plugin.
Build solution, or perform a debug-launch, which causes msbuild to...
rebuild the changed plugin(s)
nuget then packages and uploads each plugin to a local repository (which can be just a subfolder of the VS solution)
rebuild the base application.
refresh the base application's nuget-plugin dependencies, which were just updated in prior steps. Notes:
This assumes MSBuild magically knows not to perform this last step until all plugins are built, packaged, and uploaded.
The "foo" application could itself use nuget.core to refresh the packages, but in this case I'm assuming that the VS build process did this step.
I would like to know if this story is common enough that there are "common" (msbuild?) scripts for this.
My own guess of how this should be handled is as follows:
All plugin projects are placed in a common "Plugins" folder somewhere in the VS solution folder structure.
The base application "project dependencies" are configured with references to all the plugin projects.
Note: I don't like the idea of managing these project dependencies manually.
The base application "foo.csproj" has a build step that scans the "foo.csproj" XML for dependencies it has in the "plugins" folder, and initiates the nuget packaging and deployment for each.
The base application then initiates the nuget "update all". Hopefully this is possible even though msbuild already mid-stride in execution.
In short, the base application is able to instantly consume plugins that have been altered. This is done without check-ins, a build-server, or manual and arbitrary requests to update plugin packages.
If pre-existing scripts do not already exist for this story, then I'll make my own. But I'd still like to know:
Can step 2, immediately above, be converted to something generic? That is, how can I convince msbuild not to build the "base application" until all projects in a particular folder have already been built? Remember, I'd like not to manage the project dependencies manually.
Is there anything flawed with this overall approach?
I would be particularly interested to know if there is an already existing nuget-visual-studio integration that assists with this story that I may have overlooked.
That's quite a long question to answer, so not sure I'm covering everything in this one; I'll do my best. First, your scenario is not uncommon. The first 2 steps of your planned approach seem OK to me (you're free to choose the location of the plug-in projects).
One thing's for sure: you'll have to manually define the build order, because your solution has no idea of knowing whether the projects consuming (NuGet) plug-ins have a dependency to the projects containing the source code for those plug-ins. Instead of using the built-in Build Order dialog in Visual Studio, take a look at this post on the MSDN blog for a correct way of doing this (or you might end up with something that works locally but not on the build server).
The key MSBuild elements in the referred post are the following:
<ProjectReference Include="... foo.csproj">
<ReferenceOutputAssembly>false</ReferenceOutputAssembly>
</ProjectReference>
Now, as for the packaging, deployment and consumption of those plug-ins:
Each plug-in project should trigger package creation and publication in a post-build step. This post on my blog contains ZIP-download with quite lot of MSBuild stuff you can use to get started. E.g. I version, package and publish the NuGet package for a class library in Release builds. I'm using the NuGet command line tool to pack (command reference) and push (command reference) the package.
The consuming application project(s) should run NuGet.exe update <packages.config> (command reference) in a pre-build step.
Also pay attention you're NOT running builds in parallel.
We have a solution file that contains some solution folders: libraries, unit-tests, applications, etc.
With Visual Studio 2010 we can build only the projects of a given folder by right-clicking on it and using "build" or "rebuild".
This is perfect on the developers' workstations, and we'd like to do the same on the continuous-integration server which uses MSBuild.
So, can we build a solution folder with MSBuild?
Or will we have to create a dedicated solution for each folder?
The conclusion of the below responses is that there isn't any built-in way to do that, but some workarounds:
create a dedicated solution with only the selected projects,
use a dedicated csproj that will build the selected projects,
add a new project which will reference the selected projects in the existing solution file.
I've chosen to use a dedicated solution file as it is the less intrusive and tricky way, though not the more flexible (the dedicated .csproj solution should offer full control).
Yes, you can build Solutions folders using MSBuild.
msbuild "framework.sln" /t:First\Project:Build
will build the project in the folder called First. However, disk folders and solutions folders must match (I tried it for my solution where they don’t first).
So all you'd have to do is choose a project that has all the other projects in that folder as dependencies.
Also see:
MSBuild command-line reference
The short answer is:
Yes, if you want to build specific set
of projects (grouped together in a VS
solution) with plain MSBuild you will
have to create a dedicated MSBuild
.proj for each specific set of
projects you want to build.
You will not have to create (and maintain) Visual Studio solutions to support your build, but in the end it's the same (an MSBuild .proj vs. a Visual Studio solution).
I have a similar scenario with a set of 60+ .NET projects (.btprpj BizTalk). For developing in Visual Studio, these projects are currently organized in 12 Visual Studio solutions.
For automated build and deploy I created a set of 50+ MSBuild .proj scripts to target every single component (or set of components) I need.
MSBuild offers a lot of possibilities for automating all the stuff around the actual build of the Visual Studio projects (export from version control systems, etc.).
There are a number of other questions related to this topic:
Whats a good standard code layout for a php application (deleted)
How to structure a java application, in other words: where do I put my classes?
Recommended Source Control Directory Structure?
Structure of Projects in Version Control
I could not find any specific to VSTF, which has some capabilities like Team Build, integrated Unit Testing, etc. I'm wondering if these capabilities lead to a slightly different source layout recommendation.
Please post example of high level directory structures that you have had good luck with an explain why you like them. I'll let people vote on a "best" approach and I'll award the answer in a few days.
Here is one that I like:
Private; all of the current system deliverables
Documentation; a rollup of all the documentation across the solutions that make up the product, output would be MSDN style documentation from Sandcastle
Common; Visual Studio SLN that contains all the projects that are common across all the other solutions.
Tools; Visual Studio SLN that contains all the projects whose output is a tool. Example might be a console app that performs a set of administrative task on the larger system
Developer; each developer has their own folder which they can use for storing whatever they want
Specific Developer (1..n); this contains any build settings, scripts, and tools that this specific developer chooses to store in the source control system (they can do whatever they want here)
Specific Deliverable Solution (1..n); Visual Studio SLN that contains all the projects for a specific major deliverable
Common; solution folder that contains Visual Studio Projects that are shared within the current solution
UI; solution folder that contains Visual Studio Projects that define user experience
DataLayer; solution folder that contains Visual Studio Projects that define a data access layer
Services; solution folder that contains Visual Studio Projects that define web services
Tools; solution folder that contains Visual Studio Projects that define tools specific to this deliverable (executable utilities)
Tests; solution folder that contains Visual Studio Projects that contain unit tests
Public; all of the external dependencies associated with the system (eg. 3rd party libraries)
Vendor; dependencies provided by a specific vendor
Build; Visual Studio SLN that contains code associated with the build of the project, in our case mostly custom MSBuild tasks and Powershell scripts
Target; each successful build of the product as well as point releases
Debug; all of the debug builds that are output from weekly builds and continuous integration. Developers do not manually manage this directory
Build Number; a directory that corresponds with the current build number
Solution Output; a directory that contains all the build output for each of the projects in a given solution
Release; all of the release builds that are output manually when a milestone is reached
Build Number; a directory that corresponds with the current build number
Solution Output; a directory that contains all the build output for each of the projects in a given solution
Note: All solutions will have a Tests folder and unit test projects.
A few thoughts:
Very few files in the root of the tree. On a large team, set permissions so that no one can add new files to the root of the tree, without some kind of authorization.
The default workspace will contain:
Tools contains all executable code required to build & run unit tests, including your custom tools and scripts (perhaps assuming that Visual Studio and PowerShell are already installed on the machine).
ReferencedAssemblies has things you pick up from somewhere else, including things you buy or download and things someone on the team wrote but isn't part of this project.
If available, source code should be in here, too, so you can service it yourself. (If not available, you're taking a big riks.)
Source - all the source code, including project files.
Documents - items that are not used as part of the build, but are necessary for the proper functioning of the development effort.
Binaries - bits that have been shipped to customers, including .PDBs and other artifacts necessary for servicing. (On small projects, I fork the sources for each release, but normally a tag/label is a better choice.)
Elsewhere (e.g. $/personal) have a place for each person to do with as they please ($/personal/USERNAME). For example, my side projects go here.