I am trying to get a C# Visual Studio 2019/MSBuild job to build on a Jenkins build server. I know that my file paths are too long, so I have enabled Long File Paths in the Group Policy Editor (and verified that it has persisted in the registry editor after a server restart).
However, now I am getting the following error "ALINK: fatal error AL1065: File name ... is too long or invalid".
A quick google search led me to this page for Alchemy Software. However I have no idea what Alchemy Software is and why it is being used in the build process and why it is failing. (Although for the last point, I'm guessing that the Alchemy Software .dll is not "Long Address Aware", which I believe is necessary for an application to take advantage of Long Paths in Windows. But since I can't locate any .dll or .exe associated with this software, I can't be certain.)
Does anyone know why my build is still failing with this error, what Alchemy Software is, and how to get it to take advantage of Long Paths in Windows?
P.S. And please, no comments about how I should restructure my file paths to be shorter. I have tried doing that but it's impractical for this application. And anyway, it keeps popping up and is becoming a whack-a-mole situation, so I'd really rather fix the root cause rather than constantly putting band-aids everywhere.
I am going to set this as an answer since, after Hans Passant's very helpful comments and subsequent research, I think it's pretty definitive that this can mostly only be worked around, not resolved. (As possible exception will be discussed at the end of this answer.)
As stated in those comments, this error originates from a linker module called al.exe that is utilized by MSBuild for certain project configurations (more on that later). This linker can be found in a couple places, but for me it was being called from C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools.
MSBuild can handle long file paths as long as Windows is configured to do so (by enabling long paths in the Group Policy Editor or by modifying the registry directly). However this al.exe module cannot. And as far as I can tell, there's no way to force it to do so. So if your build tool chain requires this al.exe module to be used, you're kind of SOL.
For my particular situation, for a job in Jenkins that was failing because my paths were too long, I worked around this by changing the workspace of my job. So now instead of the default of something like D:\Jenkins\workspaces\[product]\Releases\[product_version], I changed it to D:\j\dd0_1c, which is an encoding that makes sense to me. This shortening of the folder path avoids any subsequent file paths from exceeding the limit of 260 characters. It's not a satisfying solution, but it works for my particular situation.
I did mention that there was a possible exception to all of this: if you can get MSBuild to avoid using al.exe altogether, then you can avoid this error.
I don't know all of the scenarios or workflows in which MSBuild utilizes this module, but I do know that it does get utilized when your application has localized resources, and somehow, some way, MSBuild uses al.exe when it is generating those resources. This was exactly my scenario, and I found this page and this page describing how you can reconfigure your localization projects such that MSBuild does not utilize al.exe. I did try the steps described in these pages and was able to verify that I no longer got this ALINK error from al.exe. However I never got my project to fully build since this reconfiguration caused other build errors to crop up. So in the end, for the sake of expediency (and because it was cleaner than performing a major refactor of my code), I went with the Jenkins workspace workaround.
However, it is interesting to note that you can get MSBuild to avoid using al.exe as long as your project is conducive to the solution given in those two links. So hopefully, if someone runs into this same issue, they might have more success than I in utilizing this method.
Related
I have a VS2010 C# project, that references a large set of native .dll's (a commercial java runtime). These file are referenced as 'Content' files in the project, since the need to be copied with the project.
The code in these libraries is called using PInvoke, there is no assembly reference.
Every time I compile the solution, the Visual Studio testing framework tries to load all the referenced dll files, expecting to find .net assemblies which may contain unit tests. Since the are no .net assemblies, the following exception is thrown:
Error loading some.dll: Unable to load the test container 'e:\some.dll' or one of its dependencies. If you build your test project assembly as a 64 bit assembly, it cannot be loaded. When you build your test project assembly, select "Any CPU" for the platform. To run your tests in 64 bit mode on a 64 bit processor, you must change your test settings in the Hosts tab to run your tests in a 32 bit process. Error details: Could not load file or assembly 'file:///e:\some.dll' or one of its dependencies. The module was expected to contain an assembly manifest.
This takes a whole lot of time, and I would like to tell Visual Studio to not try to load these files.
How can I tell Visual Studio to stop trying to load these files?
Correct me if I got this wrong:
You are including the P/Invoke target binaries in to the VS solution because you want the binaries to be copied over to the target directory when the solution is built. You want this because the project will execute from the target directory as soon as the VS solution is built. Correct?
Often times VS packages (both default and 3rd party) try to get smart about the solution content and will follow certain triggers (which are difficult to contain and control by ourselves) and load the solution and project content in their own ways. Fighting the battle in this area has poor ROI than employing a simpler work around (below).
While I can't provide you with an authoritative answer on how to tell VS's test package to not load all binaries, I suggest removing such binaries from the project as 'content' and leave them in your source control where they are today. Add a post-build task that will copy the said binaries over to the target. This will still give you the same result as it is working today but, takes those binaries out of reach for the test probes.
You must check out configuration settings by just right clicking on your solution name and click on "Configuration Manager"
It will open a pop up window for Configuration Manager.
Check not for the platform your projects are using it is better to choose any CPU.
Hope this can help.Give it a try:)
Because thats what your exeception says as you have quoted
Thanks
I tried to repro this issue and found that the root cause is that you have set your test project to be compiled as !AnyCpu. Is there any particular reason why you would want this for managed test code?
So unless you change this you will continue to see this message.
If you want to continue using this configuration for your test project you would need to update your .testsettings file as suggested in the message.
Sorry if this seems remedial. I am including it for the sake of completeness.
General library behavior
A library can be referenced either in the project file (and so the compiler injects to code to load the references) or dynamically at runtime with LoadLibrary() or PInvoke calls. When a referenced library is loaded, a function at the entry point is run can in turn load any libraries it depends on. When loading the library, there is a well-known set of paths that Windows will search, including %WINDIR%\Assembly and the current directory. There's a lot of good conceptual information on Wikipedia about this. I recommend reading it.
Possible Root Causes
I can't tell from your question if you are having trouble building the application, building the tests, or executing either. Generally I would not expect PInvoke to cause compile errors.
Error during app build: VS generally will show you that you have a reference to a DLL it can't find. However, you may be missing a DLL that is needed to satisfy all the dependencies. To resolve, just add the reference to the missing DLL. (This is the simplest issue, so I'm guessing this isn't what you're seeing.)
Error during test build: Since your test will reference your application/library, it also needs to have the same reference. Usually the easiest way to ensure you are getting everything is to remove all references and add a reference to the project you are testing. It's possible you some additional libraries are necessary for some tests, but not your app/lib itself. These need to be added separately.
Error during app execution: This can happen when starting the application, or later when an call to the external library is made if late binding is used.
Error during test execution: This can happen the same as with app execution. However, tests can also be "partially built" to only execute a small number of tests. In these cases, some files may not be copied. Using the [DeploymentItem()] attribute, you can specify that a test requires the presence of certain files in the test or app/lib project to function. MSDN describes how this can be done.
Resolution
For #1 & #2 the solution lies in adjusting the references in the project.
For #3 & #4, it may get trickier. There is a similar question to yours regarding Windows Mobile here which you may find useful, especially referring to using dumpbin to list out library dependencies. You can also use procmon from SysInternals to monitor file access during compile or load to see which files are not found. Then you can either include the missing file, or remove the library referencing it.
Good luck. Hope this helps.
I've just learned the hard way that Visual Studio 2010 and MSBuild extremely lenient when it comes to which vcxproj MSBuild files they will successfully execute - they will overlook missing configurations for subtasks and single files and still successfully execute the build tasks. However this is causing problems for me as it is leading to inconsistent builds across multiple configurations, plus due to the internal inconsistencies in the project files I cannot necessarily edit them in Visual Studio's property view.
So far I have managed to get rid of the most problematic inconsistencies by editing the vcxproj files by hand but this is not really an acceptable strategy given that this particular solution contains over 80 project files.
Is there a tool that can check an MSBuild file for internal consistency and highlight missing configurations along the lines of "your project files says it offers configurations X, Y and Z but the custom build task for file X only supports configurations X and Z"?
Update: The specific problem I am trying to solve right now as opposed to the more general problem of linting the vcxproj file is that of missing conditionals for certain configurations. Unfortunately adding and updating the conditionals seems to require a little more than I can safely accomplish using a find-and-replace tool. Doing it programmatically and correctly would most likely require DOM manipulation and given that I am rather familiar with the internal structure of the build files so far it appears that using a text editor that is able to do basic structural XML validation is the quickest way to accomplish the task at hand.
What I would really like to see however would be a tool that can at least highlight these problems automatically to cut down on the time spent tracking them down.
Configurations are, in general, a VS concept. They are not built-in to MSBuild but achieved using Conditional attributes on property groups. Most likely your project files are valid MSBuild projects, but some of them don't build with default parameters - I'd suggest not to edit them by hand, but either
use a find-and-replace tool to fix them
write a small app that uses Microsoft.Build.Construction API to inspect and fix the project files
There's nothing that would perform this for you, I'm afraid.
This request is currently tracked under https://github.com/dotnet/msbuild/issues/1777
This is not a real answer yet, but as soon as the issue gets resolved to a tangible solution, I'll update it here.
We have recently taken over a project from an outsourcing company. This project uses Moles and Pex for unit testing, but since we have not had the project for long, I am not very familiar with the frameworks.
That being said, we are busy upgrading this project to run in .Net 4. I have resolved most of the issues that have jumped out, but there is one that I cannot get a handle on. Some of the unit tests cannot compile because of the error:
Could not load file or assembly 'Example.Assembly, Version=0.0.0.0,
Culture=neutral, PublicKeyToken=null' or one of its dependencies. The
system cannot find the file specified.
The part that baffles me is that it is a project reference and the assembly is being copied to the output directory of the unit test. Most of the other project references are found and I cannot spot any difference between the ones that work and the ones that do not. I am not sure if this problem has to do with the pex/moles frameworks, but I thought I would mention it.
I have tried the usual things of removing and adding all the references and regenerating the moles assemblies.
Has anyone else run into this problem? Any help would be greatly appreciated.
EDIT1: Ok, after some more investigation into the build output, it appears as if it is not moles, but the .accessor files that are not generated correctly. I get the exact same problem as asked in Unit test project cannot find assembly under test (or dependencies), but unlike his problem, mine does not go away after deleting the accessor.
EDIT2: Turns out is is a program called Publicize.exe which falls over with that error. Still no idea why though. Looking at Fusion logs is looks like it does not search under the working directory for the dll that it is trying to generate the accessors for. Running it manually on a bunch of assemblies from our solution, I find it works on some, but not on others. I have not been able to identify a difference between the ones that work and the ones that don't, though.
Thanks
Ah, yes. I have read this story many times, and have the tee shirt. I run through my usual Moles first-aid kit, when encountering any issue, including this one.
Perhaps, this question will provide some help: Am I the only one getting "Assembly Not Available in the Currently Targeted Framework"?
Ensure the Moles framework is properly installed on the workstation and/or build server
Ensure the Moles assemblies are being built (see the excluded "Moles Assemblies" directory)
Check your build profile -- it may need to be set to full framework profile
Triple check your output destinations and post-build commands -- I have seems some solutions that copy the output to another location
Try using the Visual Studio Pex/Moles extension, if you are not already doing so
An invasive fix-all process is to simply create an all-new solution, projects, and test projects, and then copy the existing code files into them. It's surprising how many issues can be resolved for various project-related errors. Basically, a hard reboot for the entire solution.
Since you are updating to .NET 4, you may as well go to 4.5, and used the productized version of Moles, called "Fakes". You'll find Fakes in the Visual Studio 2012 release candidate. This significant feature hasn't received much attention.
I'm responsible for some internal tools and I want to make debugging them as easy as possible on client machines. We used to ship with full source code and debugging information and built them in exactly the same location as the typical client install path. This made debugging extremely easy.
We have now moved to using Windows Installer to deploy our software, and we symbol serve the debugging information and source index the pdbs.
This now makes debugging a right pain, as for each client I want to debug, I have to change several options in Visual Studio to enable source indexing and add a file into a Visual Studio directory to stop it complaining about running the source control commands. I also can't search the source code easily as the files are only extracted on demand.
Is there a better way we can do this? I have tried hosting archives of the source code on a network share, and using symbolic links to alias the network share with the local hard drive on build, but that requires UNC paths, which the VS2008 manifest tool can't cope with.
We also have used an undocumented linker option (/sourcemap) which does let you redirect all the source paths to an arbitrary location, but this does not work for .Net apps (as that option is availble for link.exe)
Even if we supplied the source code with the installation, the paths in the PDB would still be wrong.
Is there any way to patch a PDB (supported or unsupported by Microsoft) to redirect the paths elsewhere?
I can't help you with a direct answer, but perhaps I can inspire you with a meta-answer.
Debugging in a production environment is fundamentally wrong. Others have tried it (Netscape comes to mind) and failed miserably. For something like tools, it's even more pertinent to ensure proper function through solid unit testing, and automated tests for your release. You can setup your tests to work with different qualities of input (from perfect-textbook to medium to poor to awful) and quantities (i.e. test limits). There's a lot of books available on testing and particularly unit testing.
I understand it's tough getting out of the day-to-day to make something like this work (I'm neck deep in it myself), but it's one of the sanest, if not the only sane way out.
Does anyone have a method to overcome the 260 character limit of the MSBuild tool for building Visual Studio projects and solutions from the command line? I'm trying to get the build automated using CruiseControl (CruiseControl.NET isn't an option, so I'm trying to tie it into normal ant scripts) and I keep on running into problems with the length of the paths. To clarify, the problem is in the length of paths of projects referenced in the solution file, as the tool doesn't collapse paths down properly :(
I've also tried using DevEnv which sometimes works and sometimes throws an exception, which isn't good for an automated build on a separate machine. So please don't suggest using this as a replacement.
And to top it all, the project builds fine when using Visual Studio through the normal IDE.
It seems that it is limitation of the MSBuild. We had the same problem, and in the end, we had to get paths shortened, because did not find any other solution that worked properly.
The SUBST command stills seems to exist so remapping the root of your build folder to a drive letter may save some characters if Judah Himango's solution is no good.
I solved similar issue by adjusting CSPROJ-file:
<BaseIntermediateOutputPath>$([System.IO.Path]::GetFullPath('$(MSBuildProjectDirectory)\..\..\..\Intermediate\$(AssemblyName)_$(ProjectGuid)\'))</BaseIntermediateOutputPath>
As the result during compilation CSC.EXE receives full path instead of relative one.
Thanks to harrydev for clue on how CSC.EXE operates with the paths.
There are two kinds of long path problems relevant to build. One is paths that aren't really too long, but have lots of "..\" in them. Typically, these are references' HintPath values. MSBuild should normalize these paths down to below the max limit, so that they work.
The other kind of path is just plain too long. Sorry, but these just won't work. After looking at it a fair bit, the problem is that there just isn't sufficient API support for long paths. The BCL team (see their blog) had similar problems. Only some of the Win32 API's support the \?\ format. Arbitrary build tools, and probably 98% of apps out there, don't; and worse would probably behave badly (think of all the buffers sized for MAX_PATH).
We came to the conclusion that until there's a big ecosystem effort to make long paths work, or Windows comes up with some ingenious way to make them work anyway (like the short paths mangling?) long paths just aren't possible for MSBuild to support. Workarounds include subst, as you found; but if your tree just is simply too deep, your only options are to build it in fragments, or to shorten the folder names. Sorry.
Dan/MSBuild
I found the problem to be that when the C# compiler (csc.exe) is called it uses the projects directory path PROJECTDIRECTORY together with the output path OUTPUTPATH by simply appending them as:
PROJECTDIRECTORY+OUTPUTPATH
However, if the OUTPUTPATH is relative i.e. "..\..\Build\ProjectName\AnyCPU_Debug_Bin\" and the project directory is pretty long then the total length is longer than 259 characters since the path will be:
PROJECTPATH+"..\..\Build\ProjectName\AnyCPU_Debug_Bin\"
instead of an absolute path.
If csc.exe would make an absolute path before calling Win32 functions this would work. Since in our case the absolute path length is less than 160 characters.
For some reason the call to csc.exe from visual studio is then different from MSBuild than it is from visual studio. Do not know why.
In any case, the problem can be resolved by changing either or both PROJECTDIRECTORY and/or OUTPUTPATH paths.
Have you tried DOS paths? Or the \\?\ prefix? The .NET BCL team blog has more info.
If the path length is 260, then there is warning resolving reference, for 259 or 261 of this error does not occur. I think there is msbuild bug.
I know there is already an accepted answer, but I had a different problem while using msbuild that gave me the same error output, and led me on a circular wild-goose chase. So, for future googlers, here goes:
We have a batch file that calls msbuild, but as the build machine can build for multiple versions of Visual Studio, each batch file calls vcvarsall.bat before it runs msbuild. This has the nasty side effect of stuffing the path completely full of the same thing over and over again. When it fills up, you get the error shown in the question above: The input line is too long. A simple Google search could make you think your paths are suddenly too long for msbuild.
In my case, it was as simple as killing the session of cmd.exe and restarting, as this reverted the environment variables to their native state.