How to benchmark two different versions of the same non-NuGet library in BenchmarkDotNet? - benchmarkdotnet

Question

I'm trying to use BenchmarkDotNet to prepare performance regression testing for a library. This requires that I compare the same tests with an older (stable) version of the library. Now, there's an option to give different versions of a NuGet package to a job. There doesn't seem to be an option to reference different assemblies instead.
I've tried custom build configurations:
<ItemGroup Condition="'$(Configuration)' == 'Baseline'">
<Reference Include="MyAssembly">
<HintPath>lib\baseline\MyAssembly.dll</HintPath>
</Reference>
</ItemGroup>
<ItemGroup Condition="'$(Configuration)' != 'Baseline'">
<Reference Include="MyAssembly">
<HintPath>lib\current\MyAssembly.dll</HintPath>
</Reference>
</ItemGroup>
but when trying to use those configurations from BenchmarkDotNet via
public static void Main(string[] args) {
var summary = BenchmarkRunner.Run(typeof(Program).Assembly,
DefaultConfig.Instance
.With(Job.Default.WithCustomBuildConfiguration("Baseline"))
.With(Job.Default.WithCustomBuildConfiguration("Current")));
}
I get build errors that indicate that the assembly is not referenced at all. BenchmarkDotNet also helpfully clears up any temporary artifacts it creates so I can't even look at the generated project file to figure out how that looked.
Would the only workaround here be to wrap the library in a NuGet package? Or is there something I'm overlooking in the (for this case seemingly sparse) docs?
This issue seems to be vaguely related to the build errors I'm getting.

Answer 1

I was able to get something similar working, which may also work for you. In my case, I wanted to reference both the NuGet and the local project from the same build of the benchmark project.
After adding both the PackageReference and the ProjectReference, I (manually) added the Aliases to the ProjectReference:
<ItemGroup>
<ProjectReference Include="..\src\Combinatorics\Combinatorics.csproj" Aliases="localbuild" />
</ItemGroup>
The dll names do need to be different (since both dlls end up in the output directory). So I changed the dll name for the referenced project (but only when building locally and not when doing a continuous integration build):
<PropertyGroup>
<AssemblyName Condition="'$(CI)'!='true'">local.Combinatorics</AssemblyName>
</PropertyGroup>
Then I could use extern alias and using to use both assemblies in the same app:
extern alias localbuild;
using NugetCombinatorics = Combinatorics.Collections;
using LocalCombinatorics = localbuild::Combinatorics.Collections;
...
[BenchmarkCategory("Enumerate"), Benchmark(Baseline = true)]
public void EnumerateOld()
{
var permutations = new NugetCombinatorics.Permutations<int>(_source);
foreach (var p in permutations)
;
}
[BenchmarkCategory("Enumerate"), Benchmark]
public void EnumerateNew()
{
var permutations = new LocalCombinatorics.Permutations<int>(_source);
foreach (var p in permutations)
;
}
The duplicated code is not ideal, and I'm kinda abusing categories instead of using proper runs, but it gets the job done.