Tensorflow CPU performance anaconda vs custom build (cmake, on Windows) - performance

Question

I've build tensorflow from the master branch (commit 8cb6e535) with the latest Visual Studio Community with AVX2 support and compared the performance to the latest tensorflow build provided by anaconda (1.7.0), which was compiled to use SSE but not AVX ("Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2").
The result is that on my network architecture (Conv2D, MaxPool, ReLU, LSTM, Dense) the conda build is faster than the custom build:
batch size 32, 0.00302s vs 0.00237s (per sample, >20% faster)
The result is the average of serveral thousand batches and ommiting the first couple of averages to prevent warmup effects.
Some ideas:
the anaconda build uses MKL (for Eigen/TF)
a bug in the cmake script prevents the use of SIMD (and anaconda uses bazel to build), or one needs to do more to actually enable SIMD extensions in the cmake build (see below for how I built)
anaconda builds with cmake but some other cmake/compiler settings (/GL, /fp:fast, PGO?, ...)
the monolithic build is faster (could not find that option in the cmake build scripts)
Is there anyone who did a windows build that is provably faster than the anaconda build? Or has any idea what the anaconda people did to get a fast build?
Thanks
Build Details
I'm using cmake to build Tensorflow on Windows, following the instructions here. Make sure swig and cmake are in your PATH environment variable. For Visual Studio 2017 the command to initialize the environment is "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64.
I change /MP to /MP2 in "CMakeLists.txt" and "tf_core_kernels.cmake", otherwise I get "fatal error C1060: compiler is out of heap space" (see this issue on github, and occurances of /MP in tensorflow on github).
The cmake command is (ommiting path to swig, python, etc):
cmake .. -A x64 -T host=x64 -DCMAKE_BUILD_TYPE=Release -Dtensorflow_WIN_CPU_SIMD_OPTIONS="/arch:AVX2"
Output (shortened):
-- Building for: Visual Studio 15 2017
-- Selecting Windows SDK version 10.0.16299.0 to target Windows 6.1.7601.
-- The C compiler identification is MSVC 19.13.26132.0
-- The CXX compiler identification is MSVC 19.13.26132.0
-- [...]
-- Found PythonInterp: C:/Users/%username%/AppData/Local/Continuum/miniconda3/python.exe (found version "3.6.5")
-- Found PythonLibs: C:/Users/%username%/AppData/Local/Continuum/miniconda3/libs/python36.lib (found version "3.6.5")
-- Found SWIG: C:/Users/%username%/Repos/swigwin/swig.exe (found version "3.0.12")
Then the python wheel was build with:
MSBuild /p:Configuration=Release tf_python_build_pip_package.vcxproj
and installed into the conda environment already containing the dependencies with pip install ....
Other Info
Doing:
print("Tensorflow version: ", tf.__version__)
print("Compiler version: ", tf.__compiler_version__)
print("Monolithic build: ", tf.__monolithic_build__)
for both builds yields:
Anaconda:
Tensorflow version: 1.7.0
Compiler version: MSVC
Monolithic build: 1
Custom:
Tensorflow version: 1.8.0-rc1
Compiler version: MSVC 191326132
Monolithic build: 0