When does ExternalProject_Add build? - windows

I want to build openssl and link my project against it. In my project, I have a library called net which is the part that uses openssl. So in my net\CMakeList, I added
include_directories(
../
+ ../../../ext/openssl/inc32/
)
add_library(net STATIC ${sources})
+ ADD_DEPENDENCIES(net openssl)
In my ext folder that is used for organizing all external library, I have a fresh unzipped openssl source code in a folder named openssl. Then I edited ext\CmakeList
message(STATUS "Configuring OpenSSL")
set(openssl_dir openssl)
if (CMAKE_CL_64)
include(ExternalProject)
set(OPENSSL_CONFIGURE perl\ Configure\ VC-WIN64A)
set(OPENSSL_MAKE ms\\do_win64a\ &&\ nmake\ -f\ ms\\ntdll.mak)
ExternalProject_Add(openssl
PREFIX openssl
#-- Download Step ----------
SOURCE_DIR ${CMAKE_SOURCE_DIR}/ext/openssl
#--Configure step ----------
CONFIGURE_COMMAND ${OPENSSL_CONFIGURE}
#--Build Step ----------
BUILD_COMMAND ${OPENSSL_MAKE}
BUILD_IN_SOURCE 1
#--install Step ----------
INSTALL_COMMAND ""}
)
endif()
When I built, the compiler complained the it can't find include files, and the openssl source code was not built at all, since there is no out32dll and inc32.
My question is: When does ExternalProject_Add actually build the project? If I make my net library depending on openssl, does it mean when I build net it would need to check and build openssl first?

ExternalProject use add_custom_target internally to create project and according to its document
By default nothing depends on the custom target. Use ADD_DEPENDENCIES
to add dependencies to or from other targets.
So if no project depend on it, it will not build by default.
When this ExternalProject_Add actually builds the project?
At build time(not at cmake time).
If I make my net library depends on openssl, does it mean when I build net it would need to check and build openssl first?
No. Since you make the dependency, cmake will handle it and openssl will be build before your library.
PS: There's still a lot of work to do. Because cmake doesn't know where the openssl result is, your project may run into link error. Here is a good example from the answer of "What is the best way to build boost with cmake".It use add_library with IMPORTED GLOBAL and set_target_properties with IMPORTED_LOCATION_* to solve the problem

Install missed
First of all I think you missed an install step. See documentation:
nmake -f "ms\ntdll.mak" install
So simplified version of ExternalProject_Add is:
ExternalProject_Add(
OpenSSL
URL https://github.com/openssl/openssl/archive/OpenSSL_1_0_1h.tar.gz
CONFIGURE_COMMAND perl Configure VC-WIN64A "--prefix=${CMAKE_INSTALL_PREFIX}"
BUILD_COMMAND "ms\\do_win64a.bat"
COMMAND nmake -f "ms\\ntdll.mak"
BUILD_IN_SOURCE 1
INSTALL_COMMAND nmake -f "ms\\ntdll.mak" install
)
Find OpenSSL
Second, if you want to find openssl libraries and headers, you need to use find_package(OpenSSL) command:
find_package(OpenSSL REQUIRED)
message("Libs: ${OPENSSL_LIBRARIES}")
message("Includes: ${OPENSSL_INCLUDE_DIR}")
Builds
Now, if you put all this commands together in one project configure step will fail :) Because it's not how ExternalProject designed. Superbuild that made by few or one ExternalProject_Add commands must be configured first, then build step will install the libraries from external archives. This will conflict with your project, because openssl libraries can't be found - they are not installed by superbuild yet.
Fix
Split cmake code. Run superbuild first (i.e. download and install openssl):
> cmake -Hsuperbuild -B_builds\superbuild "-GVisual Studio 12 2013 Win64" -DCMAKE_INSTALL_PREFIX=ext\install
# note, no ext\install directory yet
> cmake --build _builds\superbuild
# install done, libraries can be found in ext\install
Then build your project, note that the libraries found on configure step:
> cmake -Huse -B_builds\use "-GVisual Studio 12 2013 Win64" -DCMAKE_INSTALL_PREFIX=ext\install
-- Found OpenSSL: .../ext/install/lib/ssleay32.lib;.../ext/install/lib/libeay32.lib (found version "1.0.1h")
Libs: .../ext/install/lib/ssleay32.lib;.../ext/install/lib/libeay32.lib
Includes: .../ext/install/include
> cmake --build _builds\use
# OK
Experimental
I have an experimental project that wrap all of this noise in one command (Windows tested on Visual Studio 2013):
hunter_add_package(OpenSSL)
find_package(OpenSSL REQUIRED)
https://github.com/ruslo/hunter

Related

cmake command issue while compile project of tbb

I tried using a demo to compile tbb in my project.
Link of the demo https://www.selectiveintellect.net/blog/2016/7/29/using-cmake-to-add-third-party-libraries-to-your-project-1
IDE for me is VS2013 and get an error about command 'make'
Performing build step for 'tbb44'
2> CMake Error at F:/CPPs/FAsT-Match-master/build/tbb44/src/tbb44-stamp/tbb44-build-Debug.cmake:49 (message):
2> Command failed: 2
2>
2> 'make' 'tbb_build_prefix=tbb44'
The original command is generated via tbb.cmake, for which it looks like
ExternalProject_Add(${TBB_PREFIX}
PREFIX ${TBB_PREFIX}
URL ${TBB_URL}
URL_MD5 ${TBB_URL_MD5}
CONFIGURE_COMMAND ""
# BUILD_COMMAND ${TBB_MAKE} -j${NCPU} tbb_build_prefix=${TBB_PREFIX}
BUILD_COMMAND ${TBB_MAKE} tbb_build_prefix=${TBB_PREFIX}
The one commented is the original and the one behind is modified.
Is this the problem of MSVC?
That blog post is just completely wrong. TBB comes with its own CMake build now, so there's absolutely no reason to go through ExternalProject like this.
Here's how I built it from source, using Visual Studio 2019 (the instructions should be pretty much the same). From a developer command prompt, using CMake 3.20:
D:\>git clone https://github.com/oneapi-src/oneTBB
D:\>cmake -S oneTBB -B oneTBB-build -DTBB_TEST=OFF
D:\>cmake --build oneTBB-build --config Release
D:\>cmake --build oneTBB-build --config Debug
D:\>cmake --install oneTBB-build --prefix oneTBB-install --config Release
D:\>cmake --install oneTBB-build --prefix oneTBB-install --config Debug
Setting TBB_TEST to OFF saves a lot of time waiting on TBB's tests to build. These commands install the Debug and Release binaries to D:\oneTBB-install. Of course, you can place this folder anywhere you like.
Once this is done, you may use your new TBB build from your project like so:
cmake_minimum_required(VERSION 3.20)
project(TBB-test)
find_package(TBB REQUIRED)
add_executable(main main.cpp)
target_link_libraries(main PRIVATE TBB::tbb)
The TBB package also includes libraries TBB::tbbmalloc and TBB::tbbmalloc_proxy.
When you build your project, you may point find_package to your TBB installation by setting the CMake variable TBB_ROOT to D:/oneTBB-install at the command line via:
D:\>cmake -S myProject -B myProject-build -DTBB_ROOT=D:/oneTBB-install

How to build libgit2 with embedded libssh2 on Windows

First we build libssh2 with the WinCNG backend using msvc:
cmake . -DCRYPTO_BACKEND=WinCNG -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=%LIBSSH2%
cmake --build . --target install
libgit2 docs say all we have to do is to set DEMBED_SSH_PATH, although it fails to mentation what to set it to. Maybe it's so obvious that it seems irrelevant. However: I'm not a C guy and have no clue of the cmake build process. From what I understand the folder where the result of the previous built resides with all it's subfolders such as include and bin should be the correct path, so using %LIBSSH2% should be fine.
When finally trying to build libgit2 running
cmake . -DBUILD_CLAR=OFF -DCMAKE_BUILD_TYPE=Release -DEMBED_SSH_PATH=%LIBSSH2% -DCMAKE_INSTALL_PREFIX="%LIBGIT2%"
cmake --build . --target install
It'll fail because it's unable to find the links to the header files / binaries. We see things along the line of:
error LNK2019: unresolved external symbol libssh2_init referenced in function
I already tried replacing the paths backslashes \ with normal slashes /, since this seems to be a common issue. I also tried explicitly setting
set CMAKE_INCLUDE_PATH=%LIBSSH2%/include
set CMAKE_LIBRARY_PATH=%LIBSSH2%/lib
both faild with the same result.
Can anyone provide a reproducible way to compile libgit2 with embedded libssh2?
Using EMBED_SSH_PATH means libssh2 sources will be compiled at the same time as libgit2s. This means you don't actually have to compile libssh2 at all.
I managed to get it to compile successfully by downloading the source for libssh2 and libgit2, then running the following commands:
cd libgit2
mkdir build
cd build
cmake .. -DBUILD_CLAR=OFF -DCMAKE_BUILD_TYPE=Release -DEMBED_SSH_PATH="C:/path-to-ssh/libssh2" -DCMAKE_INSTALL_PREFIX="%LIBGIT2%"
cmake --build . --target install
Internally, it looks for all files that match ${EMBED_SSH_PATH}/src/*.cif EMBED_SSH_PATH is set.
As for using WinCNG as the backend, using embedded SSH creates this file (libssh2_config.h) in the libssh2 directory:
#define HAVE_WINCNG
#define LIBSSH2_WINCNG
#include "../win32/libssh2_config.h"
I assume that means it is already the default backend for an embedded install.

How to build Qt5 as static libraries under Windows

To distribute a Qt appication to Windows users as a simple .exe file, one would need to link statically with a static version of Qt libraries. Assume we distribute our license under an open license so that static linking of Qt is allowed. So we just need static library binaries of Qt5Core, Qt5Gui, Qt5Widgets. How to get them?
The binary distribution of Qt5 comes with dynamic libraries only. It contains some .lib files as well - but beware, these are not the static libraries, but some auxiliary files. They can be distinguished from true static libraries by their size: true static libraries are much bigger (many MB in the case of Qt5 components). So we need to statically compile Qt5 ourselves. This turns out to be surprisingly difficult.
The official instructions (http://doc.qt.io/qt-5/windows-building.html) are almost useless: Way too short, they do not even convey an idea of the difficulty of the task. A precise, up-to-date, step-by-step guide how to compile Qt5 into static libraries is currently missing. We should have it here.
The short answer: Don't waste your time on this. Link dynamically, and let an installer distribute your application. This is the only mode that is seriously supported by Qt5 and by CMake. Working without their support and against their advise is close to hopeless. Setting up an installer is far easier (though no fun either - we currenly use black magic from https://hk.saowen.com/a/d1cf90fcfea6d511629fd5a6c8113808721a7f19656677e8a5fab370a8d35cd4).
The long (yet incomplete) answer, in case you want to outsmart me:
The following steps brought me deceptively close to a solution. I succeeded in building static libraries, but I failed to link my application: Upon getting hundreds of LNK2001 and LNK2019 errors caused by cryptic unresolved external symbols, I had to give up.
So here comes a step-by-step description of what worked for me in October 2018 on a Windows10 virtual desktop. For each installation step, a check is indicated. If a check fails then fix the problem before proceeding further.
To start, install some tools that are needed later on:
Perl, needed for zlib and openssl configuration:
Skip if "perl -v" works.
Get installer from https://www.activestate.com/activeperl
Run installer -> will install to C:\Perl64.
Check: Restart terminal and try "perl -v".
An editor that can handle Unix end-of-line. Notepad won't do. Install vim, emacs, or whatsoever.
chocolatey package manager, needed to install flex and bison:
Run admin shell (Circle Menu > search for "Command" > right-click on "Command Prompt" > run as Admin)
Copy download command from https://chocolatey.org/install#install-with-cmdexe
Paste command to admin shell, and watch installation
Check: in terminal, try command "choco"
flex and bison, needed by qtbase compilation:
It's not obvious to get the right version of flex. From gnuwin32.sourceforge.net I got an outdated flex that missed a command-line argument required during Qt compilation. Compiling flex from source introduces a tail of further dependences. The simplest solution I found uses the Chocolatey package manager.
In admin shell, run: "choco install winflexbison3". This creates a directory X that contains the binaries win_flex.exe and win_bison.exe and a supporting "data" folder. Find out the location of X. In my case, X=C:\ProgramData\chocolatey\lib\winflexbison3\tools
Check: cd X, and run "win_flex --version", "win_bison --version".
During Qt compilation, these tools will be needed under their standard names "flex" and "bison". Therefore we need symbolic links flex->win_flex, bison->win_bison.
mlink X\flex.exe X\win_flex.exe
mlink X\bison.exe X\win_bison.exe
note: mklink needs absolute paths to work reliably
note: the symlink must not go to another directory lest bison won't find the "data" folder
Add X to the %PATH%
Check: in any shell, try "flex --version" and "bison --version"
So far for the tools. Now we need two libraries that must be statically linked to Qt (magic learned from https://stackoverflow.com/a/41815812/1017348):
Build static zlib:
Download https://zlib.net/zlib1211.zip
Unpack to C:\Development\zlib-1.2.11
Edit the file win32\Makefile.msc with an editor that can handle Unix EOLs:
Find the line starting with CFLAGS
Replace -MD with -GL -MT -Zc:wchar_t-
Find the line starting with LDFLAGS
Replace -debug with -opt:icf -dynamicbase -nxcompat -ltcg /nodefaultlib:msvcrt
Build zlib using the following command (should take less than a minute):
nmake -f win32/Makefile.msc AS=ml64 LOC="-DASMV -DASMINF -DNDEBUG -I." OBJA="inffasx64.obj gvmat64.obj inffas8664.obj"
Check: as a result, the source directory must contain zlib.lib (856kB), inter alia.
Build static openssl library:
Download https://www.openssl.org/source/openssl-1.1.1.tar.gz
Unpack to C:\Development\openssl-1.1.1
Copy files from zlib: cd zlib-1.2.11; xcopy zconf.h ..\openssl-1.1.1\ ; same for zlib.h zlib.lib zlib.pdb
cd ..\openssl-1.1.1
perl Configure VC-WIN64A no-asm no-shared zlib no-zlib-dynamic threads --prefix=C:\opt\local_x64
note: I added "no-asm" to avoid installation of NASM (Netwide Assembler)
note: I changed prefix, since only admins can install to C:\Windows
Edits the file ''makefile'':
Find the line that starts with: CFLAG
Append: /Zc:wchar_t- /GL /Zi
Find the line that starts with: LDFLAGS
Replace /debug with /incremental:no /opt:icf /dynamicbase /nxcompat /ltcg /nodefaultlib:msvcrt
Find the line that starts with: CNF_EX_LIBS
Replace ZLIB1 with zlib.lib
Build: "nmake"
Check: directory must contain openssl.lib (size?)
Now we are ready to build qtbase from source, using the command-line version of the Visual Studio C++ compiler:
Download (update location for latest Qt): https://download.qt.io/archive/qt/5.11/5.11.2/submodules/qtbase-everywhere-src-5.11.2.zip
Unpacking takes ca 20'
Move the source directory to the Local Disk (C:\Development)
To work with Visual Studio, use a specially configured terminal. Use Taskbar>Circle>Search to launch "x64 Native Tools Command Prompt for VS 2017"
cd qtbase...
Check: configure --help
configure -platform win32-msvc2017 -opensource -confirm-license -release -static -openssl-linked no-dbus -no-libpng -no-libjpeg -nomake examples -nomake tests -prefix C:\opt\local_x64 -I C:\Development\openssl-1.1.1\include -L C:\Development\openssl-1.1.1 -D OPENSSL_LIBS=C:\Development\openssl-1.1.1\libssl.lib
while debugging "configure", it may be necessary to delete config.cache.
option -I does not work with relative paths
the label "win32" may look wrong if we want to use Qt in a 64bit application, but we are advised not to worry: building Qt under Windows: really with "-platform win32-msvc2017"?
I don't know how to get rid of Sql: https://bugreports.qt.io/browse/QTBUG-71253
libpng and libjpeg are needed by optional code. They are provided in src/3rdparty, which tends to provoke clashes with system provided library versions. Therefore, we build without PNG and JPG support.
nmake
Check: static libraries (big .lib and small .prl) land in directory lib/
54.7 MB Qt5Core.lib
23.4 MB Qt5Widgets.lib
18.2 MB Qt5Gui.lib
07.4 MB Qt5Network.lib
...
That's it, we have static Qt libraries. Only, as said above, this did not help me when I tried to link my application with these libraries.

Supporting configurations with ExternalProject_Add and Boost

I'm currently building Boost via ExternalProject_Add with this command:
ExternalProject_Add(Boost
PREFIX workspace/Boost
URL http://downloads.sourceforge.net/project/boost/boost/1.59.0/boost_1_59_0.tar.gz
URL_MD5 51528a0e3b33d9e10aaa311d9eb451e3
UPDATE_COMMAND ./bootstrap.sh
CONFIGURE_COMMAND ""
BUILD_COMMAND ./b2 --layout=tagged link=static -s NO_BZIP2=1 cxxflags="-std=c++11" install
INSTALL_COMMAND ""
BUILD_IN_SOURCE 1
)
The problem is that this will always build both release and debug variants/configurations for boost, no matter which configuration I actually want to build. How can I "pass" the current configuration to the ExternalProject_Add command in this case?
This seems to just work when the "internal" build tool, i.e. the one called by ExternalProject_Add is also CMake. I have no idea how to pass that information to Boost's b2, however.
The above sample is for Windows / VisualStudio2013, so a multi configuration CMake target, but ideally, I'd have something that works for single configuration targets as well.

Building LLVM OCaml bindings on Windows under MinGW using CMake?

I'm trying to build the LLVM OCaml bindings under MinGW on Windows using CMake. I've tweaked a little bit with the LLVMBuild.txt in the bindings directory without luck. How do I build them?
Edit: To answer ygrek's question:
First attempt:
svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
cd llvm/tools
svn co http://llvm.org/svn/llvm-project/cfe/trunk clang
cd ../projects
svn co http://llvm.org/svn/llvm-project/compiler-rt/trunk compiler-rt
cd ..
mkdir build
cd build
cmake -G "MinGW Makefiles" ..
mingw32-make
I realized that llvm/bindings/ocaml wasn't being built. I then connected the LLVMBuild.txt files in many directories with the CMake build system. llvm/bindings/LLVMBuild.txt didn't contain any instructions to build subdirectories so I added them:
[common]
subdirectories = ocaml
llvm/bindings/ocaml/LLVMBuild.txt didn't have an LLVMBuild.txt either so I added one:
[component_0]
type = Group
name = OCamlBindings
parent = Bindings
I tried building this, but the OCaml binding weren't built. I think it's because there are no LLVMBuild.txt in any of the ocaml subdirectories. I had considered adding LLVMBuild.txt files to all of the ocaml subdirectories but realized I didn't know what their dependencies were or how to specify them.
I didn't want to pursue this aimlessly if it isn't possible or if someone had already done the work. It seems like there's a disconnect between the CMake build system & the bindings directories.
Not a real answer, but still.
The note at the beginning of llvm/bindings/Makefile.ocaml reading
An ocaml library is a unique project type in the context of LLVM, so rules are here rather than in Makefile.rules.
seems to suggest that building ocaml bindings with cmake is not supported..

Resources