How to build openssl for M1 and for Intel? - macos

I have a project which needs to use Libcrypto - and I have two versions of Libcrypto (libcrypto.a (from OpenSSL 1.1.1) built for ARM64) and (lcrypto.a (from OpenSSL 1.0.2) for Intel). Leaving aside the issues of whether it's good practice or not to have two different versions, I can say that if I include libcrypto.a then I can build and run on M1 and it works fine on M1. If I include lcrypto.a then I can build and run on Intel and it works fine on Intel. What I can't do is include them both (linker error - The linked library 'lcrypto.a' is missing one or more architectures required by this target: arm64.) - and if I can't include them both then I can't build a fat binary, and my app is less than entirely useful!
My question is How can I include both in my project - or where can I get (and how can I include) a fat version of Libcrypto? I've looked at this https://github.com/balthisar/openssl-xcframeworks/releases and this https://developer.apple.com/forums/thread/670631 but I'm none the wiser. I think I built a Fat Binary - but the Fat Binary I thought that I built doesn't work for either architecture!

Use command lipo to combine binaries
Compile Intel and ARM versions separately (arm version requires Xcode 12).
export MACOSX_DEPLOYMENT_TARGET=10.9
cp -r openssl-1.1.1t openssl-1.1.1t-arm64
cp -r openssl-1.1.1t openssl-1.1.1t-x86_x64
Build the Intel half
cd openssl-1.1.1t-x86_x64
./Configure darwin64-x86_64-cc shared
make
NOTE: For openssl-1.1.1q use -Wno-error=implicit-function-declaration as a configure parameter
Build the Arm half
export MACOSX_DEPLOYMENT_TARGET=10.15 /* arm64 only with Big Sur -> minimum might be 10.16 or 11.0 */)
cd ../openssl-1.1.1t-arm64
./Configure enable-rc5 zlib darwin64-arm64-cc no-asm
make
NOTE: For openssl-1.1.1q use -Wno-error=implicit-function-declaration as a configure parameter
To create universal binary use command lipo:
cd ..
mkdir openssl-mac
lipo -create openssl-1.1.1t-arm64/libcrypto.a openssl-1.1.1t-x86_x64/libcrypto.a -output openssl-mac/libcrypto.a
lipo -create openssl-1.1.1t-arm64/libssl.a openssl-1.1.1t-x86_x64/libssl.a -output openssl-mac/libssl.a
Verify that resulting binary contains both architectures:
file libcrypto.a libssl.a
libcrypto.a: Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64]
libcrypto.a (for architecture x86_64): current ar archive random library
libcrypto.a (for architecture arm64): current ar archive random library
libssl.a: Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64]
libssl.a (for architecture x86_64): current ar archive random library
libssl.a (for architecture arm64): current ar archive random library
PS: If you plan to use dynamic library combine dylib files using lipo and run instal_name_tool
cd openssl-mac
install_name_tool -id '#rpath/libcrypto.1.1.1.dylib' libcrypto.1.1.1.dylib
install_name_tool -id '#rpath/libssl.1.1.dylib' libssl.1.1.dylib
otool -D libssl.1.1.dylib /* to verify */
Result:
libssl.1.1.dylib:
#rpath/libssl.1.1.dylib

Even though this question already has an accepted answer, I'd like to mention that I found an easier way to do this that doesn't require using lipo, just in case it helps make someone else's life easier.
The trick is to force it to compile for both architectures simultaneously.
Before calling Configure in the openssl source directory, create a file somewhere convenient (for the purposes of explaining I'll just have it in the home folder) named cc, and have it contain the following text:
#!/bin/bash
if [[ $* == *-arch\ x86_64* ]] && ! [[ $* == *-arch\ arm64* ]]; then
echo Forcing compilation with arm64
cc -arch arm64 $#
else
cc $#
fi
That script will automatically add -arch arm64 to any compilation command that only includes -arch x86_64, and leave all other compilation commands unmodified.
Give it execute permissions:
chmod a+x ~/cc
Then execute the following in your shell to force compilation with this shell script:
export CC=/Users/yourname/cc
Then proceed with configuring and building as though for arm64, but tell it to compile as x86_64:
./Configure enable-rc5 zlib no-asm darwin64-x86_64-cc
make
make install
The resulting static libs and dylibs will already be x86_64 / arm64 universal!

Related

ffmpeg refuses to compile for x86_64 arch on M1 mac

I'm trying to build a universal binary of ffmpeg on MacOS, by compiling it twice (for arm64 and x86_64) and then using the lipo tool to combine the resulting binaries. I'm on an M1 Mac, so uname -m returns arm64 on my machine.
When running the configure script like so:
./configure --arch=x86_64
It outputs:
install prefix /usr/local
source path .
C compiler gcc
C library
ARCH c (generic)
...
And after running make, inspecting the built binaries with lipo -archs reveals that it's building them for arm64.
The result is the same if I add --enable-crosscompile to the configure call.
Based on this post, I also tried --arch=x86, but that had the exact same result, configure script displayed arch as c (generic) and inspecting artefacts with lipo shows they are built for arm64 architecture.
Does anyone have any ideas? Why is the configure script just refusing to build for x86_64?
You can speify -cc to make clang compile x86_64 binary.
Try:
./configure --enable-cross-compile --prefix=./install_x86_64 --arch=x86_64 --cc='clang -arch x86_64'
Note: don't forget to clear the outdated files by
make distclean
By the way, I have successfully built universal binrary of FFmpeg, you can refer to my build scripts. make_compile.py compile both x86_64 and arm64 binraries, and make_universal.py use lipo to generate universal binrary.
Reference
https://ffmpeg.org/platform.html
https://github.com/FFmpeg/gas-preprocessor

Build FFTW for Apple M1 and the older intel devices

I built FFTW on my Apple m1 computer. When I run lipo -info libfftw3.a (which is located in .libs/libfftw3.a). It says it is of architecture ARM64.
In my Xcode I set the build target to 10.11, for backward compatibility.
Now when I add the FFTW library into my Xcode project, it complains:
The linked library 'libfftw3.a' is missing one or more architectures
required by this target: x86_64.
How can I solve this? Do I need to build the library in an Intel device and create a universal library with these two libraries together (using lipo) or what's the right way to solve this?
You can configure it also with this line and it will build x86_64 & arm64 binary.
./configure CFLAGS="-arch arm64 -arch x86_64 -mmacosx-version-min=11.0"
In order to build a static library for both Intel and Apple silicon archictures, I need to build FFTW for both arch individually and then make an universal library. Here is how I do it.
Go to FFTW root folder
Build for Intel arch
./configure --prefix=/path/to/build/folder/lib-intel CFLAGS="-arch x86_64 -mmacosx-version-min=10.11”
make
make install
Build for arm64
./configure --prefix=/path/to/build/folder/lib-arm64 CFLAGS="-mmacosx-version-min=10.11”
make
make install
Make an universal build
lipo -create path/to/build/folder/lib-intel/libfftw3.a path/to/build/folder/lib-intel/libfftw3.a -output libfftw3_universal.a
Include libfftw3_universal.a in the Xcode project

Building GCC on OS X 10.11

Building GCC (latest revision) on OS X 10.11.1 here, using the command line:
../gccx/configure --with-gmp="/opt/local" --with-mpfr="/opt/local" \
--with-mpc="/opt/local" --with-libiconv-prefix="/opt/local" --with-pkgversion="GCCX" \
--program-transform-name='s/^gcc$/gccx/; s/^g++$/g++x/' --enable-languages=c
Followed build instructions exactly, and getting this error:
g++ -std=gnu++98 -g -DIN_GCC -fno-strict-aliasing
-fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-format -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -DHAVE_CONFIG_H -DGENERATOR_FILE -fno-PIE -Wl,-no_pie -o build/genmatch \
build/genmatch.o ../build-x86_64-apple-darwin15.0.0/libcpp/libcpp.a build/errors.o build/vec.o build/hash-table.o ../build-x86_64-apple-darwin15.0.0/libiberty/libiberty.a Undefined symbols for architecture x86_64: "_iconv", referenced from:
convert_using_iconv(void*, unsigned char const*, unsigned long, _cpp_strbuf*) in libcpp.a(charset.o)
(maybe you meant: __Z14cpp_init_iconvP10cpp_reader, __cpp_destroy_iconv ) "_iconv_close", referenced from:
__cpp_destroy_iconv in libcpp.a(charset.o)
__cpp_convert_input in libcpp.a(charset.o) "_iconv_open", referenced from:
init_iconv_desc(cpp_reader*, char const*, char const*) in libcpp.a(charset.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) make[3]: *** [build/genmatch] Error 1 make[2]: *** [all-stage1-gcc] Error 2 make[1]: *** [stage1-bubble] Error 2 make:
*** [all] Error 2
(A complete log is available at https://gist.github.com/3cb5d044533e657f4add.)
After investigating gcc/Makefile, it seems that the BUILD_CPPLIB variable does not include $(LIBICONV), since it is in a stage1 bootstrap at the time of the error. The relevant section is preceded by
# For stage1 and when cross-compiling use the build libcpp which is
# built with NLS disabled. For stage2+ use the host library and
# its dependencies.
Yet clearly the stage1 build of build/genmatch is referencing libcpp, which uses symbols from libiconv. So something is amiss here.
How can I fix it?
General discussion
Building GCC on Mac OS X is an occasionally fraught process. I've had various issues with various versions of GCC and various versions of Mac OS X over the years. You can see an earlier explanation of what I did in Install GCC on Mac OS X — that was for building GCC 4.8.x on Mavericks 10.9.x (or possibly Mountain Lion 10.8.x); it also reports success building GCC 4.9.0 on Mavericks 10.9.x, but failure to do so on Yosemite 10.10.x.
This is an updated recipe for building GCC 5.2.0 on Mac OS X 10.11.1 El Capitan.
It starts off using XCode 7.1.1 — I don't know which other versions of XCode are OK.
Note that El Capitan has a feature SIP (System Integrity Protection) that was not in Yosemite and earlier versions. This means you cannot create arbitrary directories under /usr any more. I used to install in /usr/gcc/vX.Y.Z; that is no longer permitted in El Capitan. One major change, therefore, is that I now install in /opt/gcc/v.X.Y.Z.
I've found that having DYLD_LIBRARY_PATH set is problematic — especially on El Capitan. In a major break from the past, I'm now not setting that at all. Note that the scripts unset it. Note too that the script explicitly sets the phase 1 compilers CC and CXX to /usr/bin/clang and /usr/bin/clang++ respectively (the XCode compilers). The current versions of GCC require a capable C++ compiler instead of (or as well as) a C compiler.
I have occasionally had problems with libiconv, but at the moment I've evaded them by not having my own version installed. Similarly, I've occasionally had problems with some awk scripts in the GCC source. I had to hack it/them to get it to work OK. However, with release copy of GCC 5.2.0 source, I seem to be able to build directly out of the box.
If you've only got a single disk partition, this next point isn't crucial. If you have multiple disks, either make sure the target directory does not exist or ensure that its name is exactly what you want. On the machines at work (not Macs but Linux machines, etc), I still use /usr/gcc/vX.Y.Z as the 'official' install location, but the software ends up in some arbitrary file system where there's enough space, such as /work4/gcc, and eventually there is a symlink such that /usr/gcc/vX.Y.Z gets to /work4/gcc/vX.Y.Z. However, it is crucial that /work4/gcc/vX.Y.Z does not exist while GCC is being compiled because it will resolve the name via realpath() or its equivalent and embed /work4/gcc/vX.Y.Z into the binaries, rather than the neutral name /usr/gcc/vX.Y.Z. This limits the portability of the installation; any other machine that it is moved to has to have a directory /work4/gcc/vX.Y.Z, even though you asked to install it in /usr/gcc/vX.Y.Z.
Compiling GCC 5.2.0 on Mac OS X 10.11.1 with XCode 7.1.1
I had to work with down-versions of both GMP (5.1.3 instead of 6.0.0a) and ISL (0.14 instead of 0.15). The builds for the later versions both caused me trouble.
Note that I put the library code for GMP, MPC, MPFR, ISL and Cloog (see the GCC pre-requisites) in the GCC source directory so that GCC builds its own versions of these libraries. I've found that its the simplest way to ensure that GCC locates these libraries correctly.
Target directory: /opt/gcc/v5.2.0
Build time was about 2h 15m on a 17" MacBook Pro (early 2011) running Intel Core i7 at 2.3 GHz, with 16 GiB 1333 MHz DDR3 main memory, and a 750 GB 5400 rpm hard disk drive. The source occupies about 850 MiB; the build tree ends up at about 4.6 GiB — you need plenty of disk space. The installed code ends up at about 420 MiB.
Script used — extract-gcc-5.2.0.sh
#!/bin/bash
unset DYLD_LIBRARY_PATH
TAR=tar
VER_NUM=5.2.0
GCC_VER=gcc-${VER_NUM}
TGT_BASE=/opt/gcc
TGT_DIR=${TGT_BASE}/v${VER_NUM}
CC=/usr/bin/clang
CXX=/usr/bin/clang++
extract() {
echo "Extract $1"
$TAR -xf $1
}
if [ ! -d "$GCC_VER" ]
then extract ${GCC_VER}.tar.bz2 || exit 1
fi
(
cd ${GCC_VER} || exit
nbncl <<EOF |
cloog 0.18.1 tar.gz
gmp 5.1.3 tar.xz
# gmp 6.0.0 tar.lz
isl 0.14 tar.bz2
# isl 0.15 tar.bz2
mpc 1.0.3 tar.gz
mpfr 3.1.3 tar.xz
EOF
while read file vrsn extn
do
tarfile="../$file-$vrsn.$extn"
if [ ! -f "$tarfile" ]
then echo "Cannot find $tarfile" >&2; exit 1;
fi
if [ ! -d "$file-$vrsn" ]
then
(
set -x
extract "$tarfile" &&
ln -s "$file-$vrsn" "$file"
) || exit 1
fi
done
)
if [ $? = 0 ]
then
mkdir ${GCC_VER}-obj
cd ${GCC_VER}-obj
../${GCC_VER}/configure --prefix="${TGT_DIR}" \
CC="${CC}" \
CXX="${CXX}"
make -j8 bootstrap
fi
Script nbncl — non-blank, non-comment lines
#!/usr/bin/env perl
#
# Non-blank, non-comment lines only
use warnings;
use strict;
while (<>)
{
chomp;
s/\s+$//;
s/\s*#.*$//;
print "$_\n" unless /^$/;
}
First, see Jonathan Leffler's very complete answer. I have a few more suggestions here.
The gcc configuration and build process needs to find your system's native header files and C run-time libraries. Newer, clang-based versions of Xcode hide these pretty deeply, and older versions of gcc don't seem to know how to find them. To get gcc 4.6 to build at all, I had to create these symlinks:
ln -s /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include /usr
ln -s /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/lib/dylib1.10.5.o /usr/local/lib
ln -s /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/lib/crt1.10.5.o /usr/local/lib
ln -s /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/lib/bundle1.o /usr/local/lib
Your mileage will likely vary slightly: note that those pathnames underneath /Applications/Xcode.app/Contents have various version numbers baked in to them, which are likely to be different on your system.
(If, as Jonathan describes, the newest versions of MacOS don't allow you to put anything in /usr, you might have to create the /usr/include symlink in /usr/local/include, instead, and I suspect that would work, too.)
Also, this is mentioned elsewhere, but it's an unusual requirement, and easy to overlook: do not try to build gcc within its own source tree. Always create a build directory which is a parallel sibling, not a child underneath, of the directory into which you've extracted the gcc sources. That is, do not do this:
tar xzf gcc-x.y.z.tar.bz2
cd gcc-x.y.z # WRONG
mkdir build
cd build
../configure # WRONG
make
Instead, do this:
tar xzf gcc-x.y.z.tar.bz2
mkdir build
cd build
../gcc-x.y.z/configure
make
This is counterintuitive, I know, and it's not the way a lot of other packages work, but it definitely does work for gcc, and it's the recommended way to do it.
Another point: if you discover that your build is failing because you configured it improperly, such that you have to rerun configure with different options, it's safer to delete your entire build directory and start from scratch. The configure and build system sometimes, but it seems not 100% reliably, detects what might need rebuilding in that case. (Deleting and starting over is frustrating, I agree, but again, it can really save time in the long run.)
Finally, if you're trying to build a cross-compiler, see some additional suggestions and commentary at install gcc 4.6.1 on OS X 10.11 .
For what it's worth, MacPorts has ports for all recent versions that should be sufficiently easy for everyone (who knows how to code!) to read who doesn't want to install MacPorts but prefers to install the various dependencies mentioned here some other way.
A slightly tweaked personal version of the port for gcc 6.3.0:
https://github.com/RJVB/macstrop/blob/master/lang/gcc6/Portfile
The reason I mention that one (and post this answer) is that this tweaked version shows how to get G++ to use libc++ instead of libstdc++. That's a prerogative for being able to use G++ as a real replacement for clang++ that can be used without worrying about C++ runtime incompatibilities. This patch has allowed me to use g++ to build KDE (KF5) code and run it against Qt5 and the KF5 frameworks built with various clang compiler versions. (The patch files are in .../gcc6/files .)
Some explanation that might help interpreting the Tcl code of the linked file:
Ignore anything that's specific to $subport == "libgcc".
As you can see, you need gmp, mpc, mpfr and isl (the other dependencies should be of no interest if you're installing on your own).
The configure.args expressions construct the argument list to the configure script, configure.env and build.env add environmental variables for the configure and build (make) commands. Many of the configure options here are to ensure that the build uses dependencies from MacPorts but they'd probably be required too if you want or have to use a location not controlled by SIP and that isn't included in standard PATH definitions (the compiler still ought to work when invoked through a process that resets the path).
The configure and build are done in a build directory that sits next to the source directory, which makes it very easy to start over or just clean up without throwing away the sources.
After the configure step the build is done with "make bootstrap-lean" - which still creates about 1.7Gb of data in that build directory.

compile Boost as static Universal binary lib

I want to have a static Universal binary lib of Boost. (Preferable the latest stable version, that is 1.43.0, or newer.)
I found many Google hits with similar problems and possible solutions. However, most of them seems outdated. Also none of them really worked.
Right now, I am trying
sudo ./bjam --toolset=darwin --link=static --threading=multi \
--architecture=combined --address-model=32_64 \
--macosx-version=10.4 --macosx-version-min=10.4 \
install
That compiles and install fine. However, the produced binaries seems broken.
az#ip245 47 (openlierox) %file /usr/local/lib/libboost_signals.a
/usr/local/lib/libboost_signals.a: current ar archive random library
az#ip245 49 (openlierox) %lipo -info /usr/local/lib/libboost_signals.a
input file /usr/local/lib/libboost_signals.a is not a fat file
Non-fat file: /usr/local/lib/libboost_signals.a is architecture: x86_64
Edit: It seems that the command was wrong and I must remove the "--" for most options. So the command I am trying now (-a just means to rebuild all):
sudo ./bjam -a toolset=darwin link=static threading=multi \
architecture=combined address-model=32_64 \
macosx-version=10.4 macosx-version-min=10.4 \
install
However, this gives many strange errors (what I already had earlier), all like this:
darwin.compile.c++.pch bin.v2/libs/math/build/darwin-4.2.1/release/address-model-32_64/architecture-combined/link-static/macosx-version-min-10.4/macosx-version-10.4/threading-multi/../src/tr1/pch.hpp.gch
In file included from ./boost/math/special_functions/acosh.hpp:18,
from ./boost/math/special_functions.hpp:15,
from libs/math/build/../src/tr1/pch.hpp:9:
./boost/config/no_tr1/cmath.hpp:21:19: error: cmath: No such file or directory
This could be another problem I have when building Universal binaries: g++ on MacOSX doesn't work with -arch ppc64
I found the problem. It seems that the MacOSX 10.4 SDK is missing a bunch of symlinks for GCC 4.2.
Use this as a test case:
g++ on MacOSX doesn't work with -arch ppc64
It will report multiple errors with GCC 4.2 (missing C++ includes, missing C includes, missing libs). In all cases, you can just fix that by setting a symlink. Search in your SDK for the file and just set the symlink in the same way it is in the MacOSX 10.5 SDK.
After that, it all just worked.
We use Boost compiled for 10.4 here at work. We don't use GCC 4.2 on it though, rather we use GCC 4.0 as Apple's GCC 4.2 is not supported for the MacOS 10.4 SDK. To accomplish this you need a bjam user config file, eg. user-config-darwin.jam. Here's the contents of ours. Modify to your heart's content:
# Boost.Build Configuration
# Compiler configuration
using darwin : 8.11 : /usr/bin/g++-4.0 :
<architecture>"combined"
<address-model>"32" # this can be changed to 32_64 for 32/64 universal builds
<macosx-version>"10.4"
<macosx-version-min>"10.4"
# <root>"/Developer"
<compileflags>""
<linkflags>"" ;
Then, you need to tell bjam to use the user config jam file when compiling:
bjam --user-config=user-config-darwin.jam ... (your other options go here) ...
Now you don't have to mess with symlinks in the system SDK directories.
To build 4-way universal boost static binaries on OSX 10.6 I do the following:
Download boost from the boost website.
Extract the archive and cd into the boost_1_xx_0 folder (where xx is the version of boost you are using).
Run:
./bootstrap.sh and then
./bjam macosx-version=10.6 macosx-version-min=10.4 architecture=combined threading=multi link=static address-model=32_64
This will compile everything except for Boost.MPI (which requires the --with-mpi option). The build products get put in ./stage

Building crti.o for i386

I am trying to build a cross-compiler with x86_64 being the host and i386 being the target. I'm getting the (all to common) crti.o: No such file error. Instead of grabbing an already built crti.o and crtn.o from a distro... how might I go about building these files explicitly from glibc (or possibly gcc) sources?
FYI, I am well aware of the -m32 option for x86_64 compilers. I'd prefer to just have a 32bit-only compiler environment. Also, the reason I don't want to use any of the gazillion already build i386 compilers is because I plan on mixing and matching glibc/binutils/gcc versions depending on my testing needs.
Thanks,
Chenz
Here's one possibility (from here)
You need to install your distro's 32
bit libc-dev package, or you need to
--disable-multilib which will result in a compiler that doesn't support 32
bit mode.
Are you sure you're using configuring the cross-compile correctly? It should be
CBUILD = CHOST = x86_64-pc-linux-gnu
CTARGET = i386-pc-linux-gnu
as you're running a build on an x86_64, for a compiler to run on an x86_64, which generates code for an i386.
If you used CHOST = i386-pc-linux-gnu, you'll be trying to generate 32-bit binaries, which will need to link with a 32-bit libc. Which is fine, if you already have a 32-bit libc, but it sounds like you don't.
i.e.
$ tar xvjf gcc-*.tar.bz2
$ cd gcc-*/
$ mkdir build
$ cd build
$ ../configure --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --target=i386-pc-linux-gnu

Resources