XCode universal library build phase - Lipo can't find files - xcode

I've read several tutorials and guides on how to have XCode create a universal library. Basically you add an aggregate target with a bash script build phase to build the separate targets and lipo them together.
I have my own small script (which works because of how I named my targets) but for some reason lipo can't find the files;
fatal error:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo:
can't open input file:
/Users/username/Projects/project/plugins/build/Release-iphoneos/libproject-plugins.a
/Users/username/Projects/project/plugins/build/Release-macos/libproject-plugins.a
(No such file or directory)
However when I
Run the exact same lipo command right after the failed build, everything works and the files are found.
Add bash logic in my script to wait for the file creation the issue persists.
Replace the lipo in my script with a simple ls, the files are there.
So I'm not sure what goes wrong, it doesn't seem like xcodebuild only creates the files after lipo is called (as I first thought).
The script;
targets=$(xcodebuild -list | sed -n '/Targets/,/^$/p' | grep -v -e 'Targets:\|all\|^$')
target_results=""
for target in $targets; do
xcodebuild ${ACTION} -target $target -configuration ${CONFIGURATION}
target_results="$target_results ${PROJECT_DIR}/build/${CONFIGURATION}-$target/libproject-plugins.a"
done
xcrun lipo -create "$target_results" -o "${PROJECT_DIR}/plugins-universal.a"

This is a bash problem/mistake.
You're passing all the file names as a single argument to lipo, so it's gonna look for a single file named /Users/username/Projects/project/plugins/build/Release-iphoneos/libproject-plugins.a /Users/username/Projects/project/plugins/build/Release-macos/libproject-plugins.a.
You should use an array for the file names instead.
Initialise it with () instead of "".
Add elements with +=(...) instead of ="$var ...".
Pass every element separately to lipo with "${var[#]}" instead of "$var".
Applied to your script:
targets=$(xcodebuild -list | sed -n '/Targets/,/^$/p' | grep -v -e 'Targets:\|all\|^$');
target_results=();
for target in $targets; do
xcodebuild ${ACTION} -target $target -configuration ${CONFIGURATION};
target_results+=("${PROJECT_DIR}/build/${CONFIGURATION}-$target/libproject-plugins.a");
done;
xcrun lipo -create "${target_results[#]}" -o "${PROJECT_DIR}/plugins-universal.a";

Technically this is not an answer to your question. But I would like to recommend that you choose a different option to what you are doing unless you really need to do it.
Building universal static libraries in this fashion is a very old way of doing things and as you are finding out, complicated and quite problematic to get working well.
A more current and much simpler approach (IMHO) is to build a framework. XCode has templates for frameworks. Frameworks are easier to work with and do not require you to do any of the messing around with multiple targets, bash and lipo as you are.
Further, you can then employ Carthage to manage them as dependencies for other projects with very little effort.

Related

Linker fails in sandbox when running through Bazel but works when sandboxed command is executed from the command line

I'm trying to get our cross-toolchain (standard Yocto toolchain) working with Bazel. I followed the instructions on https://github.com/bazelbuild/bazel/wiki/Building-with-a-custom-toolchain but the linker fails every time I try to build a simple test program. It says
external/toolchain_e6500/sysroots/x86_64-fslsdk-linux/usr/bin/powerpc64-fsl-linux/../../libexec/powerpc64-fsl-linux/gcc/powerpc64-fsl-linux/4.9.2/real-ld: cannot find /lib64/libc.so.6
external/toolchain_e6500/sysroots/x86_64-fslsdk-linux/usr/bin/powerpc64-fsl-linux/../../libexec/powerpc64-fsl-linux/gcc/powerpc64-fsl-linux/4.9.2/real-ld: cannot find /usr/lib64/libc_nonshared.a
external/toolchain_e6500/sysroots/x86_64-fslsdk-linux/usr/bin/powerpc64-fsl-linux/../../libexec/powerpc64-fsl-linux/gcc/powerpc64-fsl-linux/4.9.2/real-ld: cannot find /lib64/ld64.so.1
The sysroot is set properly. When running the linker command outside of the sandbox (e.g. with --spawn_strategy=standalone or just manually) it works always. The strange thing is that when I use --debug_sandbox and run the emitted command from the command line, it works, too.
I've been debugging this issue for two days now, including straceing the Bazel daemon and comparing the real-ld input but I didn't find anything suspicious.
There must be a difference between the execution environments but I'm out of ideas now. Here is the failing command as printed by --debug_sandbox:
(cd /home/sick/.cache/bazel/_bazel_sick/2a7ae5e27644389520091aa03d045c73/execroot/__main__ && \
exec env - \
PATH=/home/sick/bin:/home/sick/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/sick/bin \
PWD=/proc/self/cwd \
TMPDIR=/tmp \
/home/sick/.cache/bazel/_bazel_sick/2a7ae5e27644389520091aa03d045c73/execroot/__main__/_bin/linux-sandbox -t 15 -w /home/sick/.cache/bazel/_bazel_sick/2a7ae5e27644389520091aa03d045c73/sandbox/linux-sandbox/2/execroot/__main__ -w /tmp -w /dev/shm -D -- tools/compiler_e6500/e6500_gcc/powerpc64-fsl-linux-gcc -o bazel-out/e6500-fastbuild/bin/test '--sysroot=external/toolchain_e6500/sysroots/ppc64e6500-fsl-linux' -no-canonical-prefixes -pie -Wl,-z,relro,-z,now -Wl,-S -Wl,#bazel-out/e6500-fastbuild/bin/test-2.params)
You can take a look at the workspace here https://github.com/jasal82/bazel-cross-eval
I can provide the toolchain if needed.
UPDATE 2018-09-25
I did some more investigation after reading the answer regarding the linker script below. Section 4.4.2 in this manual says that
In case a sysroot prefix is configured, and the filename starts with
the / character, and the script being processed was located inside the
sysroot prefix, the filename will be looked for in the sysroot prefix.
Otherwise, the linker will try to open the file in the current
directory.
Obviously the script is not being considered to be located inside the sysroot prefix by ld and thus used literally (i.e. resolved relative to the current dir, probably the sandbox root). That would explain the observed behavior. However, I still don't understand why it does not happen when I run the command manually, i.e. not through the Bazel daemon.
Could this be related to the relative sysroot path used in the CROSSTOOL file? As far as I understood you cannot specify an absolute path to the sysroot due to the sandboxing. What is the recommended way to handle this in Bazel? I would like to avoid having to patch the toolchain.
Have a look at lib.so in your sysroot. It probably looks something like this:
/* GNU ld script
Use the shared library, but some functions are only in
the static library, so try that secondarily. */
OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /lib/x86_64-linux-gnu/libc.so.6 /usr/lib/x86_64-linux-gnu/libc_nonshared.a AS_NEEDED ( /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 )
Change the paths in it to be relative to the directory that libc.so is in. You'll have to perform a similar operation on libm.so and libpthread.so.
The GNU linkers resolve symlinks before evaluating whether a script is within the sysroot. The current Bazel Linux sandbox implementation makes a symlink farm of all an action's inputs. Thus, the libc.so.6 linker script is detected as being in the sysroot outside the sandbox but not within the sandbox.

How to do dry run with CMake?

With GNU make, a dry run can be performed via
make -n
Is there a similar option for CMake? The best I could come up with is
cmake ..
make -n | grep -vi cmake
However, I am being too restrictive and am wondering if there is an easier way to achieve this.
According to the command line syntax, if you use -- on the command line with cmake, the parameters following that are passed directly to the native build tool. Eg. what you did is somewhat equivalent to this:
cmake ..
cmake --build -- -n

Stop GNU make from compiling by default

I just want to use GNU make to compress some files.
So I wrote the Makefile as follows:
lib.tar.lzma: $(shell find ~/lib -name "*")
rm -f lib.tar.lzma
tar -cavf lib.tar.lzma -C ~/ lib/
However, after I run make, it automatically compile the c++ source code in that directory.
How can I stop it from compiling them? I just want to compress them.
Update:
I got the following error:
<builtin>: recipe for target '/home/xxx/lib/app' failed
It seems a built-in recipe.
(We don't know your entire Makefile and your full file tree, so this is only a guess; I assume that you have shown us a fragment of your much bigger Makefile)
However, after I run make, it automatically compile the c++ source code in that directory.
This is probably happening because your $(shell find ~/lib -name "*") is expanded to something containing your object files. Since they are in your dependencies their source file is recompiled if it is newer. BTW you might want to use instead $(shell cd .. ; find lib -name "*") or if lib has no subdirectory even $(wildcard ../lib/*)
You probably don't need any dependency for that lib.tar.lzma target, so just have:
lib.tar.lzma:
rm -f lib.tar.lzma
tar -cavf lib.tar.lzma -C ~/ lib/
BTW, that -C ~/ perhaps should be -C $$HOME since make use /bin/sh to run commands, and that POSIX shell don't know about ~ ; perhaps a -C .. might be better ...
Perhaps you might write some shell script make-backup.sh to do a more clever tar and you would then code
lib.tar.lzma: make-backup.sh
./make-backup.sh $#
However, perhaps you do have dependencies (e.g. if you need to archive some generated files). Then you need to list them explicitly and wisely (you certainly don't want to depend on all the files; perhaps only the source ones). Also, you might not need to archive any object files *.o, if you have some (but YMMV).
I recommend using make --trace or remake -x to debug your Makefile.
BTW, having a Makefile only for a backup is useless; write a shell script instead.
I also strongly recommend using some version control system (like git) if you don't use any. Notice that git has an archive subcommand which might be a more clever backup.

Is it possible to dump the AST while building an Xcode project?

I've been doing some work on analyzing Swift projects using their ASTs, and I would like to know if it is possible to generate it somehow when building a Swift project with Xcode.
Right now, I am able to print the AST on the terminal when running the swiftc -dump-ast command for single files and simple projects. However, it becomes harder when using it for more complex projects.
For this reason, I'd like to use xcode. I already tried to pass the -dump-ast flag to the compiler in Build Settings > Swift Compiler - Custom Flags > Other Swift Flags. The flag was indeed passed to the compiler (the output does report calling swiftc with the -dump-ast flag when building). I tried to build the project both with xcode and through the xcodebuildcommand below, but neither dumped the ast.
xcodebuild -target 'CompilingTest.xcodeproj' -scheme 'CompilingTest' -
configuration "Debug" -sdk iphoneos -arch "armv7"
CONFIGURATION_BUILD_DIR="TestBuild" ONLY_ACTIVE_ARCH=NO
Now, I'm reasoning that either Xcode's build process redirects swiftc's output to some file, or it silences it somehow. Any thoughts?
Any help would be greatly appreciated.
Dumping the AST of your app is not possible to do from only changing
Xcode build settings. The main reason for this is that Xcode is making a
lot of decisions about compiler flags to pass to swiftc which
are not all compatible with dumping the AST and you cannot stop Xcode from
doing this.
But it is possible to do outside of Xcode with Xcode's help (or in
Xcode using my script mentioned below). To do this you'll need to
capture the swiftc command Xcode runs for your project, and then
change it a bit to dump the AST.
First, build your project, then go to the Report Navigator in Xcode (the
last tab in the Navigator pane on the left). From here either save your
entire build log with the save button across the top, or copy the
swiftc command directly from Xcode. You're looking for the command
called "Compile Swift Sources" for your app target (NOTE: this only
contains the compile command for the one target, if you want the AST for
multiple targets you'll need to perform these steps multiple times). If
you can't find this step, you may need to clean your project and compile
again (or look at an older build log).
After you've copied the entire swiftc command from Xcode, you'll need
to head to the command line and change the command a bit. Here's what
you'll need to do:
Remove -emit-dependencies
Remove -emit-module -emit-module-path FILEPATH
Remove -emit-objc-header -emit-objc-header-path FILEPATH
Remove -c (right before -jN and the list of files)
Remove -parseable-output
Add -dump-ast
Append > output.ast 2>&1 to your shell command
Why these?
Numbers 1, 2, 3, and 4 must be removed because when dumping the AST
you cannot also emit another type of file. You can see the logic
around these errors in the open source Swift compiler
here.
Number 4 is a case of this since -c is an alias of -emit-object.
Number 5 makes it easier for you to redirect the AST output while also
stopping swiftc from outputting other information about this
command. If you plan to parse the output of this AST command from
another program instead of redirecting the output in the shell you
might want to leave this option in (you end up getting JSON with it).
Number 6 is what causes the command to output the AST. Depending on what you want to use this output for, you can
also consider using -print-ast which prints a more
class-dump style output vs
the more verbose, classical parsable AST output of -dump-ast.
Number 7 is simple shell redirection so that you can redirect the
(probably huge) output of your AST to a single file. Change
output.ast to whatever file you want. You'll need 2>&1 because the
AST is dumped to stderr instead of stdout. The order also matters
here.
Also note that Xcode's build log escapes spaces but not other characters
that shells may not like. For example if you have & in any of your
folder / directory paths, you'll have to escape that manually.
If all of this sounds like too much work, I threw together a script to
do this processing for you and you can set it up in Xcode. You can find
it on GitHub.

figure out ld command arguments from xcode run script build phase

How can I determine what will be the arguments of the ld command in the build process from inside a script running as "run script build phase"?
I was looking at xcodebuild -dry-run as an option, but then I need to understand what should be the arguments I supply it.
Any idea for a robust solution?
EDIT:
It seems that xcodebuild doesn't support LD and LDPLUSPLUS when the project includes swift source code. So the solution that #fpotter offered doesn't work on project with swift.
Any thoughts?
Xcode doesn't expose a nice way to do this. In a Run Script build phase, all you have to work with are the Xcode build settings provided to you in the environment.
If you really need the entire argument list to ld, there's a hack you can use. With the LD and LDPLUSPLUS build settings, you can make Xcode call a script of your own instead of the real ld. From that script, you could capture the args, call through to the real linker, and then do whatever post processing you like there rather than in a Run Script build phase.
Here's how you could do that:
Create an .xcconfig for your target.
It should look like this:
LD = $(SRCROOT)/ld-wrapper/clang
LDPLUSPLUS = $(SRCROOT)/ld-wrapper/clang++
SRCROOT points to your project's directory. The LDPLUSPLUS line is only required if your app has C++ or ObjC++ code. If you don't want to create an xcconfig, you can also add these as User-Defined build settings via the Xcode UI.
Create wrapper scripts for Xcode to call.
Install a script like this at <your project root>/ld-wrapper/wrapper.sh:
#!/bin/bash
set -o errexit
# Choose which clang to run (clang or clang++) depending on how we're invoked.
# If we're invoked via the 'clang' link, we'll run 'clang'. If we're invoked
# via the 'clang++' link, we'll run 'clang++'.
CLANG_PATH="$DEVELOPER_DIR"/Toolchains/XcodeDefault.xctoolchain/usr/bin/$(basename "$0")
"$CLANG_PATH" "$#"
echo "clang args: $#"
echo "do any post processing here."
Create symlinks for the wrapper script for clang and clang++:
cd <project root>/ld-wrapper
ln -s wrapper.sh clang
ln -s wrapper.sh clang++
That's it. It's ugly, but it works.

Resources