I have the following scenario:
A proprietary MacOS game that statically links a graphics framework MoltenVK into its main Mach-O x86-64 binary.
The version of MoltenVK linked in is very old.
I have a newer version of MoltenVK in the form of a .dylib (along with newer versions of the SPIRV compiler, libVulkan, etc. if they are needed, also in dylib form).
The older and newer version of MoltenVK are ABI compatible, meaning the exported symbol names and function signatures should be identical from the old to the new version of MoltenVK.
And the root cause of this journey into MacOS linkage:
The game does not run correctly on my version of macOS (10.15 Catalina Beta 3). I have isolated the problem to MoltenVK due to a crash backtrace.
I want to test whether updating MoltenVK will solve the problem, both as a temporary workaround and to help the developers isolate the problem.
Is it possible to force the binary to use a version of the symbols defined in a dynamically loaded .dylib instead of the version defined within the binary itself? I want to patch all symbols available in each of the .dylibs I have, because it would probably break if I only patched some symbols but not others (presumably MoltenVK only works if the code of each symbol in the framework is from the same version of MoltenVK).
Note: I am unable to recompile the main Mach-O binary of the game because I do not have source code. I am willing to bypass security safeguards on my local system to do this if it is possible at all; I accept the risk of doing dangerous things while running a Beta (non-production) OS anyway.
I'd prefer if answers and comments focus on the technical solution to the question asked, but if further justification is needed, I am trying to isolate this problem as quickly as possible to give the game's developers as much time as possible to fix it before the final release of macOS 10.15. If I stay silent, the problem has a chance that it won't be detected; then folks will upgrade to the final macOS 10.15 and notice the game doesn't work. That's not fun for anyone, because then we have to either stay on Mojave and wait for the game developer to update their game, or go without the game for possibly weeks or months.
Static linking implies the library gets effectively baked into your final executable binary. So there's no easy technical way of hooking the calls and redirecting them somewhere else (like DYLD_INTERPOSE or DYLD_INSERT_LIBRARIES allows for external dylibs).
Patching the binary would require going through each MoltenVK call the game makes and doing quite cumbersome post processing.
By post processing I mean either:
writing a dyld call using dlopen & dlsym tandem. You would still
need dlopen & dlsym symbols already used in the binary (they're part of
libSystem aka C std lib, but you still need the dedicated dyld opcodes
to be able to actually use them). Eventually you'd need to put the assembly opcodes somewhere in binary to make everything work. This will be quite hard.
firing lldb debugger, preparing the dlsym addresses to call by hand and patching the binary on the fly for each call (you'd probably need write permissions in __TEXT segment to do it, but that's the easy part). If you know what you're doing this is probably the most feasible approach. The main drawback is it's volatile, if you break something you'd start from scratch.
Add a LC_LOAD_DYLIB command to the binary and dyld opcodes referenced by LC_DYLD_INFO_ONLY , that would be super hard
In any case your best friends are Hopper dissassembler and MachOView to inspect the binary.
Elementary knowledge of x86 (and/or x86-64) assembly is a must to follow. I think playing with the original source code could be a way more viable option.
Related
If I build a DLL with Rust language, does it require libgcc*.dll to be present on run time?
On one hand:
I've seen a post somewhere on the Internet, claiming that yes it does;
rustc.exe has libgcc_s_dw2-1.dll in its directory, and cargo.exe won't run without the dll when downloaded from the http://crates.io website;
On the other hand:
I've seen articles about building toy OS kernels in Rust, so they most certainly don't require libgcc dynamic library to be present.
So, I'm confused. What's the definite answer?
Rust provides two main toolchains for Windows: x86_64-pc-windows-gnu and x86_64-pc-windows-msvc.
The -gnu toolchain includes an msys environment and uses GCC's ld.exe to link object files. This toolchain requires libgcc*.dll to be present at runtime. The main advantage of this toolchain is that it allows you to link against other msys provided libraries which can make it easier to link with certain C\C++ libraries that are difficult to under the normal Windows environment.
The -msvc toolchain uses the standard, native Windows development tools (either a Windows SDK install or a Visual Studio install). This toolchain does not use libgcc*.dll at either compile or runtime. Since this toolchain uses the normal windows linker, you are free to link against any normal Windows native libraries.
If you need to target 32-bit Windows, i686- variants of both of these toolchains are available.
NOTE: below answer summarizes situation as of Sep'2014; I'm not aware if it's still current, or if things have changed to better or worse since then. But I strongly suspect things have changed, given that 2 years have already passed since then. It would be cool if somebody tried to ask steveklabnik about it again, then update below info, or write a new, fresher answer!
Quick & raw transcript of a Rust IRC chat with steveklabnik, who gave me a kind of answer:
Hi; I have a question: if I build a DLL with Rust, does it require libgcc*.dll to be present on run time? (on Windows)
I believe that if you use the standard library, then it does require it;
IIRC we depend on one symbol from it;
but I am unsure.
How can I avoid using the standard library, or those parts of it that do? (and/or do you know which symbol exactly?)
It involves #[no_std] at your crate root; I think the unsafe guide has more.
Running nm -D | grep gcc shows me __gc_personality_v0, and then there is this: What is __gxx_personality_v0 for?,
so it looks like our stack unwinding implementation depends on that.
I seem to recall I've seen some RFCs to the effect of splitting standard library, too; are there parts I can use without pulling libgcc in?
Yes, libcore doesn't require any of that.
You give up libstd.
Also, quoting parts of the unsafe guide:
The core library (libcore) has very few dependencies and is much more portable than the standard library (libstd) itself. Additionally, the core library has most of the necessary functionality for writing idiomatic and effective Rust code. (...)
Further libraries, such as liballoc, add functionality to libcore which make other platform-specific assumptions, but continue to be more portable than the standard library itself.
And fragment of the current docs for unwind module:
Currently Rust uses unwind runtime provided by libgcc.
(The transcript was edited slightly for readability. Still, I'll happily delete this answer if anyone provides something better formatted and more thorough!)
Is anyone hardening their code in an attempt to detect injections? For example, if someone is trying to intercept a username/password via NSUrlConnection, they could use LD_PRELOAD/DYLD_LIBRARY_PATH, provide exports for my calls into NSUrlConnection, and then forward the calls to the real NSUrlConnection.
Ali gave excellent information below, but I'm trying to determine what measures should be take for a hostile environment, where a phone might be jail broken. Most applications don't have to care, but one class of apps do - high integrity software.
If you are hardening, what method(s) are you using? Is there a standard way to detect injections on Macs and iPhones? How are you defeating framework injections?
For iOS / CocoaTouch, loading dynamic libraries is not allowed* (except for the System frameworks). To build and distribute an Application thru the AppStore, you can only link with static libraries and system frameworks, no dynamic library.
So on iOS you can't use that for code injection, neither can you use LD_PRELOAD of course (as you don't have access to such environment variables on iOS).
Except for jailbroken iPhones probably, but people jailbreaking their iPhone should take upon themselves that jailbreaking is by definition lifting all securities provided by iOS to avoid things such as injections (so you can't expect to remove the lock on your door to avoid having to use your key… and still expect that you're still protected against thieves robbing your house ;-))
That's the advantage of the Sandboxing + CodeSigning + No dylib constraints on iOS. No Code injection possible.
(On OSX it is still possible anyway, inparticular using LD_PRELOAD)
[EDIT] Since iOS8, iOS also allows dynamic frameworks. But as that's still sandboxed (you can only load code-signed frameworks that are inside your application bundles, and can't load frameworks that comes from outside your app bundle) injection is still not possible*
*except if the user jailbreaks its phone but it means that s/he chose to get rid of all protections and purpose and thus put its phone at risk — we can't crack our phone security and still expect it to provide all the protections those securities provided
This is an answer specific to UNIX like operating systems, I apologize if it doesn't make sense for your question but I don't know your platform well. Simply don't create a dynamically linked executable.
There are two ways I can think of to do this. Method #2 is probably best for you. They're both similar.
Important for both, the executable must be statically compiled using -static at build time
Method 1 - static exe, manual load shared libraries by their trusted full paths
Manually dlopen each library you need via a full path and then get the function addresses via dlsym at runtime and assign them to function pointers to use them. You'll need to do this for every external function you want to use. I believe reentrant unsafe functions won't like this so for those that use static variables- you'll need to use the reentrant safe versions, these end with "_r" i.e. use strtok_r instead of strtok
This will be difficult or simple depending on what your app does and how many functions you're using.
Method 2 - Statically link the executable, period
You can solve your subversion problem by just linking a static executable to avoid using dynamic libraries at all. This will generate a much larger exe than the the dlopen()/dlsym() method. Build using the -static compile flag and instead of using, for example gcc bah.c -o bah lssl use gcc -static bah.c -o bah /usr/lib/libssl.a to use the statically compiled version of the libraries you need instead of the dynamic shared libraries. In other words, use -static and don't use -l while building
For either method:
Once built, use file bah to confirm the executable is statically linked. Or confirm by running ldd on it
Note you'll need statically compiled versions of all the libraries you're linking against present in your system. These files end with.a instead of .so)
Also note upgrading system libraries will not update your executable. If there's a new security bug in OpenSSL, you'll need to get the latest libssl.a and recompile it. If you use the dlopen()/dlsym() method you won't have this problem but you will have portability issues if symbols change in different versions
Each method has its pros and cons based on your needs.
Taking the method 1 dlopen and dlsym approach makes your code more "obfuscated" and smaller, but sacrifices portability in most cases so probably isn't what you want. The upside is that it can possibly benefit when security bugs are fixed system wide.
I have a product which bootloader and application are compiled using a compiler (gnuarm GCC 4.1.1) that generates "arm-elf".
The bootloader and application are segregated in different FLASH memory areas in the linker script.
The application has a feature that enables it to call the bootloader (as a simple c-function with 2 parameters).
I need to be able to upgrade existing products around the world, and I can safely do this using always the same compiler.
Now I'd like to be able to compile this product application using a new GCC version that outputs arm-eabi.
Everything will be fine for new products, where both application and bootloader are compiled using the same toolchain, but what happens with existing products?
If I flash a new application, compiled with GCC 4.6.x and arm-none-eabi, will my application still be able to call the bootloader function from the old arm-elf bootloader?
Furthermore, not directly related to the above question, can I mix object files compiled with arm-elf into a binary compiled with arm-eabi?
EDIT:
I think is good to make clear I am building for a bare metal ARM7, if it makes any difference...
No. An ABI is the magic that makes binaries compatible. The Application Binary Interface determines various conventions on how to communicate with other libraries/applications. For example, an ABI will define calling convention, which makes implicit assumptions about things like which registers are used for passing arguments to C functions, and how to deal with excess arguments.
I don't know the exact differences between EABI and ABI, but you can find some of them by reading up on EABI. Debian's page mentions the syscall convention is different, along with some alignment changes.
Given the above, of course, you cannot mix arm-elf and arm-eabi objects.
The above answer is given on the assumption that you talk to the bootloader code in your main application. Given that the interface may be very simple (just a function call with two parameters), it's possible that it might work. It'd be an interesting experiment to try. However, it is not ** guaranteed** to work.
Please keep in mind you do not have to use EABI. You can generate an arm-elf toolchain with gcc 4.6 just as well as with older versions. Since you're using a binary toolchain on windows, you may have more of a challenge. I'd suggest investigating crosstool-ng, which works quite well on Linux, and may work okay on cygwin to build the appropriate toolchain.
There is always the option of making the call to bootloader in inline assembly, in which case you can adhere to any calling standard you need :).
However, besides the portability issue it introduces, this approach will also make two assumptions about your bootloader and application:
you are able to detect in your app that a particular device has a bootloader built with your non-EABI toolchain, as you can only call the older type bootloader using the assembly code.
the two parameters you mentioned are used as primitive data by your bootloader. Should the bootloader use them, for example, as pointers to structs then you could be facing issues with incorrect alignment, padding and so forth.
I Think that this will be OK. I did a migration something like this myself, from what I remember I only ran into a problem to do with handling division.
This is the best info I can find about the differences, it suggests that if you don't have struct alignment issues, you may be OK.
Reference this question about compiling. I don't understand how my program for Mac can use the right -arch, compile with those -arch flags, the -arch flags be for the system I am on (a ppc64 g5), and still produce the wrong object code.
Also, if I used a cross compiler and was on Linux, produced 10.5 code for mac, how would this be any different than what I described above?
Background is that I have tried to compile various apache modules. They compile with the -arch ppc, ppc64, etc. I get no errors and I get my mod_whatever.so. But, apache will always complain that some symbol isn't found. Apparently, it has to do with what the compiler produces, even though the file type says it is for ppc, ppc64, i386, x_64 (universal binary) and seems to match all the other .so mods I have.
I guess I don't understand how it could compile for my system with no problem and then say my system can't use it. Maybe I do not understand what a compiler is actually giving me.
EDIT: All error messages and the complete process can be seen here.
Thank you.
Looking at the other thread and elsewhere and without a G5 or OSX Server installation, I can only make a few comments and suggestions but perhaps they will help.
It's generally not a good idea to be modifying the o/s vendor's installed software. Installing a new Apache module is less problematic than, say, overwriting an existing library but you're still at the mercy of the vendor in that a Software Update could delete your modifications and, beyond that you have to figure out how the vendor's version was built in the first place. A common practice in the OS X world is to avoid this by making a completely separate installation of an open source product, like Apache, using, for instance, MacPorts. That has its cons, too: to achieve a high-level of independence, MacPorts will often download and build a lot of dependent packages for things which are already in OS X but there's no harm in that other than some extra build cycles and disk space.
That said, it should be possible to build and install apache modules to supplement those supplied by Apple. Apple does publish the changes it makes to open source products here; you can drill down in the various versions there to find the apache directory which contains the source, Makefile and applied patches. That might be of help.
Make sure that the mod_*.so you build are truly 64-bit and don't depend on any non-64 bit libraries. Use otool -L mod_*.so to see the dynamic libraries that each references and then use file on those libraries to ensure they all have ppc64 variants.
Make sure you are using up-to-date developer tools (Xcode 3.1.3 is current).
While the developer tool chain uses many open source components, Apple has enhanced many of them and there are big differences in OS X's ABIs, universal binary support, dynamic libraries, etc. The bottom line is that cross-compilation of OS X-targeted object code on Linux (or any other non-OS X platform) is neither supported nor practical.
Been using Lua 5.0 in a Mac OS X universal binary app for some years. Lua scripts are compiled using luac and the compiled scripts are bundled with the app. They have worked properly in Tiger and Leopard, Intel or PPC.
To avoid library problems at the time, I simply added the Lua src tree to my Xcode project and compiled as is, with no problems.
It was time to update to a more modern version of Lua so I replaced my source tree with that of 5.1.4. I rebuilt luac using make macosx (machine is running Leopard on Intel).
Uncompiled scripts work properly in Tiger and Leopard, Intel and PPC, as always.
However, now compiled scripts fail to load on PPC machines.
So I rebuilt luac with the 'ansi' flag, and recompiled my scripts. Same error. Similarly, a build flag of 'generic' produced no joy.
Can anyone please advise on what I can do next?
Lua's compiled scripts are pretty much the raw bytecode dumped out after a short header. The header documents some of the properties of the platform used to compile the bytecode, but the loader only verifies that the current platform has the same properties.
Unfortunately, this creates problems when loading bytecode compiled on another platform, even if compiled by the very same version of Lua. Of course, scripts compiled by different versions of Lua cannot be expected to work, and since the version number of Lua is included in the bytecode header, the attempt to load them is caught by the core.
The simple answer is to just not compile scripts. If Lua compiles the script itself, you only have to worry about possible version mismatches between Lua cores in your various builds of your application, and that isn't hard to deal with.
Actually supporting a full cross compatibility for compiled bytecode is not easy. In that email, Mike Pall identified the following issues:
Endianess: swap on output as needed.
sizeof(size_t), affects huge string constants: check for overflow when
downgrading.
sizeof(int), affectsMAXARG_Bx and MAXARG_sBx: check for overflow when
downgrading.
typeof(lua_Number): easy in C, but only when the host and the target
follow the same FP standard; precision
loss when upgrading (rare case);
warn about non-integer numbers when
downgrading to int32.
From all the discussions that I've seen about this issue on the mailing list, I see two likely viable approaches, assuming that you are unwilling to consider just shipping the uncompiled Lua scripts.
The first would be to fix the byte order as the compiled scripts are loaded. That turns out to be easier to do than you'd expect, as it can be done by replacing the low-level function that reads the script file without recompiling the core itself. In fact, it can even be done in pure Lua, by supplying your own chunk reader function to lua_load(). This should work as long as the only compatibility issue over your platforms is byte order.
The second is to patch the core itself to use a common representation for compiled scripts on all platforms. This has been described as possible by Luiz Henrique de Figueiredo:
....
I'm convinced that the best route to
byte order or cross-compiling is
third-party dump/undump pairs. The
files ldump.c and lundump.c are
completely replaceable; they export a
single, well-defined, entry point. The
format of precompiled chunks is not
sacred at all; you can use any format,
as long as ldump.c and lundump.c agree
about it. (For instance, Rici Lake is
considering writing a text format for
precompiled chunks.)
....
Personally, I'd recommend giving serious consideration to not pre-compiling the scripts and thus avoid the platform portability issues entirely.
Edit: I've updated my description of the bytecode header thanks to lhf's comment. I hadn't read this part of the Lua source yet, and I probably should have checked it before being quite so assertive about what information is or is not present in the header.
Here is the fragment from lundump.c that forms a copy of the header matching the running platform for comparison to the bytecode being loaded. It is simply compared with memcmp() for an exact match to the header from the file, so any mismatch will cause the stock loader (luaU_undump()) to reject the file.
/*
* make header
*/
void luaU_header (char* h)
{
int x=1;
memcpy(h,LUA_SIGNATURE,sizeof(LUA_SIGNATURE)-1);
h+=sizeof(LUA_SIGNATURE)-1;
*h++=(char)LUAC_VERSION;
*h++=(char)LUAC_FORMAT;
*h++=(char)*(char*)&x; /* endianness */
*h++=(char)sizeof(int);
*h++=(char)sizeof(size_t);
*h++=(char)sizeof(Instruction);
*h++=(char)sizeof(lua_Number);
*h++=(char)(((lua_Number)0.5)==0); /* is lua_Number integral? */
}
As can be seen, the header is 12 bytes long and contains a signature (4 bytes, "<esc>Lua"), version and format codes, a flag byte for endianness, sizes of the types int, size_t, Instruction, and lua_Number, and a flag indicating whether lua_Number is an integral type.
This allows most platform distinctions to be caught, but doesn't attempt to catch every way in which platforms can differ.
I still stand by the recommendations made above: first, ship compilable sources; or second, customize ldump.c and lundump.c to store and load a common format, with the additional note that any custom format should redefine the LUAC_FORMAT byte of the header so as to not be confused with the stock bytecode format.
You may want to use a patched bytecode loader that supports different endianness.
See this.
I would have commented on RBerteig's post, but I apparently don't have enough reputation yet to be able to do so. In working on bringing LuaRPC up to speed with Lua 5.1.x AND making it work with embedded targets, I've been modifying the ldump.c and lundump.c sources to make them both a bit more flexible. The embedded Lua project (eLua) already had some of the patches you can find on the Lua list, but I've added a bit more to make lundump a little more friendly to scripts compiled on different architectures. There's also cross-compilation support provided so that you can build for targets differing from the host system (see luac.c in the same directory as the links below).
If you're interested in checking out the modifications, you can find them in the eLua source repository:
http://svn.berlios.de/wsvn/elua/trunk/src/lua/lundump.c
http://svn.berlios.de/wsvn/elua/trunk/src/lua/lundump.h
http://svn.berlios.de/wsvn/elua/trunk/src/lua/ldump.c
Standard Disclaimer:
I make no claim that the modifications are perfect or work in every situation. If you use it and find anything broken, I'd be glad to hear about it so that it can be fixed.
Lua bytecode is not portable. You should ship source scripts with your application.
If download size is a concern, they are generally shorter than the bytecode form.
If intellectual property is a concern, you can use a code obfuscator, and keep in mind that disassembling Lua bytecode is anything but difficult.
If loading time is a concern, you can precompile the sources locally in your installation script.
I conjecture that you compiled the scripts on an Intel box.
Compiled scripts are wildly unportable. If you really want to precompile scripts, you'll need to include two versions of each compiled script: one for Intel and one for PPC. Your app will have to interrogate which program it's running on and use the correct compiled script.
I don't have enough reputation to comment, so I have to provide this as an answer instead even though it's not an appropriate answer to the question asked. Sorry.
There is an Lua Obfuscator available here:
http://www.capprime.com/CapprimeLuaObfuscator/CapprimeLuaObfuscator.aspx
Full disclosure: I am the author of the obfuscator and I am aware it is not perfect. Feedback is welcome and encouraged (there is a feedback page available from the above page).