In an attempt to build 64-bit PicoLisp on Mac OS X (10.6.7) it seems I've encountered a problem with the OSX Dev. Tools GNU assembler ('as'), which is version 1.38. My Xcode is version 3.2.4. The lines in my (generated) x86-64.darwin.base.s that cause the problems typically look like this:
call foo#plt
... and they give me error messages like this:
x86-64.darwin.base.s:41694:junk `#plt' after expression
64-bit PicoLisp builds without problems on Linux and SunOS. These platforms probably have more capable (newer) versions of GNU assemblers. I don't know if newer GNU assemblers are avilable for OSX/Darwin ... or could there be other solutions?
If you'd like to try this yourself on OSX/Darwin, I can give you some files that are needed.
Unfortunately, I think there are at least two significant issues here:
"PLT" is an ELF concept, but OS X uses a completely different object / executable file format - Mach-O. Mach-O uses a similar mechanism but with a different name.
MacOS prefixes asm symbol names with a leading underscore. So you want call _foo. call foo would assemble but not link. As on GNU/Linux (ELF), the linker indirects through dyld_stub entries for you when you call a symbol that turns out to be in a dynamic library.
(Non-interposable symbols are the default even when building shared libraries, unlike on ELF systems)
Apple's as appears to be derived from a fork of a much earlier version of the GNU assembler, and, in some places, the syntax (and command line options) are rather different from recent versions (even where the concepts are the same).
It looks like there has been some work on i386 and x86-64 Mach-O support quite recently in binutils; it might be worth investigating the most recent version (2.21). But if the generated assembly code is trying to do clever things which are ELF-specific, you're probably going to have some major problems anyway...
PicoLisp has been supported on Mac for quite some time now. Just go to the standard download site.
Related
I have the following scenario:
A proprietary MacOS game that statically links a graphics framework MoltenVK into its main Mach-O x86-64 binary.
The version of MoltenVK linked in is very old.
I have a newer version of MoltenVK in the form of a .dylib (along with newer versions of the SPIRV compiler, libVulkan, etc. if they are needed, also in dylib form).
The older and newer version of MoltenVK are ABI compatible, meaning the exported symbol names and function signatures should be identical from the old to the new version of MoltenVK.
And the root cause of this journey into MacOS linkage:
The game does not run correctly on my version of macOS (10.15 Catalina Beta 3). I have isolated the problem to MoltenVK due to a crash backtrace.
I want to test whether updating MoltenVK will solve the problem, both as a temporary workaround and to help the developers isolate the problem.
Is it possible to force the binary to use a version of the symbols defined in a dynamically loaded .dylib instead of the version defined within the binary itself? I want to patch all symbols available in each of the .dylibs I have, because it would probably break if I only patched some symbols but not others (presumably MoltenVK only works if the code of each symbol in the framework is from the same version of MoltenVK).
Note: I am unable to recompile the main Mach-O binary of the game because I do not have source code. I am willing to bypass security safeguards on my local system to do this if it is possible at all; I accept the risk of doing dangerous things while running a Beta (non-production) OS anyway.
I'd prefer if answers and comments focus on the technical solution to the question asked, but if further justification is needed, I am trying to isolate this problem as quickly as possible to give the game's developers as much time as possible to fix it before the final release of macOS 10.15. If I stay silent, the problem has a chance that it won't be detected; then folks will upgrade to the final macOS 10.15 and notice the game doesn't work. That's not fun for anyone, because then we have to either stay on Mojave and wait for the game developer to update their game, or go without the game for possibly weeks or months.
Static linking implies the library gets effectively baked into your final executable binary. So there's no easy technical way of hooking the calls and redirecting them somewhere else (like DYLD_INTERPOSE or DYLD_INSERT_LIBRARIES allows for external dylibs).
Patching the binary would require going through each MoltenVK call the game makes and doing quite cumbersome post processing.
By post processing I mean either:
writing a dyld call using dlopen & dlsym tandem. You would still
need dlopen & dlsym symbols already used in the binary (they're part of
libSystem aka C std lib, but you still need the dedicated dyld opcodes
to be able to actually use them). Eventually you'd need to put the assembly opcodes somewhere in binary to make everything work. This will be quite hard.
firing lldb debugger, preparing the dlsym addresses to call by hand and patching the binary on the fly for each call (you'd probably need write permissions in __TEXT segment to do it, but that's the easy part). If you know what you're doing this is probably the most feasible approach. The main drawback is it's volatile, if you break something you'd start from scratch.
Add a LC_LOAD_DYLIB command to the binary and dyld opcodes referenced by LC_DYLD_INFO_ONLY , that would be super hard
In any case your best friends are Hopper dissassembler and MachOView to inspect the binary.
Elementary knowledge of x86 (and/or x86-64) assembly is a must to follow. I think playing with the original source code could be a way more viable option.
As far as I understand(Correct me if i'm wrong), the output of a compiler depends on the Architecture version used, Compiler, and operating system.
Lets say Im using ubuntu release 16.04 x84-64 and compiling a c file with gcc version 5.4(or any other mix of OS,arch,compiler for the example) .
As I understood it until now, if I were to compile the same c file but with a different ubuntu release, with the same arch and compiler version it should have produced the same assembly code.
After a few tries I have got the impression that this is incorrect, how is this possible?
Does the output of a compiler depend on the release of the specific OS?
One of the examples is compiling https://github.com/tbuktu/libntru on 2 different ubuntu versions and receiving different assembly.
The different OS's may have different versions of the default libraries installed (which get linked into your final application). Thus the end result may be slightly different.
If you are just doing a few ubuntu versions the odds of differences goes down as the overall architecture differences may not be reflected either in your test or may not change on the same os family with the same compiler family for long periods of time. Where you are more likely to see differences in a test like that is as you get older versions of the same distro, newer/newest versions of the compiler are not ported/supported directly as an apt-get. maybe you can get them to work by hand building but gcc in particular is really bad about that their code only builds with relatively recent prior or following versions get too far apart and gcc cant build gcc. What I would first expect to see is strictly due to gcc version differences you start to see differences in the compiler.
A better test is take a simple .c file and build for windows any version (using the same version of gcc built for that system) and ubuntu/linux any version. Should more quickly see differences.
Two different compilers should show differences for reasonably sized projects, or knowledge based targeted small code samples, llvm/clang vs gcc for example. Different versions of the same compiler or compiler family will somewhat by definition show differences over time, does 6.x vs 6.x+1 gcc show differences well yes if you know where to look but often not, but gcc 3.x vs gcc 7.x should and then depending on the test you can narrow in from there.
You have compiler to compiler differences on the same os and system that are expected to show differences.
You have various reasons why system to system differences with the same compiler will show differences.
And then combinations of the above would naturally also show differences.
The bigger question is why do you care, the educational information is that you shouldn't expect the same C source code to build the same way if you change the compiler, compiler settings, or operating system. It can have anywhere from no differences to huge differences based on any of the above. Starting quite simply with optimization and other tuning settings and going from there.
This question was emerged from this question.
The problem is that there is a NVidia driver for Linux, compiled wth GCC 4.5. The kernel is compiled with GCC 4.6. Well, the stuff doesn't work because of the version number difference between GCCs. (the installer says the driver won't work - for details please visit the link above)
Could one disguise a binary compiled with GCC 4.5 to a binary compiled with GCC 4.6? If it is possible, under what circumstances would it work well?
Your problem is called ABI: Application Binary Interface. This is a set of rules (among others) how functions in a piece of code get their arguments (ordering, padding of types on the stack), naming of the function so the linker can resolve symbols and padding/alignment of fields in structures.
GCC tries to keep the ABI stable between compiler versions but that's not always possible.
For example, GCC 4.4 fixed a bug in packed bit-fields which means that old/new code can't read structures using this feature properly anymore. If you would mix versions before and after 4.4, data corruption would occur without any crashes.
There is no indication in the 4.6 release notes that the ABI was changed but that's something which the Linux kernel can't know - it just reads the compiler version used to compile the code and if the first two numbers change, it assumes that running the code isn't safe.
There are two solutions:
You can compile the Nvidia driver with the same compiler as the kernel. This is strongly recommended
You can patch the version string in the binary. This will trick the kernel into loading the module but at the risk of causing data corruption to internal data structures.
Reference this question about compiling. I don't understand how my program for Mac can use the right -arch, compile with those -arch flags, the -arch flags be for the system I am on (a ppc64 g5), and still produce the wrong object code.
Also, if I used a cross compiler and was on Linux, produced 10.5 code for mac, how would this be any different than what I described above?
Background is that I have tried to compile various apache modules. They compile with the -arch ppc, ppc64, etc. I get no errors and I get my mod_whatever.so. But, apache will always complain that some symbol isn't found. Apparently, it has to do with what the compiler produces, even though the file type says it is for ppc, ppc64, i386, x_64 (universal binary) and seems to match all the other .so mods I have.
I guess I don't understand how it could compile for my system with no problem and then say my system can't use it. Maybe I do not understand what a compiler is actually giving me.
EDIT: All error messages and the complete process can be seen here.
Thank you.
Looking at the other thread and elsewhere and without a G5 or OSX Server installation, I can only make a few comments and suggestions but perhaps they will help.
It's generally not a good idea to be modifying the o/s vendor's installed software. Installing a new Apache module is less problematic than, say, overwriting an existing library but you're still at the mercy of the vendor in that a Software Update could delete your modifications and, beyond that you have to figure out how the vendor's version was built in the first place. A common practice in the OS X world is to avoid this by making a completely separate installation of an open source product, like Apache, using, for instance, MacPorts. That has its cons, too: to achieve a high-level of independence, MacPorts will often download and build a lot of dependent packages for things which are already in OS X but there's no harm in that other than some extra build cycles and disk space.
That said, it should be possible to build and install apache modules to supplement those supplied by Apple. Apple does publish the changes it makes to open source products here; you can drill down in the various versions there to find the apache directory which contains the source, Makefile and applied patches. That might be of help.
Make sure that the mod_*.so you build are truly 64-bit and don't depend on any non-64 bit libraries. Use otool -L mod_*.so to see the dynamic libraries that each references and then use file on those libraries to ensure they all have ppc64 variants.
Make sure you are using up-to-date developer tools (Xcode 3.1.3 is current).
While the developer tool chain uses many open source components, Apple has enhanced many of them and there are big differences in OS X's ABIs, universal binary support, dynamic libraries, etc. The bottom line is that cross-compilation of OS X-targeted object code on Linux (or any other non-OS X platform) is neither supported nor practical.
Been using Lua 5.0 in a Mac OS X universal binary app for some years. Lua scripts are compiled using luac and the compiled scripts are bundled with the app. They have worked properly in Tiger and Leopard, Intel or PPC.
To avoid library problems at the time, I simply added the Lua src tree to my Xcode project and compiled as is, with no problems.
It was time to update to a more modern version of Lua so I replaced my source tree with that of 5.1.4. I rebuilt luac using make macosx (machine is running Leopard on Intel).
Uncompiled scripts work properly in Tiger and Leopard, Intel and PPC, as always.
However, now compiled scripts fail to load on PPC machines.
So I rebuilt luac with the 'ansi' flag, and recompiled my scripts. Same error. Similarly, a build flag of 'generic' produced no joy.
Can anyone please advise on what I can do next?
Lua's compiled scripts are pretty much the raw bytecode dumped out after a short header. The header documents some of the properties of the platform used to compile the bytecode, but the loader only verifies that the current platform has the same properties.
Unfortunately, this creates problems when loading bytecode compiled on another platform, even if compiled by the very same version of Lua. Of course, scripts compiled by different versions of Lua cannot be expected to work, and since the version number of Lua is included in the bytecode header, the attempt to load them is caught by the core.
The simple answer is to just not compile scripts. If Lua compiles the script itself, you only have to worry about possible version mismatches between Lua cores in your various builds of your application, and that isn't hard to deal with.
Actually supporting a full cross compatibility for compiled bytecode is not easy. In that email, Mike Pall identified the following issues:
Endianess: swap on output as needed.
sizeof(size_t), affects huge string constants: check for overflow when
downgrading.
sizeof(int), affectsMAXARG_Bx and MAXARG_sBx: check for overflow when
downgrading.
typeof(lua_Number): easy in C, but only when the host and the target
follow the same FP standard; precision
loss when upgrading (rare case);
warn about non-integer numbers when
downgrading to int32.
From all the discussions that I've seen about this issue on the mailing list, I see two likely viable approaches, assuming that you are unwilling to consider just shipping the uncompiled Lua scripts.
The first would be to fix the byte order as the compiled scripts are loaded. That turns out to be easier to do than you'd expect, as it can be done by replacing the low-level function that reads the script file without recompiling the core itself. In fact, it can even be done in pure Lua, by supplying your own chunk reader function to lua_load(). This should work as long as the only compatibility issue over your platforms is byte order.
The second is to patch the core itself to use a common representation for compiled scripts on all platforms. This has been described as possible by Luiz Henrique de Figueiredo:
....
I'm convinced that the best route to
byte order or cross-compiling is
third-party dump/undump pairs. The
files ldump.c and lundump.c are
completely replaceable; they export a
single, well-defined, entry point. The
format of precompiled chunks is not
sacred at all; you can use any format,
as long as ldump.c and lundump.c agree
about it. (For instance, Rici Lake is
considering writing a text format for
precompiled chunks.)
....
Personally, I'd recommend giving serious consideration to not pre-compiling the scripts and thus avoid the platform portability issues entirely.
Edit: I've updated my description of the bytecode header thanks to lhf's comment. I hadn't read this part of the Lua source yet, and I probably should have checked it before being quite so assertive about what information is or is not present in the header.
Here is the fragment from lundump.c that forms a copy of the header matching the running platform for comparison to the bytecode being loaded. It is simply compared with memcmp() for an exact match to the header from the file, so any mismatch will cause the stock loader (luaU_undump()) to reject the file.
/*
* make header
*/
void luaU_header (char* h)
{
int x=1;
memcpy(h,LUA_SIGNATURE,sizeof(LUA_SIGNATURE)-1);
h+=sizeof(LUA_SIGNATURE)-1;
*h++=(char)LUAC_VERSION;
*h++=(char)LUAC_FORMAT;
*h++=(char)*(char*)&x; /* endianness */
*h++=(char)sizeof(int);
*h++=(char)sizeof(size_t);
*h++=(char)sizeof(Instruction);
*h++=(char)sizeof(lua_Number);
*h++=(char)(((lua_Number)0.5)==0); /* is lua_Number integral? */
}
As can be seen, the header is 12 bytes long and contains a signature (4 bytes, "<esc>Lua"), version and format codes, a flag byte for endianness, sizes of the types int, size_t, Instruction, and lua_Number, and a flag indicating whether lua_Number is an integral type.
This allows most platform distinctions to be caught, but doesn't attempt to catch every way in which platforms can differ.
I still stand by the recommendations made above: first, ship compilable sources; or second, customize ldump.c and lundump.c to store and load a common format, with the additional note that any custom format should redefine the LUAC_FORMAT byte of the header so as to not be confused with the stock bytecode format.
You may want to use a patched bytecode loader that supports different endianness.
See this.
I would have commented on RBerteig's post, but I apparently don't have enough reputation yet to be able to do so. In working on bringing LuaRPC up to speed with Lua 5.1.x AND making it work with embedded targets, I've been modifying the ldump.c and lundump.c sources to make them both a bit more flexible. The embedded Lua project (eLua) already had some of the patches you can find on the Lua list, but I've added a bit more to make lundump a little more friendly to scripts compiled on different architectures. There's also cross-compilation support provided so that you can build for targets differing from the host system (see luac.c in the same directory as the links below).
If you're interested in checking out the modifications, you can find them in the eLua source repository:
http://svn.berlios.de/wsvn/elua/trunk/src/lua/lundump.c
http://svn.berlios.de/wsvn/elua/trunk/src/lua/lundump.h
http://svn.berlios.de/wsvn/elua/trunk/src/lua/ldump.c
Standard Disclaimer:
I make no claim that the modifications are perfect or work in every situation. If you use it and find anything broken, I'd be glad to hear about it so that it can be fixed.
Lua bytecode is not portable. You should ship source scripts with your application.
If download size is a concern, they are generally shorter than the bytecode form.
If intellectual property is a concern, you can use a code obfuscator, and keep in mind that disassembling Lua bytecode is anything but difficult.
If loading time is a concern, you can precompile the sources locally in your installation script.
I conjecture that you compiled the scripts on an Intel box.
Compiled scripts are wildly unportable. If you really want to precompile scripts, you'll need to include two versions of each compiled script: one for Intel and one for PPC. Your app will have to interrogate which program it's running on and use the correct compiled script.
I don't have enough reputation to comment, so I have to provide this as an answer instead even though it's not an appropriate answer to the question asked. Sorry.
There is an Lua Obfuscator available here:
http://www.capprime.com/CapprimeLuaObfuscator/CapprimeLuaObfuscator.aspx
Full disclosure: I am the author of the obfuscator and I am aware it is not perfect. Feedback is welcome and encouraged (there is a feedback page available from the above page).