All texts on how to create a compiler stop after explaining lexers and parsers. They don't explain how to create the machine code. I want to understand the end-to-end process.
Currently what I understand is that, the Windows exe file formats are called Portable Executable. I read about the headers it has and am yet to find a resource which explains this easily.
My next issue is, I don't see any resource which explains how machine code is stored in the file. Is it like 32-bit fixed length instructions stored one after another in the .text section?
Is there any place which at least explains how to create an exe file which does nothing (it has a No Op instruction). My next step then would be linking to dll files to print to console.
Nice question! I don't have much expertise on this specific question, but this is how I would start:
PE or ELF does not create pure machine code. It also contains some header info etc. Read more: Writing custom data to executable files in Windows and Linux
I assume you are looking for how does ELF/PE file hold the machine code, you can get that from this question (using objdump): How do you extract only contents of an ELF section
Now, if you want to know how the content part is generated in the first place, i.e. how is the machine code generated, then that's the task of the compiler's code generation.
Try out some resource editor like ResourceEditor to understand the exe or simply ildasm.
PS: These are mostly Unix solutions, but I am sure, PE should be doing something fundamentally similar.
I think the best way to approach it will be first try to analyze how existing PE/ELFs work, basically reverse engineering. And to do that, Unix machine will be a good point to start. And then do your magic :)
Not same but a similar question here.
Update:
I generated an object dump out of a sample c code. Now, I assume that's what you are targeting right? You need to know do you generate this file (a.out)?
https://gist.github.com/1329947
Take a look at this image, a life time of a c code.
Source
Now, just to be clear, you are looking to implement the final step, i.e. conversion of object code to executable code?
As in many of his articles, I'd say Matt Pietrek's piece about PE internals remains the best introdction to the matter more than a decade after being written.
Iv'e used "Wotsit's File Format" for years... all the way back to the days of MS-Dos :-) and back to when it was just a collection of text files you could download from most BBS systems called "The Game programmers file type encyclopaedia"
It's now owned by the people that run Gamedev.Net, and probably one of the best kept secrets on the internet.
You'll find the EXE format on this page : http://www.wotsit.org/list.asp?fc=5
Enjoy.
UPDATE June 2020 - The link above seems to be now dead, I've found the "EXE" page listed on this web archive page of the wotsit site: https://web.archive.org/web/20121019145432/http://www.wotsit.org/list.asp?al=E
UPDATE 2 - I'm keeping the edit as it was when I added the update erlier, thanks to those who wanted to edit it, but it's for a good reason I'm rejecting it:
1) Wotsit.org may at some point in the future come back online, if you actually try visiting the url, you'll find that it's not gone, it does still respond, it just responds with an error message. This tells me that someone is keeping the domain alive for whatever reason.
2) The archive links do seem to be a bit jittery, some work, some don't, sometimes they seem to work, then after a refresh they don't work, then they do work again. I remember from experience when wotsit was still online, they they had some very strange download/linking detection code in, and this probably caused archive.org to get some very wierd results, I do remember them taking this stance because of the huge number of 3rd party sites trying to cash in on their success, by pretending to be affiliate's and then direct linking to wotsit from an ad infested site.
Until the wotsit domain is removed entirely from the internet and not even the DNS responds, then would be the time to wrap everything up into single archive links, until then, this is the best way to maintain the link.
Not surprisingly the best sites for information about writing PE format files are all about creating viruses.
A search of VX Heavens for "PE" gives a whole bunch of tutorials for modifying PE files
Some information about making PE files as small as possible: Tiny PE.
The minimalistic way to mess around with code generation, if you're just looking to try a few simple things out, is to output MS-DOS .COM files, which have no header or metadata. Sadly, you'd be restricted to 16-bit code. This format is still somewhat popular for demos.
As for the instruction format, from what I recall the x86 instruction set is variable-length, including 1-byte instructions. RISC CPUs would probably have fixed-length instructions.
For Linux, one may read and run the examples from
"Programming from the Ground Up" by Jonathan Bartlett:
http://www.cs.princeton.edu/courses/archive/spr08/cos217/reading/ProgrammingGroundUp-1-0-lettersize.pdf
Then of course one may prefer to hack Windows programs. But perhaps the former
gives a better way to understand what really goes on.
Executable file format is dependent on the OS. For windows it is PE32(32 bit) or PE32+(64 bit).
The way the final executable look like depends on the ABI (application binary interface) of the OS. The ABI tells how the OS loader should load the exe and how it should relocate it, whether it is dll or plain executable etc..
Every object file(executable or dll or driver) contains a part called sections. This is where all of our code, data, jump tables etc.. are situated.
Now, to create an object file, which is what a compiler does, you should not just create the executable machine code, but also the headers, symbol table, relocation records, import/export tables etc..
The pure machine code generation part is completely dependent on how much optimized you want your code to be. But to actually run the code in the PC, you must have to create a file with all of the headers and related data(check MSDN for precise PE32+ format) and then put all of the executable machine code(which your compiler generated) into one of the sections(usually code resides in section called .text). If you have created the file conforming to the PE32+ format, then you have now successfully created an executable in windows.
Related
So I compiled winipt and ptdump from libipt I got a trace but ptdump prints garbage. Additionally I created a dump using MiniDumpWriteDump with MiniDumpWithIptTrace and opened it both with VS and windbg but I don't think they are recognizing the trace either.
I'm currently downloading intel studio since it has widnbg extensions for ipt support but I've heard also of REPT.
In the video it's showing some futuristic looking windbg and it's setting a hw breakpoint which then triggers on backwards execution.
You are almost there, but partly you are doing it wrong and Alex's code is not so complete.
Note that Alex's code is a POC as he says, IMHO it's not really complete to go right to a a file and decoder. Also note that ptdump is just an example. If you can find a sample trace that is just pure IPT packets then it will work as is.
Step through how ipttool outputs the data (for one thing it saves a header that you are not accounting for), vs how ptdump loads a binary.
winipt is used in this project:
https://github.com/googleprojectzero/winafl
Unfortunately as we both found there is no official support for Intel PT on Windows yet like there is on Linux which has complete support for it.
Yea apparently they added support to MiniDumpWriteDump(). You can see the support for it by symbols in dbgcore.dll. Could not find any documentation for it either. On MSDN they define the IPT type enums but then there is zero descriptions of what they do.
Parsing a normal/old mini-dump is probably a project in it's self. I bet though if you can parse a normal/default mini-dump, you will figure out how to parse the IPT section out of it too.
Anyhow depending on what you want to do, probably just a a buffer full and it want's to write to a file.
Update:
Actually it looks like PssNtCaptureSnapshot() would give us direct access to a IPT snapshot using the PSS_CAPTURE_IPT_TRACE flag, then followed by a PssQuerySnapshot() function to get the data. Still one will have to parse it considering the headers like struct IPT_TRACE_DATA.
https://learn.microsoft.com/en-us/windows/win32/api/processsnapshot/nf-processsnapshot-psscapturesnapshot
I am in a process of learning things in reverse order for fun, and I have decided to dissect Windows 10, bit-by-bit, and learn what makes a great OS function. And I also suppose that my question will be geared in other ways as well.
My question is, how do I look at something like Windows bootmgr source code properly? I have opened the file - which the file type is redundantly called "File" - and even though it is in Assembly language, it is completely impossible to read. My guess is that whoever wrote the File did something to encrypt the File so that it is unreadable, and thus unchangeable/unable to be edited.
Let me be perfectly clear: my purpose is not to change the bootmgr File to change windows, but rather to get a better understanding of how an OS works via reading, and also through trial and error.
Any help that anyone can give would be greatly appreciated. I love to learn about these things, and I just have been completely unable to find the answer I am looking for on any site thus far, including this one...IDK if I need to refine my searches or what.
Thank in advanced for your help. :)
Ps. I shall include a picture of what I am seeing in Notepad++ so you can get a better understanding of what I need here .
I think you may be confusing assembly language with machine code. Machine code is the language that your computer's processor understands. Assembly language is a series of symbols that are used to represent machine code. Compiled executables are stored in machine code.
That said, the standard way to view the machine code for a compiled binary is through the use of a program called a hex editor. A hex editor will display the binary code in a numerical format, rather than attempting to interpret the binary as text, like your editor is trying to do in the screenshot you supplied. Frhed is a popular hex editor, but there are many good ones to choose from.
I want to do these two things with my application (Windows only):
Allow user to insert (with a tool) a native code into my application before starting it.
Run this user-inserted code straight from memory during runtime.
Ideally, it would have to be easy for user to specify this code.
I have two ideas how to do this that I'm considering right now:
User would embed a native dll into application's resources. Application would load this dll straight from memory using techniques from this article.
Somehow copy assembly code of .dll method specified by user into my application resources, and execute this code from heap as described in this article.
Are there any better options to do this? If not, any thoughts on what might cause problems in those solutions?
EDIT
I specifically do not want to use LoadLibrary* calls as they require dll file to be already on hard drive which I'm trying to avoid. I'm also trying to make dissasembling harder.
EDIT
Some more details:
Application code is under my control and is native. I simply want to provide user with a way to embed his own customized functions after my application is compiled and deployed.
User code can have arbitrary restrictions placed on it by me, it is not a problem.
The aim is to allow third parties to statically link code into a native application.
The obvious way to do this is to supply the third parties with the application's object files and a linker. This could be wrapped up in a tool to make it easy to use.
As ever, the devil is in the detail. In addition to object files, applications contain manifests, resources, etc. You need to find a linker that you are entitled to distribute. You need to use a compiler that is compatible with said linker. And so on. But this is certainly feasible, and likely to be more reliable than trying to roll your own solution.
Your option 2 is pretty much intractable in my view. For small amounts of self-contained code it's viable. For any serious amount of code you cannot realistically hope for success without re-inventing the wheel that is your option 1.
For example, real code is going to link to Win32 functions, and how are you going to resolve those? You'd have to invent something just like a PE import table. So, why do so when DLLs already exist. If you invented your own PE-like file format for this code, how would anyone generate it? All the standard tools are in the business of making PE format DLLs.
As for option 1, loading a DLL from memory is not supported. So you have to do all the work that the loader would do for you if it were loading from file. So, if you want to load a DLL that is not present on the disk, then option 1 is your only choice.
Any half competent hacker will readily pull the DLL from the executing process though so don't kid yourself that running DLLs from memory will somehow protect your code from inspection.
This is something called "application virtualization", there are 3rd party tools for that, check them on google.
In a simple case, you may just load "DLL" into memory, apply relocs, setup imports and call entry point.
I have a very old application which I bought about 15-years ago, it consists of 5 .exe files used for storing patients' profiles and information. The problem is that this application was programmed to work on a specific computer.
And because I have little knowledge about cracking, I tried to disassemble it using win32dasm but there was no information about string references, tried Hiew to replace JE by JNE and after many, many tries one of the executables worked, but I failed to patch the other files.
Is there any way I can find the exact jump and patch it?
thanks ;)
Yes, there is a way, there's always a way. If you can run the code, you can get at it to modify it.
However, if it was licensed to work on a specific computer, you may not actually have the legal right to use it on another computer. The first thing you should do is figure out is the legality of what you're trying.
Having said that, I've used OllyDbg in the past to do this sort of work. It wasn't nefarious, it's just that the code failed on machines with lots of memory - turns out it used a signed comparison instruction rather than an unsigned one.
The basic idea would be to record the string output when you run it on a failing machine, then locate that string in memory and watch for any piece of code referencing it. You should then be able to backtrack from there to find the conditional jump which brought you there and patch it so that it doesn't (eg, replace the entire jCC instruction with nop bytes.
But, and I stress this, it will require some investigation, this isn't something you can do just by pressing a button and letting the computer work it out. You may well have to dig deep into the assembly to understand how it's working.
What would be the best way of inserting functionality into a binary application (3d party, closed source).
The target application is on OSX and seems to have been compiled using gcc 3+. I can see the listing of functions implemented in the binary and have debugged and isolated one particular function which I would like to remotely call.
Specifically, I would like to call this function - let's call it void zoomByFactor(x,y) - when I receive certain data from a complex HIDevice.
I can easily modify or inject instructions into the binary file itself (ie. the patching does not need to occur only in RAM).
What would you recommend as a way of "nicely" doing this?
Edit:
I do indeed need to entire application. So I can't ditch it and use a library. (For those who need an ethical explanation: this is a proprietary piece of CAD software whose company website hasn't been updated since 2006. I have paid for this product (quite a lot of money for what it is, really) and have project data which I can not easily migrate away from it. The product suits me just fine as it is, but I want to use a new HID which I recently got. I've examined the internals of the application, and I'm fairly confident that I can call the correct function with the relevant data and get it to work properly).
Here's what I've done so far, and it is quite gheto.
I've already modified parts of the application through this process:
xxd -g 0 binary > binary.hex
cat binary.hex | awk 'substitute work' > modified.hex
xxd -r modified.hex > newbinary
chmod 777 newbinary
I'm doing this kind of jumping through hoops because the binary is almost 100 megs large.
The jist of what I'm thinking is that I'd jmp somewhere in the main application loop, launch a thread, and return to the main function.
Now, the questions are: where can I insert the new code? do I need to modify symbol tables? alternatively, how could I make a dylib load automatically so that the only "hacking" I need to do is inserting a call to a normally loaded dylib into the main function?
For those interested in what I've ended up doing, here's a summary:
I've looked at several possibilities. They fall into runtime patching, and static binary file patching.
As far as file patching is concerned, I essentially tried two approaches:
modifying the assembly in the code
segments (__TEXT) of the binary.
modifying the load commands in the
mach header.
The first method requires there to be free space, or methods you can overwrite. It also suffers from extremely poor maintainability. Any new binaries will require hand patching them once again, especially if their source code has even slightly changed.
The second method was to try and add a LC_ LOAD_ DYLIB entry into the mach header. There aren't many mach-o editors out there, so it's hairy, but I actually modified the structures so that my entry was visible by otool -l. However, this didn't actually work as there was a dyld: bad external relocation length at runtime. I'm assuming I need to muck around with import tables etc. And this is way too much effort to get right without an editor.
Second path was to inject code at runtime. There isn't much out there to do this. Even for apps you have control over (ie. a child application you launch). Maybe there's a way to fork() and get the initialization process launched, but I never go that.
There is SIMBL, but this requires your app to be Cocoa because SIMBL will pose as a system wide InputManager and selectively load bundles. I dismissed this because my app was not Cocoa, and besides, I dislike system wide stuff.
Next up was mach_ inject and the mach_star project. There is also a newer project called
PlugSuit hosted at google which seems to be nothing more than a thin wrapper around mach_inject.
Mach_inject provides an API to do what the name implies. I did find a problem in the code though. On 10.5.4, the mmap method in the mach_inject.c file requires there to be a MAP_ SHARED or'd with the MAP_READ or else the mmap will fail.
Aside from that, the whole thing actually works as advertised. I ended up using mach_ inject_ bundle to do what I had intended to do with the static addition of a DYLIB to the mach header: namely launching a new thread on module init that does its dirty business.
Anyways, I've made this a wiki. Feel free to add, correct or update information. There's practically no information available on this kind of work on OSX. The more info, the better.
In MacOS X releases prior to 10.5 you'd do this using an Input Manager extension. Input Manager was intended to handle things like input for non-roman languages, where the extension could popup a window to input the appropriate glyphs and then pass the completed text to the app. The application only needed to make sure it was Unicode-clean, and didn't have to worry about the exact details of every language and region.
Input Manager was wildly abused to patch all sorts of unrelated functionality into applications, and often destabilized the app. It was also becoming an attack vector for trojans, such as "Oompa-Loompa". MacOS 10.5 tightens restrictions on Input Managers: it won't run them in a process owned by root or wheel, nor in a process which has modified its uid. Most significantly, 10.5 won't load an Input Manager into a 64 bit process and has indicated that even 32 bit use is unsupported and will be removed in a future release.
So if you can live with the restrictions, an Input Manager can do what you want. Future MacOS releases will almost certainly introduce another (safer, more limited) way to do this, as the functionality really is needed for language input support.
I believe you could also use the DYLD_INSERT_LIBRARIES method.
This post is also related to what you were trying to do;
I recently took a stab at injection/overriding using the mach_star sources. I ended up writing a tutorial for it since documentation for this stuff is always so sketchy and often out of date.
http://soundly.me/osx-injection-override-tutorial-hello-world/
Interesting problem. If I understand you correctly, you'd like to add the ability to remotely call functions in a running executable.
If you don't really need the whole application, you might be able to strip out the main function and turn it into a library file that you can link against. It'll be up to you to figure out how to make sure all the required initialization occurs.
Another approach could be to act like a virus. Inject a function that handles the remote calls, probably in another thread. You'll need to launch this thread by injecting some code into the main function, or wherever else is appropriate. Most likely you'll run into major issues with initialization, thread safety, and/or maintaining proper program state.
The best option, if its available, is to get the vendor of your application to expose a plugin API that lets you do this cleanly and reliably in a supported manner.
If you go with either hack-the-binary route, it'll be time consuming and brittle, but you'll learn a lot in the process.
On Windows, this is simple to do, is actually very widely done and is known as DLL/code injection.
There is a commercial SDK for OSX which allows doing this: Application Enhancer (free for non-commercial use).