I've been trying to learn about the PE file structure, and I compiled the following short program to examine the output:
#include <stdio.h>
int get_5() {
return 5;
}
int main() {
int five = get_5();
fputc(five, stdout);
return five;
}
This uses the fputc function, which needs to be linked in from a .dll file. I compiled this in MSYS using gcc -o program.exe program.c.
When I look in the resulting executable's .idata section, it shows that it imports fputc from msvcrt.dll. It includes a name hint that I think is used to find where to link it within the .dll.
My question, however, is how does Windows then use this information to find where in the .dll to take the function? When I look inside msvcrt.dll, it does not contain a .edata section, which I thought is where information for exporting symbols is located. If there were an export table, I would assume the program loader looks through the ordinal and/or name pointer tables in the export section to get a pointer to the requested function. So, without an export table, how do the hint and name help to find the location of the function?
I think I found an explanation. While the .dll's do not contain a .edata section, they contain an image data directory entry for an export table in their optional header, which points to data in the .rdata section that is in the same format as a .edata section would have. Though I am still unclear on when a file would have an actual .edata section and would it would just have export data in .rdata. I imagine the same is true for imports, as well.
Related
Got a new issue I've not come across before that's appeared when using the Espressif ESP32 ESP-IDF standard setup under VSCode. It uses the GNU compiler.
I'm getting "multiple definition of" errors on variables that share the same name, but which should be local.
So I use a .c and .h pair of files approach.
In my .c files I do this at the top
#define IO_EXPANDER_C //<<<This is a unique define for this file pair
#include "io-pca9539.h"
In my .h files I do this:
#ifdef IO_EXPANDER_C
//----- INTERNAL ONLY MEMORY DEFINITIONS -----
uint8_t *NextReadDataPointer;
//----- INTERNAL & EXTERNAL MEMORY DEFINITIONS -----
//(Also defined below as extern)
int SomeVariableIWantAvailableGlobally;
#else
//----- EXTERNAL MEMORY DEFINITIONS -----
extern int SomeVariableIWantAvailableGlobally;
#endif
It's a great simple system, any other .c file that includes the .h file (without the #define above its include statemnt), gets all of its extern variables, none of its local variables.
But, compiling in VSCode with my ESP-IDF based project, I'm getting "multiple definition of" errors relating to "NextReadDataPointer"
I use the same variable name NextReadDataPointer in another file pair in just the same way, but it's never declared anywhere as extern and each file pair uses a separate #define (IO_EXPANDER_C and LED_C). I do this all the time normally and I can't see any obvious mistakes.
I've never seen a C compiler do this before, it's as if it's mixing up the local definitions somehow. A #define should only have scope in the file it is declared in and in any includes within that file.
Even odder, the error is not generated if the project is built but a function is called from just one of the file pairs that share the same local variable name. It's only generated when functions are called from both file pairs from my main application.
Can anyone shed light on whether the GNU C compiler does something funky for a standard ESP-IDF project as it's got me baffled?
uint8_t *NextReadDataPointer; creates a variable which is visible across all translation units, i.e. it's the opposite of "private". If you include this header in multiple c files and the linker tries to link those together; it'll see a conflict. The keyword you're looking for is static, for example static uint8_t *NextReadDataPointer; creates a variable that is not visible across translation units. The reason you don't see the problem if calling a function from only one of those two files is because in this case the linker doesn't bother looking into the other one.
Personally I'd avoid such clever preprocessor hacks because it's quite difficult to see how files include one another and debug the resulting problems. I'd suggest sticking to the standard way of declaring shared things in header files and keeping the private stuff inside the c file (prepended by static).
I'm using NVRTC to compile a kernel. The relevant API call is:
nvrtcResult nvrtcCreateProgram (
nvrtcProgram* prog,
const char* src,
const char* name,
int numHeaders,
const char** headers,
const char** includeNames )
As you can see, the source is a raw string, and not associated with a file. That means that when you --generate-line-info, you get line numbers, but no related filename. And that means that if you then use, say, NSight Compute - you won't be able to see your kernel source code.
Obviously, neither NSight Compute itself, nor NVRTC itself, can figure out that the raw source is mirrored in some file. But there has to be some way to get around this:
Perhaps I'm missing something in the NVRTC API which can make the source <-> file association?
Perhaps we can manipulate the resulting compiled program (reasonably, not manually, or write-my-own-new-API) to make the association?
Perhaps we can shove the source code into the compiled program somehow?
Here's my initial workaround:
Place your source in a file, say my_kernel.cuh.
Create the string:
#include "my_kernel.cuh"
Compile just this string using NVRTC
Now, NVRTC is able to associate included files' sources with the files, so it's only a stub that will be missing in terms of source<->file association.
Caveat: You will need to be careful about paths - NVRTC's include paths, the working directory from which you invoke your program vs the directory of the source file etc.
It seems NVRTC does provides a default filename, such that if you place your source in the file with that name - NSight Compute may be able to find it.
The name is the one you passed to nvrtcCreateProgram() as the name argument.
So, if your kernel function (i.e. your __global__ function) is in my_kernel.cuh, and you place this file in the working directory of the profile program (which you tell NSight Compute about), or in one of the include directories you built your program with, you'll be able to read your source. If the original file's own directory is also one of the include directories, then you're in luck and you don't even have to make a copy.
I have defined a DLL-export as follows:
__declspec(dllexport)
DWORD WINAPI DllBootstrap(LPVOID addr) {
return 0;
}
Now, using DUMPBIN, the symbol is displayed as follows:
1 0 0001100A ?DllBootstrap##YGKPAX#Z = #ILT+5(?DllBootstrap##YGKPAX#Z)
And this is how the memory looks in Visual Studio:
ยก}....ReflectDLL.dll.?DllBootstrap##YGKPAX#Z..........................................
when inspecting PIMAGE_EXPORT_DIRECTORY.AddressOfNames.
What I need is a clean symbol, i.e., DUMPBIN should output something like:
1 0 0001100A DllBootstrap
and PIMAGE_EXPORT_DIRECTORY.AddressOfNames should point to:
DllBootstrap..........................................
How can I achieve this?
WIN32 BUILDS:
As #RbMm indicated, to retain your function name as-is and get no name decoration, you must use a .DEF file (and remove the __declspec(dllexport) specifier). Then create a DEF file with the line below and either specify it with the /DEF linker option or add it to your Visual Studio project and it will be picked up automatically by the linker:
EXPORTS DllBootstrap
If you don't want to deal with an external .DEF file and you will be using the Visual C++ compiler, the simplest way to limit decoration using just code is to declare your function with 'extern "C"'. This results in decoration including a preceding underscore and appends an "#" along with the argument's byte count in decimal. The following code for example:
extern "C" __declspec(dllexport)
DWORD WINAPI DllBootstrap(LPVOID addr) {
return 0;
}
produces an exported name of:
_DllBootstrap#4
This is how stdcall functions are decorated when C++ name-mangling is disabled with 'extern "C"'. NOTE: WINAPI maps to __stdcall. Retaining 'extern "C"' and changing the convention to __cdecl, you won't get any decoration whatsoever, but module entrypoints should generally remain stdcall as you have it listed in your sample.
If you still want to avoid a .DEF file, there is one last hack you can employ. Add the following line to your code:
#pragma comment(linker,"/EXPORT:DllBootstrap=_DllBootstrap#4")
This will pass an argument to the linker creating a new undecorated name symbol which maps to the decorated name. This isn't very clean as the original name will still exist in your DLL, but you will get your clean exported name.
WIN64 BUILDS (UPDATE):
As Hans Passant commented, for anyone using the Visual C++ 64-bit compiler, there is only the 64-bit calling convention (stdcall, cdecl, etc. keywords are ignored). While C++ mangling will still occur under this compiler, no additional decoration is made to the exported names. In this case, 'extern "C"' would be enough when the sample is compiled as C++ code; if compiled as C, no modifications would be necessary.
Sorry if the title is not very clear. I am using MinGW with GCC 6.3.0 to build a x86 32-bit DLL on Windows (so far). I'll spare you the details why I need hacky offsets amongst its sections accessible from code, so please do not ask if it's useful or not (because I don't want to bother explaining that).
So, if I can get the following testcase to work, I'm good. Here's my problem:
In a C++ file, I want to access a linker symbol as an absolute numeric value, not relocated, directly. Remember that I am building a 32-bit DLL which requires a .reloc section for relocations, but in this case I do NOT want relocation, in fact a relocation would screw it up completely.
Here's an example: retrieve the offset of say __imp__MessageBoxW#16 relative to __IAT_start__, in case you don't know what they are, __imp__MessageBoxW#16 is the relocated pointer to the actual function at runtime, and __IAT_start__ is a linker symbol in the default script file. Here's where it is defined:
.idata BLOCK(__section_alignment__) :
{
/* This cannot currently be handled with grouped sections.
See pe.em:sort_sections. */
KEEP (SORT(*)(.idata$2))
KEEP (SORT(*)(.idata$3))
/* These zeroes mark the end of the import list. */
LONG (0); LONG (0); LONG (0); LONG (0); LONG (0);
KEEP (SORT(*)(.idata$4))
__IAT_start__ = .;
KEEP (SORT(*)(.idata$5))
__IAT_end__ = .;
KEEP (SORT(*)(.idata$6))
KEEP (SORT(*)(.idata$7))
}
So far, no problem. Because GAS doesn't allow me to "subtract" two externally defined symbols (both symbols are defined in the linker), I have to define the symbol in the linker script, so at the end of the linker script I have this:
test_symbol = ABSOLUTE("__imp__MessageBoxW#16" - __IAT_start__);
Then in C++ I use this little inline asm to retrieve this relative difference which is supposed to be a fixed value once linked:
asm("movl $test_symbol, %0":"=r"(var));
Now var should contain that fixed number right? Wrong!
Because test_symbol is an "undefined" symbol as far as the assembler is concerned, it makes it relocated. Or I don't know why, but I tried so many things to force it to be an "absolute constant value symbol" instead of a "relocated symbol" to no avail. Even editing the linker script with many things like LD_FEATURE("SANE_EXPR") and others, doesn't work at all.
Its value is correct only if the DLL does not get relocated.
You see, either GNU LD or the assembler adds an entry in the .reloc section for that movl instruction, which is WRONG!
Is there a way to force it to treat an external/undefined symbol as a fixed CONSTANT and apply no relocation to it whatsoever? Basically, omit it from the .reloc section.
I am going crazy with this, please tell me there's something easy I overlooked, I searched for hours!
In other words, is there a way to use a Linker Symbol from within inline asm/C++ without having it relocated whatsoever? No entry to the .reloc section or anything, basically same as a constant like $1234. So if a DLL gets loaded into another base address, that constant would be the same everytime.
UPDATE: I forgot about this question but decided to bring an update, since it seems it's likely not possible as nobody even commented. For anyone else in the same boat as me, I presume this is a limitation of the COFF object format itself. In other words, external symbols are implicitly relocated, and it doesn't seem there's a way against this.
I didn't "fix" it the way I wanted, I did it in a very hacky way though. If anyone is interested, here's my ugly "hack":
First I put a special "custom" instruction in the inline assembly where I reference this external symbol from C++. This "custom" instruction holds a placeholder instruction that grabs the symbol (normal x86 asm instruction with a dummy constant, e.g. 1234) and a way to identify it. Then let GCC generate the assembly files (.S files), then I parse the assembly with a simple script and when I find that "custom" instruction I insert a label for the linker (make it .global) and at the same time add a directive to a custom "on-the-fly" generated linker script that gets included from my main linker script at the end.
This places data in a temporary section in the resulting DLL with absolute offsets to the custom instruction that I need, but without relocation.
Next, I parse the binary DLL itself, in particular that temporary section I added with all this hack. I take the offsets from there, convert them to file offsets, and modify the DLL's .text section directly where those offsets point (remember those placeholder instructions? it is replacing their immediate constants 1234 with the respective value from the linker's non-relocated constant). Then I strip the temporary section from the DLL, and it's done. Of course, all of this is done automatically by a helper program and script
It's an insane hack, but it works and it's fully automatic now that I got it going. If my assumption is correct that COFF doesn't support non-relocated external symbols, then it's really the only way to use linker constants from C++ without them being relocated, which would be a disaster.
Firstly sorry for my poor English.
I'm now writing a tool to build import library of a dll in windows, it will output the binary form of obj files. then I can put them together by linker.
Currently it can generate OBJs with import descriptor, import lookup table, import thunk.
I nearly make it work but get stuck with some problem.
I researched the import library (.lib) files generated by VC and GCC, deciding to imitate what GCC does.
I found that it contains IMAGE_IMPORT_DESCRIPTORs and IMAGE_THUNK_DATA32s just the same as what I saw in an EXE.
So I make my own one as it does, but the linker won't generate the EXE as I want.
I hope it links my lib in this order:
.idata$2 (IMAGE_IMPORT_DESCRIPTOR)
.idata$4 as FirstThunk
.idata$4 contains NullThunk
.idata$5 as OriginalFirstThunk
.idata$5 contains NullThunk
.idata$6 contains DLL's filename
I knew that the number after section name + $ can direct the linker to put the data in order, and that's NULL_IMPORT_DESCRIPTOR in .idata$3 does.
the problem is, the linker strips the section which is not referred even if another section in the same obj is referred. for example, I put NullThunk and DLL's name in a file called dllname, contains
section 1: .idata$4, 0x00000000
section 2: .idata$5, 0x00000000
section 3: .idata$6, a.dll\0
symbols: _DllName, external, sect3+0
the _DllName is referred by the import descriptor so it's linked, but .idata$4 and .idata$5 is stripped.
so the NullThunk is not linked, what I see in CFF Explorer is that the EXE which linker generated imports hundreds of symbols from my DLL.
The second question is, how can I direct the linker to make sure that NullThunk is put after IAT? because they are all in .idata$5.
And another problem, when I try to make my lib contains 2 imported functions, the linker select only one of them, throw the other away. In detail, I have 2 functions: int __stdcall add(int, int); and sub. In the linker's generated code, call to sub seems correct but call to add become "call [RVA:0] (ff 25 00400000)". What happened?
After a week's trial and error finally I gave up and turn to asking for help.
this is the file my tool generated(I packed them into a lib):
http://filebin.ca/19oJUzj8z1vN/add.lib
neither GCC nor VC generated correct EXE when link to this import library.
How can I solve these problems?
Regards,
LeiMing
I have solved this problem.
I just rename all members in generated archive (.lib file) to the same name,
and put the nullthunk object in the end of the archive (last member),
the 3 problems disappear.
I don't know the reason in details but it works.
Now my tool can generate import library successfully.
and other points that need to care about are the order of offsets, strings in first link member and second link member. Padding between archive members is also a point to care.