How to implement rundll2-callable function? - winapi

I'm trying to write an exe that also exports functions which can be called with rundll32. Is this possible and if so, why isn't it working like this?
I closely followed Microsoft's advice on this.
#define RUNDLL32(func) extern "C" __declspec(dllexport) void CALLBACK func(HWND hWnd, HINSTANCE hInst, LPSTR lpszCmdLine, int nCmdShow)
RUNDLL32(MyFunc)
{
MessageBox(0, 0, 0, 0);
}
But when called with
rundll32 myprog.exe,_MyFunc#16
rundll32 crashes/DEP kicks in.

Rundll32.exe uses LoadLibrary() to load the executable image. This is not likely to work out well for an EXE, it doesn't expect to get loaded on an address that's that not its default. Which is guaranteed to happen, rundll32.exe already occupies that default address. Not sure if you could tinker with the linker so that it doesn't omit the relocation records.
But don't bother with this approach, just create a DLL instead of an EXE. And pass real arguments to MessageBox(). And, yes, use a .def file to rename the exported function.

Related

GetProcAddress doesn't work for functions other than void

I have a problem with GetProcAddress:
I wrote a simple DLL with just one function in it:
extern "C" LRESULT WINAPI Function(HWND Hwnd, UINT Message,
WPARAM wParam, LPARAM lParam)
{
Beep(1000, 1000);
return CallNextHookEx(0, Message, wParam, lParam);
}
When I try to get function's address GetProcAddress fails with the ErrorCode 127 (ERROR_PROC_NOT_FOUND). However, if I use void as the function type it works perfectly. I can't really figure out why it behaves like this. Any suggestions would be greatly appreciated!
BTW: DependencyWalker shows that the function's name is indeed "Function" no changes have been applied.
There are only two failure modes for GetProcAddress:
you didn't export the function
you didn't get the name right
The exported named of this function is not "Function" unless you used a .def file to rename the export or created a 64-bit DLL. It will be "_Function#16" for a 32-bit build. The #16 postfix would be strongly associated with the fact that you have trouble making it work for functions with any arguments.
From the Visual Studio Command Prompt run Dumpbin.exe /exports on your DLL to see the exports. Delete a .pdb file in the same directory if there is one.
It's good idea to use module definition (.def file) with names of exported functions instead of __declspec(dllexport). It's much easier to manage them.
Also this
#define DllExport extern "C" __declspec (dllexport)
causes that exported dll function names are without any c++ "decorations"

VB - Linking a DLL in implicit way

I'm working on a VB6 graphic interface and I need to make an implicit linking to a DLL.
The motivation for this comes from my previous question. The DLL in question uses static TLS, __declspec(thread), and of course this fails horribly when the DLL is linked explicitly using LoadLibray.
I'd really like to avoid modifications to the DLL, so does anyone know how to trick a VB6 executable to link to a specific DLL implicitly?
Create an IDL file for your DLL that describes your exported functions in a moduleclause.
Compile with the MIDL compiler and reference the resulting tlb file from your VB6 project (Project - References).
And remove all Declare Functions.
The tlb file is only used for compilation (in this case), you don't have to include it into the setup.
Here is a sample IDL that import functions from standard OS dlls
[
uuid(YOURTYPE-LIBG-UIDH-ERE0-000000000000),
version(1.0),
helpstring ("My Type Library 1.0")
]
library MyTypeLib
{
importlib("stdole2.tlb");
typedef struct {
long Data1;
short Data2;
short Data3;
BYTE Data4[8];
} VBGUID;
typedef VBGUID CLSID;
[dllname("OLEAUT32")]
module OleAut32
{
[entry("SysAllocString")]
BSTR SysAllocString([in] long lpStr);
...
};
[dllname("USER32")]
module User32
{
[entry("RegisterClipboardFormatA")]
UINT RegisterClipboardFormat([in] LPSTR lpszFormat);
[entry("FillRect")]
DWORD FillRect([in] DWORD hDC, [in] int lpRect, [in] DWORD hBrush);
...
};
[dllname("BOGUS")]
module Strings
{
const LPSTR CLSID_DsQuery = "{8A23E65E-31C2-11D0-891C-00A024AB2DBB}";
const LPSTR CLSID_DsFindObjects = "{83EE3FE1-57D9-11D0-B932-00A024AB2DBB}";
...
}
}
Finally I was able to solve the problem thanks to GSerg and David Heffernan help.
Here the IDL to be used to generate the .tlb
[
uuid(12345678-1234-1234-1234-123456789ABC),
version(1.0)
]
library myTypeLib
{
[dllname("myLib.dll")]
module myLib
{
[entry("myFunc")]
int __stdcall myFunc( LPSTR filename_in, LPSTR filename_out, LPSTR ErrMsg);
};
};
To compile it use the command "midl" in the Visual Studio command prompt.
The resulting .tlb file should be placed in the same directory of the VB6 project, together with the DLL.
In the VB6 project under Project->References it's possible to add the .tlb file.
If all is gone well, pressing F2, would be possible to notice "myTypeLib" in the list of the available library.
Now it's possible to call "myFunc" inside the VB6 project!
However there are two issue to point out:
1)Some variable types are not compatible between VB6 and C. An example of this issue is rapresented by char arrays. While in VB6 they are declared as Dim myStr as String, in C they are usually declared as char myStr[MAX_DIM];. To make possible the translation between VB6 and C, without modifing the DLL, it's possible to declare on VB6 side the strings as Dim myStr as String * 256, while in the IDL file the corrispondent string should be passed to the function as LPSTR myStr.
2)VB6 does not link the DLLs until the .exe is created. But if a DLL is not linked, then its functions are not visible. For this reasons, all the function of the implicitly linked DLLs that have to be used in the VB6 project, must be included in the IDL file.
Moreover, for the same reason, even after all the functions have been included in the IDL file, won't be possible to run the program from the IDE (it will crash) as to debug it. The only way to run the application is to create the .exe.

What does this line of C/C++ preprocessor mean?

This is Line 519 of WinNT.h (BUILD Version: 0091)
#define DECLARE_HANDLE(name) struct name##__{int unused;}; typedef struct name##__ *name
Why do we need a pointer to an struct with a single int member with a weird name called unused?
And will we ever need to use a line of code like this one?
HINSTANCE hInstance = new HINSTANCE__;
Overall declaring different data types with the same structures, doesn't make sense to me. What's the idea behind this?
DECLARE_HANDLE(HRGN);
DECLARE_HANDLE(HRSRC);
DECLARE_HANDLE(HSPRITE);
DECLARE_HANDLE(HLSURF);
DECLARE_HANDLE(HSTR);
DECLARE_HANDLE(HTASK);
DECLARE_HANDLE(HWINSTA);
DECLARE_HANDLE(HKL);
The point is for the different handles to have different types so that, for example, a HINSTANCE isn't assignable to a HANDLE. If they were all defined as "void*", then there are classes of errors that the compiler could not detect.
And will we ever need to use a line of code like this one?
HINSTANCE hInstance = new HINSTANCE__;
You usually use a HINSTANCE value returned by a Windows system call; I have never seen code executing a line like that.
They don't actually point to anything to memory; they are just used to refer to objects (files, resource, semaphores, windows) when making calls to the Windows API. While they're nothing more than just indexes into kernel's object tables, the developers decided that they make it a pointer to an unused structure which would make them "opaque" and cause less confusion between other types. The DECLARE_HANDLE is a function macro that does just that - declaring opaque types for handles.

What does APIENTRY do?

I encountered it first time and found no dedicated page on msdn. What does APIENTRY mean?
APIENTRY is an alias for WINAPI.
WINAPI itself is a definition for the type of calling convention used for windows API calls, the stdcall.
Basically this is explaining to the compiler how to handle the stack and arguments when calling this function. You don't usually need to worry about it unless you are making function pointers to these types of functions.
It's just a #define for WINAPI, which is the standard decoration for a Windows entrypoint.
#define APIENTRY WINAPI

Determine the current HINSTANCE?

The HINSTANCE of a win32 application is passed to WinMain, but is there any other way of determining the current HINSTANCE (in case you couldn't tell, I'm very new to win32 programming!)? I need to create a window inside of a library and (since the library is cross platform), id prefer not to have to pass it in.
If memory serves, GetModuleHandle(NULL); returns the instance handle.
__ImageBase is your friend, especially in the case of libraries.
Note that the linked blog post (by R. Chen, although not the same post as the one linked by Brian Bondy) is worth reading (including the comments!)
If you are using MFC, you can use AfxGetInstanceHandle.
If you are not using MFC you can use: GetWindowLong(hWnd, GWL_HINSTANCE)
The function AfxGetStaticModuleState() does the trick.
If you call it within a dll, the functions returns the handle to the dll, if the call within a exe it returns the handle to the executable.
DWORD size;
TCHAR fileName [MAX_PATH];
HMODULE hModule = AfxGetStaticModuleState()->m_hCurrentInstanceHandle;
::GetModuleFileName (hModule, fileName, size);

Resources