"Vista-only" heap corruption in a MFC application - visual-studio

Ours is a MFC application, which links to a win32 DLL. When the application invokes this DLL function, the argument "buffer" becomes a "Bad Pointer", after entering the function stack. This results in an application crash.
static MyClass* Instance(string& buffer);
I changed the argument type to "char *", but it only pushes the crash to next statement in the function. How to detect this heap corruption?
Few hints
This crash is reproducible, even if I invoke this DLL function from the very start of our application (CWinApp constructor). Could this memory corruption be caused by loading of resources, manifest etc?
The crash is ocurring in Vista and Win7, but not in XP.
Both these projects were recently migrated from Visual Studio 2002 to VS2008.
Code that invokes the function
CString data = "some string";
string str = data.GetBuffer();
data.ReleaseBuffer();
MyClass *obj = MyClass::Instance(str);

There were two mistakes:
Couple of custom built C++ files were not compiled with MD switch. We had to add -MD to the custom build script to make the CRT consistant with other objects.
There were LNK2005 conflicts between LIBCMT.LIB and MSVCRT.LIB, which were otherwise ignored due to the /FORCE switch. We resolved these conflicts by removing LIBCMT.LIB in Linker->Input
Thanks all for your help.

My guess is this is wrong usage of calling conventions or a CRT mismatch (i go with calling conventions).
Try building a stub DLL with the same function signature (which does nothing) and make it work with your MFC app.
Here's an example...
HTH

Related

Running app puts up: "The application has failed to start because XYZ.DLL was not found." BUT it actually runs fine?

I have a question, I came across running an x86 app on XP x64 that throws up a couple (2) of the "The application has failed to start because XYZ.DLL was not found". My question is, it still runs fine! So how does that work? I know if some function is missing you get that and the app doesn't actually run. What causes this message yet it runs fine? I don't think LoadLibrary() would put up a message? Is it from some #pragma comment( lib, "XYZ.lib" ) in a library even if that module not used?
TIA!!
My question is, it still runs fine! So how does that work?
If a DLL function is statically linked and can't be found at runtime, the OS will fail to create and run the process at all. So, the obvious answer is that the DLL function is linked dynamically instead via calls to LoadLibrary() and GetProcAddress() at runtime.
I know if some function is missing you get that and the app doesn't actually run.
If the missing DLL function is linked statically, yes.
What causes this message yet it runs fine? I don't think LoadLibrary() would put up a message?
Actually, it can. Use SetErrorMode() to avoid that. This is stated as much in the LoadLibrary() documentation:
To enable or disable error messages displayed by the loader during DLL loads, use the SetErrorMode function.
See Silently catch windows error popups when calling LoadLibrary().
Is it from some #pragma comment( lib, "XYZ.lib" ) in a library even if that module not used?
Linking to a DLL's .lib creates static linkage to the DLL. Unless the linker has a delay-load feature available AND the project is making use of that feature, in which case any static calls to the lib's referenced DLL functions are converted into runtime calls to LoadLibrary()/GetProcAddress() by the compiler+linker for you.

How to solve LNK2019 unresolved external symbol DriverEntry referenced in function GsDriverEntry?

While I was compiling this project https://github.com/namazso/hdd_serial_spoofer
I got the error message above ,how can I solve this ?
I'm using vs 2017 and wdk 10 .
(Must compile in release ,debug mode is not supported .There is no DriverEntry function in this project ,the EntryPoint(void* ntoskrn, void* image, void* alloc) function in hwid.cpp is the real entry point .)
I did a lot of research but still failed to get it work .I'm a noob in kernel mode driver development .
The project uses (an apparently ignored) option
<EntryPointSymbol> to define EntryPoint as the entry.
This is documented here, but current documentation appears to mean this is really only for .exe and .dll projects.
The form of the mesage called from the Windows driver system
NTSTATUS DriverInitialize(
_DRIVER_OBJECT *DriverObject,
PUNICODE_STRING RegistryPath
)
Is incompatible with the EntryPoint in the project
EntryPoint(void* ntoskrn, void* image, void* alloc)
This is not so bad, as none of the parameters which are called for EntryPoint are used.
So the simplest implementation would be
extern "C"
{
DRIVER_INITIALIZE DriverEntry;
_Use_decl_annotations_
NTSTATUS
DriverEntry(
struct _DRIVER_OBJECT *DriverObject,
PUNICODE_STRING RegistryPath
)
{
EntryPoint(NULL, NULL, NULL);
return STATUS_SUCCESS;
}
}
Kernel development is not for the faint hearted, and running invalid kernel code on your computer could make it difficult to boot, or in extream cases damage the computer. I did not review any of the code in the project for correctness.
Please run the code in a virtual machine (vmware, virtualbox, hyper-v) to limit the damage it could do
This is not a normal driver, the kind that WDF directly supports. It is a "driverless driver", it uses an undocumented hack that is appealing to the kind of programmers that write rootkits for fun and profit. The DriverEntry() function is not actually the entrypoint for a driver, it is callback. Much like the WinMain() function is not actually the entrypoint for a native Win32 program. The EntryPoint() function in the project's source code is the replacement for the native driver entrypoint. Beware that the project appears to have rootkitty-like behavior, designed to fool a simplistic copy-protection scheme that checks a drive serial number.
The GsDriverEntry() function is the real entrypoint in a normal KMDF driver. It performs essential initialization to support the /GS compiler option, designed to detect buffer overflow. After that's done it calls DriverEntry(). The project replaces this entrypoint with EntryPoint().
This project was written with an old version of the Visual Studio project template. Several changes are necessary to get it to build properly:
C/C++ > Code Generation > Security Check. Must be "Disable security check (/GS-)", the original project file got this right.
Same property page > Control Flow Guard. Must be set to "No" to prevent a linker error. This option adds additional security checks that cannot work and must be disabled.
C/C++ > General > SDL checks. Use the dropdown arrow to override to "inherit from parent" so the option appears blank. More security checks that needs to be disabled, suppresses a warning that sdl- is incompatible with /gs-.
Same property page > Warning level. Override to "Level3 (/W3)", suppresses warnings about function arguments not being used.
Linker > Input > Additional Dependencies. Click the dropdown arrow > Edit. Untick the "Inherit from parent" checkbox and change to $(DDK_LIB_PATH)ntoskrnl.lib. Note the $(KernelBufferOverflowLib) entry in the Inherited values listbox, resolves to bufferoverflowfastfailk.lib, that is the one that contains GsDriverEntry() and produced the linker error.
Linker > Advanced > Entry Point. Must be "EntryPoint", the original project template got that right.
After this it builds clean. I did not test the resulting hwid.sys, looks a bit too evil to expose my machine to it.
Don't use .cpp source files to write your driver. Change them to .c and it should work. That was the case for me.
I know it looks trivial, or unlikely but remember Windows Kernel code is C code and Visual Studio makes various assumptions given the file extension of your source files.
There are several causes that generate the problem. In my case, a try/except in the code, which compiles on x64, but doesn't compile on x86.
Linker Tools Error LNK2019
https://learn.microsoft.com/en-us/cpp/error-messages/tool-errors/linker-tools-error-lnk2019?view=msvc-170

How to use VS2010 built dlls in VS2008

In my work we have VS2008.
Some partners in the project I work, use VS2010 (can't use VS2008)
They have to build a dll based and I have to use that dll in my framework...
I managed to build the main app and link the dll's.
the app starts, and objects from the VS2010 dll's are created,
but the app crashes when i try to delete these objects...
Windows has triggered a breakpoint in app.exe. This may be due to a
corruption of the heap, which indicates a bug in app.exe or any of the
DLLs it has loaded.
Have you any ideas on how to fix this?
Your co-workers' DLLs are linked against VS2010's runtime library. Your code is linked against VS2008's runtime library.
When you call some function from the VS2010 dll to allocate a new object, it will be allocated on that library's heap. When you call "delete" on that object, VS2008's runtime library will try to free it from its own heap. Since they're different, you get that error.
If you're going to mix runtimes like that, you need the VS2010 dll to expose free()-style functions (not just C++ destructors) for each type. There are other things you should be very careful with when mixing runtime libraries like that, such as using STL containers, or any sort of "copy-on-write" objects. In general, it's easier to avoid it.
Object that was allocated in dll or in an exe should be deallocated in same place. You need to talk about that with your partners. For the goal may be used overloading of allocating and deallocating operators http://www.cprogramming.com/tutorial/operator_new.html

DLL linker issue

I am compiling a DLL twice (once for x86, once for x64) and I have set /ENTRY to "DllMain". I am using the /MT runtime library option to statically link against the runtime library. This all work fine when doing the x86 build, but the x64 build fails with this:
error LNK2019: unresolved external symbol main referenced in function __tmainCRTStartup
{project directory}\LIBCMT.lib(crt0.obj)
Why does this work for the x86 build and not the x64 build? Is there something I am missing here?
Not a direct answer but it may be strictly related: as said in the comment, you should avoid changing the entrypoint in that way: normally the real entrypoint is taken by a "fake" DllMain provided by the CRT to initialize its internal data structures (as explained here), so you're bypassing it. Probably the size reduction is due to CRT init code being removed.
Your dll is working with a non-initialized CRT, which is very bad. You should leave the default entrypoint, which, incidentally, should solve your problem.
By the way, notice that actually you could make a dll without the CRT (and it would become really small), but you shouldn't use the CRT at all, without even linking against it (/NODEFAULTLIB switch). This means that you could just use libraries you explicitly link against (e.g. the Windows API), but I suspect you would lose several C++ features (I think at least exceptions and RTTI).
This could be a silly question, but are you sure you're linking as a DLL in the x64 case (ie. specifying the /DLL switch) - since the complaint is about main, I wonder whether it's trying to link as a an executable?

Strange VB6 build problems (related to nlog)

This I think is related to my use of the nlog C++ API (and my question on the nlog forum is here); the purpose of my asking this question here is to get a wider audience to my problem and perhaps to also get some more general ideas behind the VB6 IDE's failure to build in my particular scenario.
Briefly, the problem that I am having is that I am having trouble building VB6 components which reference unmanaged C++ components which have calls to nlog's C\C++ API (which is defined in NLogC.DLL). The build problems are not occurring during compile time, they are occurring when the binary is being built which suggests to me that it's some kind of linker type problem? Don't know enough about how VB6 binaries are produced to tell. The VB6 binary is produced, but it is corrupted and crashes shortly after it is invoked.
Has anyone had any similar experiences with VB6 (doesn't have to be related to nlog or C++)?
edit: Thanks for all the responses to this rather obscure problem. Still no headway unfortunately; my findings since I posted this:
'Tweaking' the compile options doesn't appear to help in this problem.
Adding a reference to the nlog-enabled C++ component from a 'blank' VB6 project doesn't crash it or cause weird build problems. So it isn't a 'native' VB6 issue, possibly an issue with the interaction between nlog and the various components and 3rd party libraries used by other referenced components?
As for C++ calling conventions: the nlog-enabled C++ component is - as far as I can see - compliant to these conventions and indeed works fine when referenced by VB6 as long as it is not making any nlog API calls. Not sure if the nlogc.DLL itself is VB6 compliant but I would have thought that that is immaterial since the API calls are being made from the C++ component; VB6 shouldn't know or care about what the C++ component is referencing (that's as far as my understanding on this goes...)
edit2: I should also note that the error message obtained during build is: "Errors during load. Please refer to "xxx" for details". When I bring up the log file, all that there is in there is: "Cannot load control xxx". Interestingly, all references to that particular control disappears from that particular project resulting in compile errors if I were to try to build again.
Got around the problem by using NLog's COM interface (NLog.ComInterop.DLL) from my unmanaged C++ code. Not as easy to do as the C\C++ API but at least it doesn't crash my VB6 components.
I would try tweaking some of the Compile options found in the Project, Properties menu, Compile panel to see if they yield any additional hints as to what is going wrong.
For example if you compile the executable to p-code rather than native code does it still crash on startup.
What error message do you get when you run your compiled binary?
I doubt the compiler/linker is the problem: project references in a VB6 project are not linked into the final executable. A project reference in VB6 is actually a reference to a COM type library (which may or may not be embedded in a .dll or other binary file type). Project references primarily serve two purposes:
The IDE extracts type information from the referenced type libraries which it then displays in the Object Browser (and in the Intellisense drop-down)
At compile-time, the compiler extracts the type information stored in the referenced libraries, including the CLSID of each class that you instantiate, and embeds this data into the executable. This allows your executable to create instances of classes contained in the libraries that you referenced.
Note that the compiled binary doesn't link to any code in the referenced libraries, and it doesn't even contain the filenames of the referenced libraries. The final executable only contains the CLSID's and other type information that it needs to instantiate COM objects at run-time.
It is much more likely that the issue is with NLog, or with how you are calling it from your code, rather than something gone awry in the VB6 compile process.
If you think it might be a linker problem, this should crash it the same way:
create a new standard project (of any kind)
add a new module and copy the "declare"-statements into it
compile
If it doesn't crash it is something else.
It would help an exact description of the error or a screenshot of what going on.
One thing to check is wherever NLogC.DLL or the C++ DLL you built have the correct calling convention defined. Basically you can't have the DLL function names mangled or use anything but the STDCALL calling convention. If the C++ DLL has not been created with those two things in mind then it will fail to work with VB6.
MSDN Article on Calling convention.
"Cannot load control xxx" errors can be caused by .oca files which were created from a different version of an .ocx than currently used. If that is the case, deleting the .oca files helps.

Resources