C assembly pushl not working for WriteFile function - winapi

Question

i used C-assembly to call the (_GetStdHandle#4) function to get (output) handle then used (_WriteFile#20) function to write my string on console using handle that i got from (_GetStdHandle#4).
i used (pushl) in my source code for each function to pass the parameters but something's is wrong because (WriteFile)) function return error (6) which is invalid handle but the handle is valid ... so something's wrong with passing argument ... yes ... my problem is passing argument to (_WriteFile) function using (pushl) ... in this code, i used (g) for each argument because there is no reason to move the parameters to register then push the registers ... so i didn't used (r) but if i use (r), the program work without any problem (which mov the parameters to registers first then push the registers (which i want to push the parameters without moving them into the registers)
this code is show nothing and the problem is from (WriteFile) function and if i use (r) for (WriteFile) parameters, the print will be done but why i can't use "g" to not mov the parameters to registers ?
typedef void * HANDLE;
#define GetStdHandle(result, handle) \
__asm ( \
"pushl %1\n\t" \
"call _GetStdHandle#4" \
: "=a" (result) \
: "g" (handle))
#define WriteFile(result, handle, buf, buf_size, written_bytes) \
__asm ( \
"pushl $0\n\t" \
"pushl %1\n\t" \
"pushl %2\n\t" \
"pushl %3\n\t" \
"pushl %4\n\t" \
"call _WriteFile#20" \
: "=a" (result) \
: "g" (written_bytes), "g" (buf_size), "g" (buf), "g" (handle))
int main()
{
HANDLE handle;
int write_result;
unsigned long written_bytes;
GetStdHandle(handle, -11);
if(handle != INVALID_HANDLE_VALUE)
{
WriteFile(write_result, handle, "Hello", 5, & written_bytes);
}
return 0;
}
the Assembly code for this program is :
.file "main.c"
.def ___main; .scl 2; .type 32; .endef
.section .rdata,"dr"
LC0:
.ascii "Hello\0"
.text
.globl _main
.def _main; .scl 2; .type 32; .endef
_main:
LFB25:
.cfi_startproc
pushl %ebp
.cfi_def_cfa_offset 8
.cfi_offset 5, -8
movl %esp, %ebp
.cfi_def_cfa_register 5
andl $-16, %esp
subl $16, %esp
call ___main
/APP
pushl $-11
call _GetStdHandle#4
# 0 "" 2
/NO_APP
movl %eax, 12(%esp)
cmpl $-1, 12(%esp)
je L2
leal 4(%esp), %eax
/APP
pushl $0
pushl %eax
pushl $5
pushl $LC0
pushl 12(%esp)
call _WriteFile#20
# 0 "" 2
/NO_APP
movl %eax, 8(%esp)
L2:
movl $0, %eax
leave
.cfi_restore 5
.cfi_def_cfa 4, 4
ret
.cfi_endproc
LFE25:
.ident "GCC: (MinGW.org GCC-6.3.0-1) 6.3.0"
what is the problem ?

Answer 1

I would question the need for calling the WINAPI through wrappers like this rather than calling them directly. You can declare prototypes for the stdcall calling convention with
__attribute__((stdcall))
If you don't need to use inline assembly you shouldn't. GCC's inline assembly is hard to get right. Getting it wrong can make the code appear to work until one day it doesn't, especially if optimizations are enabled. David Wohlferd has a good article on why you shouldn't use inline assembly if you don't need to.
The primary problem can be seen in this section of generated code:
pushl $0
pushl %eax
pushl $5
pushl $LC0
pushl 12(%esp)
call _WriteFile#20
GCC has computed the memory operand (handle) for the first parameter as 12(%esp) . The problem is that you have altered ESP with the previous pushes and now offset 12(%esp) is no longer where handle is.
To get around this problem you can pass memory addresses through registers or as immediates (if possible). Rather than use g constraint which includes m (memory constraints), simply use ri for registers and immediates. This prevents memory operands from being generated. If you pass pointers through registers you will also need to add the "memory" clobber.
The STDCALL(WINAPI) calling convention allows a function to destroy EAX, ECX, and EDX (AKA the volatile registers). It is possible that GetStdHandle and WriteFile will clobber ECX and EDX as well as return a value in EAX. You need to ensure that ECX and EDX are listed as clobbers as well (or have a constraint that marks it as output), otherwise the compiler may assume the values in those registers are the same before and after the inline assembly blocks are completed. If they are different it could cause subtle bugs.
With these changes your code could look something like:
#define INVALID_HANDLE_VALUE (void *)-1
typedef void *HANDLE;
#define GetStdHandle(result, handle) \
__asm ( \
"pushl %1\n\t" \
"call _GetStdHandle#4" \
: "=a" (result) \
: "g" (handle) \
: "ecx", "edx")
#define WriteFile(result, handle, buf, buf_size, written_bytes) \
__asm __volatile ( \
"pushl $0\n\t" \
"pushl %1\n\t" \
"pushl %2\n\t" \
"pushl %3\n\t" \
"pushl %4\n\t" \
"call _WriteFile#20" \
: "=a" (result) \
: "ri" (written_bytes), "ri" (buf_size), "ri" (buf), "ri" (handle) \
: "memory", "ecx", "edx")
int main()
{
HANDLE handle;
int write_result;
unsigned long written_bytes;
GetStdHandle(handle, -11);
if(handle != INVALID_HANDLE_VALUE)
{
WriteFile(write_result, handle, "Hello", 5, &written_bytes);
}
return 0;
}
Notes:
I marked the WriteFile inline assembly as __volatile so that the optimizer can't remove the entire inline assembly if it thinks result isn't being used. The compiler doesn't know that a side effect of the function is that the display is updated. Mark the function volatile to prevent the inline assembly from being removed entirely.
GetStdHandle doesn't have a problem with potential memory operands because there are no further uses of constraints after the initial push %1. The problem you are encountering is only an issue when ESP has been modified (via a PUSH/POP or change to ESP directly) and there is a possible use of a memory constraint in that inline assembly afterwards.