Size of shadow area on the stack in assembler language - winapi

I found the sample program below somewhere on the Web. Various copies of it abound, usually with small differences. But my question concerns the size of the shadow area at the top of the stack when calling a function from the Windows API. This program works perfectly as shown, with decimal 40 subtracted from the stack pointer, to allow room for the 4 parameters that are passed in registers, plus one more. However, in this case there is no 5th parameter, and yet if the sub rsp, 40 is changed to sub rsp, 32, and no other changes are made, the 'Hello world' window is no longer displayed! Is there some reason why when only 4 parameters are involved, all of which are passed in registers, it's still necessary to reserve 40 (5*8) bytes at the top of the stack rather than only 32 (4*8)?
; Sample x64 Assembly Program
; Chris Lomont 2009 www.lomont.org
; command to assemble is:
; ml64 hello.asm /link /subsystem:windows /defaultlib:kernel32.lib /defaultlib:user32.lib /entry:Start
extrn ExitProcess: PROC ; in kernel32.lib
extrn MessageBoxA: PROC ; in user32.lib
.data
caption db '64-bit hello!', 0
message db 'Hello World!', 0
.code
Start PROC
sub rsp, 40 ; shadow space, aligns stack
mov rcx, 0 ; hWnd = HWND_DESKTOP
lea rdx, message ; LPCSTR lpText
lea r8, caption ; LPCSTR lpCaption
mov r9d, 0 ; uType = MB_OK
call MessageBoxA ; call MessageBox API function
mov ecx, eax ; uExitCode = MessageBox(...)
call ExitProcess
Start ENDP
End

Related

error C2146: syntax error: missing ';' before identifier 'SEGMENT' in masm Visual Studio [duplicate]

I'm here to ask you some stuff about VS2017.
In the past I had used WinAsm for MASM and I never got problems with it.
However, when I'm trying to do something with MASM in VS2017, I always gonna get problems and stuff...
I've checked the whole internet about "how to set up VS for MASM", but nothing has helped me as I'm always getting troubles...
Is there any way to use Visual Studio 2017 for MASM32/64bit without any kind of headache?
Can someone give me the ultimate guide to set up VS2017 for assembly programming?
Thanks you very much and sorry for my weak english.
How to build a x64/x86-project with a standalone x64/x86 assembly file
1) Start Visual Studio (Community) 2017 and choose FILE - New - Project.
2) In the next window choose Empty Project.
3) Make sure, that the project is highlighted in the Solution Explorer and and choose PROJECT - Build Customizations....
4) In the next window tick masm(.targets,.props) and click on OK.
5) Choose PROJECT - Add New Item from the menu.
6) In the next window choose C++File(.cpp) and - IMPORTANT! - give it a name with an .asm extension. Click on Add.
7) Now you can fill the file with content.
Source.asm:
EXTERN GetStdHandle : PROC
EXTERN WriteFile : PROC
EXTERN ExitProcess : PROC
.DATA?
hFile QWORD ?
BytesWritten DWORD ?
.DATA
hello BYTE 'Hello world!', 13, 10
.CODE
main PROC
; https://blogs.msdn.microsoft.com/oldnewthing/20160623-00/?p=93735
sub rsp, 40 ; Shadow space (4 * 8) & 1 parameter (8 bytes)
; https://learn.microsoft.com/en-us/cpp/build/stack-allocation
and spl, -16 ; Align to 16
; https://msdn.microsoft.com/library/windows/desktop/ms683231.aspx
mov ecx, -11 ; DWORD nStdHandle = STD_OUTPUT_HANDLE
call GetStdHandle ; Call WinApi
mov hFile, rax ; Save returned handle
; https://msdn.microsoft.com/library/windows/desktop/aa365747.aspx
mov rcx, hFile ; HANDLE hFile (here: Stdout)
lea rdx, hello ; LPCVOID lpBuffer
lea r9, BytesWritten ; LPDWORD lpNumberOfBytesWritten
mov r8d, LENGTHOF hello ; DWORD nNumberOfBytesToWrite
mov qword ptr [rsp+32], 0 ; LPOVERLAPPED lpOverlapped = NULL
call WriteFile ; Call WinAPI
exit:
; https://msdn.microsoft.com/library/windows/desktop/ms682658.aspx
xor ecx, ecx ; Set RCX to null for return value
call ExitProcess ; Call WinAPI to exit
main ENDP
end
This is a 64-bit Console application that starts at the procedure main.
8) Change the Solution Platforms to x64
9) Choose PROJECT - Properties.
10) In the Properties window you have to complete two linker options:
Entry Point: main
SubSystem: Console (/SUBSYSTEM:CONSOLE)
Choose at the left side Configuration Properties - Linker - All Options , change both options at once and click OK.
11) Build and run the .exe with CTRL-F5. The application will be opened in a new window.
Now overwrite Source.asm with a 32-bit Console application:
.MODEL flat, stdcall
; https://learn.microsoft.com/en-us/cpp/assembler/masm/proto
GetStdHandle PROTO STDCALL, ; https://learn.microsoft.com/en-us/windows/console/getstdhandle
nStdHandle: SDWORD
WriteFile PROTO STDCALL, ; https://learn.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-writefile
hFile: DWORD, ; output handle
lpBuffer: PTR BYTE, ; pointer to buffer
nNumberOfBytesToWrite: DWORD, ; size of buffer
lpNumberOfBytesWritten: PTR DWORD, ; num bytes written
lpOverlapped: PTR DWORD ; ptr to asynchronous info
ExitProcess PROTO STDCALL, ; https://learn.microsoft.com/en-us/windows/desktop/api/processthreadsapi/nf-processthreadsapi-exitprocess
dwExitCode: DWORD ; return code
.DATA ; https://learn.microsoft.com/en-us/cpp/assembler/masm/dot-data
Hallo db "Hello world!",13,10
.DATA? ; https://learn.microsoft.com/en-us/cpp/assembler/masm/dot-data-q
lpNrOfChars dd ?
.CODE ; https://learn.microsoft.com/en-us/cpp/assembler/masm/dot-code
main PROC ; learn.microsoft.com/en-us/cpp/assembler/masm/proc
invoke GetStdHandle, -11 ; -> StdOut-Handle into EAX
invoke WriteFile, eax, OFFSET Hallo, LENGTHOF Hallo, OFFSET lpNrOfChars, 0
invoke ExitProcess, 0
main ENDP
END main ; https://learn.microsoft.com/en-us/cpp/assembler/masm/end-masm
Change the Solution Platforms to x86 (No. 8 above) and complete the project properties with SubSystem: Console (/SUBSYSTEM:CONSOLE) (No. 10 above). You must not set the Entry point, because ml32 expects the entry point after the END directive (END main). Build and run the .exe with CTRL-F5.

FASM write Hello World to console with NO includes or dependencies at all

I've seen
How to write hello world in assembler under Windows?
and
Writing hello,world to console in Fasm with DOS
How to write to the console in fasm?
I've tried / seen code like this MASM example from this answer
;---ASM Hello World Win64 MessageBox
extrn MessageBoxA: PROC
extrn ExitProcess: PROC
.data
title db 'Win64', 0
msg db 'Hello World!', 0
.code
main proc
sub rsp, 28h
mov rcx, 0 ; hWnd = HWND_DESKTOP
lea rdx, msg ; LPCSTR lpText
lea r8, title ; LPCSTR lpCaption
mov r9d, 0 ; uType = MB_OK
call MessageBoxA
add rsp, 28h
mov ecx, eax ; uExitCode = MessageBox(...)
call ExitProcess
main endp
End
(to which I get an error "Illegal instruction" on windows 64 bit extrn MessageBoxA:PROC because FASM doesn't understand that MASM directive.)
also this FASM example from this question
; Example of 64-bit PE program
format PE64 GUI
entry start
section '.text' code readable executable
start:
sub rsp,8*5 ; reserve stack for API use and make stack dqword aligned
mov r9d,0
lea r8,[_caption]
lea rdx,[_message]
mov rcx,0
call [MessageBoxA]
mov ecx,eax
call [ExitProcess]
section '.data' data readable writeable
_caption db 'Win64 assembly program',0
_message db 'Hello World!',0
section '.idata' import data readable writeable
dd 0,0,0,RVA kernel_name,RVA kernel_table
dd 0,0,0,RVA user_name,RVA user_table
dd 0,0,0,0,0
kernel_table:
ExitProcess dq RVA _ExitProcess
dq 0
user_table:
MessageBoxA dq RVA _MessageBoxA
dq 0
kernel_name db 'KERNEL32.DLL',0
user_name db 'USER32.DLL',0
_ExitProcess dw 0
db 'ExitProcess',0
_MessageBoxA dw 0
db 'MessageBoxA',0
but it displays a message box and also has external dependencies "kernel32.dll" and "user32.dll"
also tried this example from the FASM forum
format pe console
include 'win32ax.inc'
entry main
section '.data!!!' data readable writeable
strHello db 'Hello World !',13,10,0
strPause db 'pause',0
section '.txt' code executable readable
main:
; you can use crt functions or windows API.
cinvoke printf,strHello
cinvoke system,strPause; or import getc()
; or
; invoke printf,srtHello
; add esp, 4
; or use WriteFile and GetStdHandle APIs
push 0
call [ExitProcess]
section '.blah' import data readable
library kernel32,'kernel32.dll',\
msvcrt,'msvcrt.dll' ;; C-Run time from MS. This is always on every windows machine
import kernel32,\
ExitProcess,'ExitProcess'
import msvcrt,\
printf,'printf',\
system,'system'
but it depends on win32ax.inc and other imports
also
format PE console
include 'win32ax.inc'
.code
start:
invoke WriteConsole,<invoke GetStdHandle,STD_OUTPUT_HANDLE>,"Hello World !",13,0
invoke Sleep,-1
.end start
but requires "win32ax.inc" import
closest I could find without the win32ax from the FASM forum:
format pe64 console
entry start
STD_OUTPUT_HANDLE = -11
section '.text' code readable executable
start:
sub rsp,8*7 ; reserve stack for API use and make stack dqword aligned
mov rcx,STD_OUTPUT_HANDLE
call [GetStdHandle]
mov rcx,rax
lea rdx,[message]
mov r8d,message_length
lea r9,[rsp+4*8]
mov qword[rsp+4*8],0
call [WriteFile]
mov ecx,eax
call [ExitProcess]
section '.data' data readable writeable
message db 'Hello World!',0
message_length = $ - message
section '.idata' import data readable writeable
dd 0,0,0,RVA kernel_name,RVA kernel_table
dd 0,0,0,0,0
kernel_table:
ExitProcess dq RVA _ExitProcess
GetStdHandle dq RVA _GetStdHandle
WriteFile dq RVA _WriteFile
dq 0
kernel_name db 'KERNEL32.DLL',0
user_name db 'USER32.DLL',0
_ExitProcess db 0,0,'ExitProcess',0
_GetStdHandle db 0,0,'GetStdHandle',0
_WriteFile db 0,0,'WriteFile',0
but still requires the kernel32.dll and user32.dll
Any way to do this without any external DLLs at all? I know just the program fasm itself does it, and prints to the console, doesn't it?
Any way to do this without any external DLLs at all?
Under Windows: Definitely no!
Windows uses some methods (probably syscall) to enter the operating system, however, there are no official entry points.
This means that it is (unlikely but) possible that exactly the same program that shows the "Hello world" message box in the current Windows version will do something completely different after the next Windows update!
Because Microsoft is assuming that every Windows program is only calling the OS by using the .dll files that match the kernel version, they can do this.
I don't know about Windows 10, but an older Windows version (I don't remember if it was XP, Vista or 7) even simply assumed that an .exe file returns at once if it does not use any .dll file: The program was not even started in this case!
I know just the program fasm itself does it, and prints to the console
That is not the case, fasm is also using the kernel32 APIs.
FWIW kernel32 is loaded into the memory space of every process in Windows, so there is no penalty or overhead in using the kernel32 APIs.
You may like this Windows example in €ASM, which doesn't explicitly mention any DLL and doesn't require other external libraries.
Just save the source as "bluej.asm", assemble and link with euroasm bluej.asm and run as bluej.exe.
Nevertheless, you won't get away without using API functions imported from the default Windows system library "kernel32.dll".
bluej PROGRAM Format=PE, Entry=Start:
IMPORT GetStdHandle,WriteFile,ExitProcess
Start: PUSH -11 ; Param 1: standard output handle identificator.
CALL GetStdHandle; Return StdOutput handle in EAX.
PUSH 0 ; Param 5: no overlap.
PUSH Written ; Param 4: Address of a variable to store number of written bytes.
PUSH MsgSize ; Param 3: Number of bytes to write.
PUSH Msg ; Param 2: Address of text.
PUSH EAX ; Param 1: Output file handle.
CALL WriteFile ; System call.
PUSH 0 ; Errorlevel.
CALL ExitProcess ; System call.
Written DD 0
Msg DB "Hello, world!"
MsgSize EQU $ - Msg
ENDPROGRAM
What constitures as "dependency" to you? If you want to avoid even operating system DLL's, then you're probably out of luck. You can't rely on syscall numbers alone.
"no dependencies" can also mean "just using existing OS DLL's", such as ntdll, kernel32, etc., but without using 3rd party DLL's that may not be present, such as a specific version of the C runtime.
One method I would like to show is retrieving function pointers from the PEB. This is code that I've written and that I personally use, if I want to have shellcode that has no import section.
PebGetProcAddress works similarly to GetProcAddress, except that the DLL name and function name must be a hash, and the DLL must be loaded by using LoadLibrary.
This may not answer your question exactly, but I hope it gets you somewhat closer to your goal or help others who read it.
PebApi.asm
proc PebGetProcAddress ModuleHash:DWORD, FunctionHash:DWORD
local FirstEntry:DWORD
local CurrentEntry:DWORD
local ModuleBase:DWORD
local ExportDirectory:DWORD
local NameDirectory:DWORD
local NameOrdinalDirectory:DWORD
local FunctionCounter:DWORD
; Get InMemoryOrderModuleList from PEB
mov eax, 3
shl eax, 4
mov eax, [fs:eax] ; fs:0x30
mov eax, [eax + PEB.Ldr]
mov eax, [eax + PEB_LDR_DATA.InMemoryOrderModuleList.Flink]
mov [FirstEntry], eax
mov [CurrentEntry], eax
; Find module by hash
.L_module:
; Compute hash of case insensitive module name
xor edx, edx
mov eax, [CurrentEntry]
movzx ecx, word[eax + LDR_DATA_TABLE_ENTRY.BaseDllName.Length]
test ecx, ecx
jz .C_module
mov esi, [eax + LDR_DATA_TABLE_ENTRY.BaseDllName.Buffer]
xor eax, eax
cld
.L_module_hash:
lodsb
ror edx, 13
add edx, eax
cmp al, 'a'
jl #f
sub edx, 0x20 ; Convert lower case letters to upper case
##: dec ecx
test ecx, ecx
jnz .L_module_hash
; Check, if module is found by hash
cmp edx, [ModuleHash]
jne .C_module
; Get module base
mov eax, [CurrentEntry]
mov eax, [eax + LDR_DATA_TABLE_ENTRY.DllBase]
mov [ModuleBase], eax
; Get export directory
mov eax, [ModuleBase]
add eax, [eax + IMAGE_DOS_HEADER.e_lfanew]
mov eax, [eax + IMAGE_NT_HEADERS32.OptionalHeader.DataDirectoryExport.VirtualAddress]
add eax, [ModuleBase]
mov [ExportDirectory], eax
; Get name table
mov eax, [ExportDirectory]
mov eax, [eax + IMAGE_EXPORT_DIRECTORY.AddressOfNames]
add eax, [ModuleBase]
mov [NameDirectory], eax
; Get name ordinal table
mov eax, [ExportDirectory]
mov eax, [eax + IMAGE_EXPORT_DIRECTORY.AddressOfNameOrdinals]
add eax, [ModuleBase]
mov [NameOrdinalDirectory], eax
; Find function in export directory by hash
mov [FunctionCounter], 0
.L_functions:
mov eax, [ExportDirectory]
mov eax, [eax + IMAGE_EXPORT_DIRECTORY.NumberOfNames]
cmp eax, [FunctionCounter]
je .E_functions
; Compute hash of function name
xor edx, edx
mov esi, [NameDirectory]
mov esi, [esi]
add esi, [ModuleBase]
xor eax, eax
cld
.L_function_hash:
lodsb
test al, al
jz .E_function_hash
ror edx, 13
add edx, eax
jmp .L_function_hash
.E_function_hash:
; Check, if function is found by hash
cmp edx, [FunctionHash]
jne .C_functions
; Return function address
mov eax, [ExportDirectory]
mov eax, [eax + IMAGE_EXPORT_DIRECTORY.AddressOfFunctions]
add eax, [ModuleBase]
mov ebx, [NameOrdinalDirectory]
movzx ebx, word[ebx]
lea eax, [eax + ebx * 4]
mov eax, [eax]
add eax, [ModuleBase]
ret
.C_functions:
add [NameDirectory], 4
add [NameOrdinalDirectory], 2
inc [FunctionCounter]
jmp .L_functions
.E_functions:
; Function not found in module's export table
xor eax, eax
ret
.C_module:
; Move to next module, exit loop if CurrentEntry == FirstEntry
mov eax, [CurrentEntry]
mov eax, [eax + LIST_ENTRY.Flink]
mov [CurrentEntry], eax
cmp eax, [FirstEntry]
jne .L_module
; Module not found
xor eax, eax
ret
endp
PebApi.inc
macro pebcall modulehash, functionhash, [arg]
{
common
if ~ arg eq
reverse
pushd arg
common
end if
stdcall PebGetProcAddress, modulehash, functionhash
call eax
}
Example
PEB_User32Dll = 0x63c84283
PEB_MessageBoxW = 0xbc4da2be
; pebcall translates to a call to PebGetProcAddress and the call to the returned function pointer
pebcall PEB_User32Dll, PEB_MessageBoxW, NULL, 'Hello, World!', NULL, MB_OK
How to generate hashes for module names and function names
#define ROTR(value, bits) ((DWORD)(value) >> (bits) | (DWORD)(value) << (32 - (bits)))
DWORD ComputeFunctionHash(LPCSTR str)
{
DWORD hash = 0;
while (*str)
{
hash = ROTR(hash, 13) + *str++;
}
return hash;
}
DWORD ComputeModuleNameHash(LPCSTR str, USHORT length)
{
DWORD hash = 0;
for (USHORT i = 0; i < length; i++)
{
hash = ROTR(hash, 13) + (str[i] >= 'a' ? str[i] - 0x20 : str[i]);
}
return hash;
}

Visual C++ always builds succeeded but no output is created. Problem with the highlight asm extension VS2019 [duplicate]

I'm here to ask you some stuff about VS2017.
In the past I had used WinAsm for MASM and I never got problems with it.
However, when I'm trying to do something with MASM in VS2017, I always gonna get problems and stuff...
I've checked the whole internet about "how to set up VS for MASM", but nothing has helped me as I'm always getting troubles...
Is there any way to use Visual Studio 2017 for MASM32/64bit without any kind of headache?
Can someone give me the ultimate guide to set up VS2017 for assembly programming?
Thanks you very much and sorry for my weak english.
How to build a x64/x86-project with a standalone x64/x86 assembly file
1) Start Visual Studio (Community) 2017 and choose FILE - New - Project.
2) In the next window choose Empty Project.
3) Make sure, that the project is highlighted in the Solution Explorer and and choose PROJECT - Build Customizations....
4) In the next window tick masm(.targets,.props) and click on OK.
5) Choose PROJECT - Add New Item from the menu.
6) In the next window choose C++File(.cpp) and - IMPORTANT! - give it a name with an .asm extension. Click on Add.
7) Now you can fill the file with content.
Source.asm:
EXTERN GetStdHandle : PROC
EXTERN WriteFile : PROC
EXTERN ExitProcess : PROC
.DATA?
hFile QWORD ?
BytesWritten DWORD ?
.DATA
hello BYTE 'Hello world!', 13, 10
.CODE
main PROC
; https://blogs.msdn.microsoft.com/oldnewthing/20160623-00/?p=93735
sub rsp, 40 ; Shadow space (4 * 8) & 1 parameter (8 bytes)
; https://learn.microsoft.com/en-us/cpp/build/stack-allocation
and spl, -16 ; Align to 16
; https://msdn.microsoft.com/library/windows/desktop/ms683231.aspx
mov ecx, -11 ; DWORD nStdHandle = STD_OUTPUT_HANDLE
call GetStdHandle ; Call WinApi
mov hFile, rax ; Save returned handle
; https://msdn.microsoft.com/library/windows/desktop/aa365747.aspx
mov rcx, hFile ; HANDLE hFile (here: Stdout)
lea rdx, hello ; LPCVOID lpBuffer
lea r9, BytesWritten ; LPDWORD lpNumberOfBytesWritten
mov r8d, LENGTHOF hello ; DWORD nNumberOfBytesToWrite
mov qword ptr [rsp+32], 0 ; LPOVERLAPPED lpOverlapped = NULL
call WriteFile ; Call WinAPI
exit:
; https://msdn.microsoft.com/library/windows/desktop/ms682658.aspx
xor ecx, ecx ; Set RCX to null for return value
call ExitProcess ; Call WinAPI to exit
main ENDP
end
This is a 64-bit Console application that starts at the procedure main.
8) Change the Solution Platforms to x64
9) Choose PROJECT - Properties.
10) In the Properties window you have to complete two linker options:
Entry Point: main
SubSystem: Console (/SUBSYSTEM:CONSOLE)
Choose at the left side Configuration Properties - Linker - All Options , change both options at once and click OK.
11) Build and run the .exe with CTRL-F5. The application will be opened in a new window.
Now overwrite Source.asm with a 32-bit Console application:
.MODEL flat, stdcall
; https://learn.microsoft.com/en-us/cpp/assembler/masm/proto
GetStdHandle PROTO STDCALL, ; https://learn.microsoft.com/en-us/windows/console/getstdhandle
nStdHandle: SDWORD
WriteFile PROTO STDCALL, ; https://learn.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-writefile
hFile: DWORD, ; output handle
lpBuffer: PTR BYTE, ; pointer to buffer
nNumberOfBytesToWrite: DWORD, ; size of buffer
lpNumberOfBytesWritten: PTR DWORD, ; num bytes written
lpOverlapped: PTR DWORD ; ptr to asynchronous info
ExitProcess PROTO STDCALL, ; https://learn.microsoft.com/en-us/windows/desktop/api/processthreadsapi/nf-processthreadsapi-exitprocess
dwExitCode: DWORD ; return code
.DATA ; https://learn.microsoft.com/en-us/cpp/assembler/masm/dot-data
Hallo db "Hello world!",13,10
.DATA? ; https://learn.microsoft.com/en-us/cpp/assembler/masm/dot-data-q
lpNrOfChars dd ?
.CODE ; https://learn.microsoft.com/en-us/cpp/assembler/masm/dot-code
main PROC ; learn.microsoft.com/en-us/cpp/assembler/masm/proc
invoke GetStdHandle, -11 ; -> StdOut-Handle into EAX
invoke WriteFile, eax, OFFSET Hallo, LENGTHOF Hallo, OFFSET lpNrOfChars, 0
invoke ExitProcess, 0
main ENDP
END main ; https://learn.microsoft.com/en-us/cpp/assembler/masm/end-masm
Change the Solution Platforms to x86 (No. 8 above) and complete the project properties with SubSystem: Console (/SUBSYSTEM:CONSOLE) (No. 10 above). You must not set the Entry point, because ml32 expects the entry point after the END directive (END main). Build and run the .exe with CTRL-F5.

Hello world in NASM with LINK.EXE and WinAPI

I'm trying to get a simple Hello world program in NASM to run.
I want to print to the console without using C-Libraries, interfacing directly with WinAPI.
I am using the Visual Studio provided LINK.EXE for linking.
Here's my code so far:
section .data
message: db 'Hello world!',10 ; 'Hello world!' plus a linefeed character
messageLen: db $-message ; Length of the 'Hello world!' string
global _start
extern GetStdHandle
extern WriteConsoleW
extern ExitProcess
section .text
_start:
; DWORD bytes;
mov rbp, rsp
sub rsp, byte 8
; hStdOut = GetStdHandle(STD_OUTPUT_HANDLE)
mov ecx, -11
call GetStdHandle
; WriteFile(hstdOut, message, length(message), &bytes, 0);
mov rcx, rax
mov rdx, message
mov r8, messageLen
lea r9, [rsp-4]
push 0
call WriteConsoleW
; ExitProcess(0)
mov rcx, 0
call ExitProcess
ret
Which I assemble and link like this:
nasm -f win64 .\ASM.ASM
link /entry:_start /nodefaultlib /subsystem:console .\ASM.obj "C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\um\x64\kernel32.lib" "C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\um\x64\user32.lib"
However when I run the resulting .exe file, I get nothing.
Some things I tried so far are
Using the decorated names (like _GetStdHandle#4), which resulted in the linker complaining about unresolved references
Not trying to print anything and calling Sleep, which resulted in the process sleeping indefinitely
Exiting with a different return code, which once again did nothing
What am I doing wrong?
EDIT: Fixed calling convention
There are three problems with your revised code. The first is:
message: db 'Hello world!',10 ; 'Hello world!' plus a linefeed character
messageLen: db $-message ; Length of the 'Hello world!' string
You defined messageLen to be a byte containing the length of the message and storing that value at the address of messageLen. You then do this:
mov r8, messageLen
That would move the address of label messageLen to r8. What you really should have done is define messageLen as an assembly time constant like this:
messageLen equ $-message ; Length of the 'Hello world!' string
The second problem is that you define the the string as a sequence of single byte characters:
message: db 'Hello world!',10 ; 'Hello world!' plus a linefeed character
There is nothing wrong with this, but to print them out you need to use the Ansi version of the function WriteConsole which is WriteConsoleA. Using WriteConsoleW printed the string as Unicode (UTF-16 on Windows 2000 and later, UTS-2 on NT4 and earlier versions of Windows).
The third problem is with regards to a mandatory 32 bytes of shadow space before the stack based parameter(s) are placed on the stack before making a function call. You also need to make sure the stack (RSP) is a 16-byte aligned value at the point of making a function call. These requirement can be found in the Microsoft 64-bit calling convention.
Code that would take this into account would look like this:
section .data
message: db 'Hello world!',10 ; 'Hello world!' plus a linefeed character
messageLen equ $-message ; Length of the 'Hello world!' string
global _start
extern GetStdHandle
extern WriteConsoleA
extern ExitProcess
section .text
_start:
; At _start the stack is 8 bytes misaligned because there is a return
; address to the MSVCRT runtime library on the stack.
; 8 bytes of temporary storage for `bytes`.
; allocate 32 bytes of stack for shadow space.
; 8 bytes for the 5th parameter of WriteConsole.
; An additional 8 bytes for padding to make RSP 16 byte aligned.
sub rsp, 8+8+8+32
; At this point RSP is aligned on a 16 byte boundary and all necessary
; space has been allocated.
; hStdOut = GetStdHandle(STD_OUTPUT_HANDLE)
mov ecx, -11
call GetStdHandle
; WriteFile(hstdOut, message, length(message), &bytes, 0);
mov rcx, rax
mov rdx, message
mov r8, messageLen
lea r9, [rsp-16] ; Address for `bytes`
; RSP-17 through RSP-48 are the 32 bytes of shadow space
mov qword [rsp-56], 0 ; First stack parameter of WriteConsoleA function
call WriteConsoleA
; ExitProcess(0)
; mov rcx, 0
; call ExitProcess
; alternatively you can exit by setting RAX to 0
; and doing a ret
add rsp, 8+8+32+8 ; Restore the stack pointer.
xor eax, eax ; RAX = return value = 0
ret

Push an argument into stack?

I know that the first four arguments are in the register (RCX, RDX, R8, R9), and that additional arguments are pushed on the stack.
Question:
How to push an argument onto the stack? I tried with (push 0) but it does not work?
Code (MASM64)
extrn ExitProcess: PROC
extrn MessageBoxExA: PROC
.data
caption db '64-bit hello!', 0
message db 'Hello World!', 0
.code
Start PROC
sub rsp, 38h
mov rcx, 0 ; hWnd = HWND_DESKTOP
lea rdx, message ; LPCSTR lpText
lea r8, caption ; LPCSTR lpCaption
mov r9d, 0 ; uType = MB_OK
push 0 ; wLanguageId
call MessageBoxExA
mov ecx, eax
add rsp, 38h
call ExitProcess
Start ENDP
End
I'm know that MessageBox and MessageBoxEx work the same way, but im trying to use MessageBoxEx because its need one parameter to be passed (for learning purpose).
I know I've asked similar question, but it is more related to vb.net while this is not.
My assembly is a little rusty, but I was under the impression that all arguments went onto the stack (in reverse order) - I'd have thought you want to be pushing r8 and rdx in as well as the other arguments. Frankly though you might as well just keep doing lea rax, param and push rax for each of the arguments that are pointers.
The order in which the arguments are passed and whether they are passed in registers or on the stack (along with whether caller or callee is responsible for cleanup) is defined by the 'Calling Convention'.
What you are probably thinking of is STDCALL or CDECL, both are calling conventions used in 32-bit Windows that pass arguments on the stack in reverse order (right to left). x64 has moved to a FastCall calling convention where the arguments are passed in forward order (from left to right) and the first 4 arguments are passed in the registers RCX, RDX, R8 & R9. Any arguments beyond 4 are passed on the stack in the same left-to-right order. The original poster had the correct calling convention setup for x64 assembly with MASM. Also, the above responder who said the shadowspace valued subtracted from RSP should be 20h (32d) is correct. The shadow space is allowing space on the stack for the 4 arguments that are passed in by the registers in FastCall.
Changing the code above to:
extrn ExitProcess: PROC
extrn MessageBoxExA: PROC
.data
caption db '64-bit hello!', 0
message db 'Hello World!', 0
.code
Start PROC
sub rsp, 20h
mov rcx, 0 ; hWnd = HWND_DESKTOP
lea rdx, message ; LPCSTR lpText
lea r8, caption ; LPCSTR lpCaption
mov r9d, 0 ; uType = MB_OK
push 0 ; wLanguageId
call MessageBoxExA
mov ecx, eax
add rsp, 20h
call ExitProcess
Start ENDP
End
Works just fine in Visual Studio on a 64-bit machine

Resources