64-bit Program - Windows "Shadow Space" Trouble - windows

I am trying to create a program in x64 assembly language but I am having problems understanding the x64 calling convention. I believe that the problem is that I do not know how much shadow space I have to reserve for the call to the CopyFile function. When, I run the program, it just crashes. I created this program using MASM. Please help me fix this code. Thank you.
includelib \Masm64\Lib\Kernel32.lib
includelib \Masm64\Lib\User32.lib
extrn GetProcessHeap : proc
extrn MessageBoxA : proc
extrn HeapAlloc : proc
extrn GetModuleFileNameA : proc
extrn ExitProcess : proc
extrn CopyFileA : proc
dseg segment para 'DATA'
file db 'C:\CopyThisFile.txt', 0
file2 db 'C:\ThisFileWasCopied.txt', 0
succ db 'Success!', 0
capt db 'Debug', 0
dseg ends
cseg segment para 'CODE'
start proc
sub rsp, 28h
xor r8, r8
mov rdx, qword ptr file2
mov rcx, qword ptr file
call CopyFileA
xor ecx, ecx
call ExitProcess
start endp
cseg ends
end

This has nothing to do with space reservation on the stack.
Your mistake instead lies in getting the string's address incorrectly. mov gets the contents (first 8 bytes) instead of the pointer to the string, hence raises a AccessViolation exception. To fix this, use lea.
format PE64 GUI 5.0
entry start
include 'WIN64A.INC'
section '.data' data readable writeable
fileStr db 'C:\\CopyThisFile.txt', 0
file2Str db 'C:\\ThisFileWasCopied.txt', 0
succ db 'Success!', 0
section '.text' code readable executable
start:
sub rsp, 28
xor r8, r8
lea rdx, qword ptr file2Str
lea rcx, qword ptr fileStr
call [CopyFileA]
xor ecx, ecx
call [ExitProcess]
section '.idata' import data readable
library kernel32,'kernel32.dll',user32,'user32.dll'
import kernel32, \
GetProcessHeap,'GetProcessHeap', \
HeapAlloc,'HeapAlloc', \
GetModuleFileNameA,'GetModuleFileNameA', \
ExitProcess,'ExitProcess', \
CopyFileA,'CopyFileA'
import user32, \
MessageBoxA,'MessageBoxA'

Related

FASM write Hello World to console with NO includes or dependencies at all

I've seen
How to write hello world in assembler under Windows?
and
Writing hello,world to console in Fasm with DOS
How to write to the console in fasm?
I've tried / seen code like this MASM example from this answer
;---ASM Hello World Win64 MessageBox
extrn MessageBoxA: PROC
extrn ExitProcess: PROC
.data
title db 'Win64', 0
msg db 'Hello World!', 0
.code
main proc
sub rsp, 28h
mov rcx, 0 ; hWnd = HWND_DESKTOP
lea rdx, msg ; LPCSTR lpText
lea r8, title ; LPCSTR lpCaption
mov r9d, 0 ; uType = MB_OK
call MessageBoxA
add rsp, 28h
mov ecx, eax ; uExitCode = MessageBox(...)
call ExitProcess
main endp
End
(to which I get an error "Illegal instruction" on windows 64 bit extrn MessageBoxA:PROC because FASM doesn't understand that MASM directive.)
also this FASM example from this question
; Example of 64-bit PE program
format PE64 GUI
entry start
section '.text' code readable executable
start:
sub rsp,8*5 ; reserve stack for API use and make stack dqword aligned
mov r9d,0
lea r8,[_caption]
lea rdx,[_message]
mov rcx,0
call [MessageBoxA]
mov ecx,eax
call [ExitProcess]
section '.data' data readable writeable
_caption db 'Win64 assembly program',0
_message db 'Hello World!',0
section '.idata' import data readable writeable
dd 0,0,0,RVA kernel_name,RVA kernel_table
dd 0,0,0,RVA user_name,RVA user_table
dd 0,0,0,0,0
kernel_table:
ExitProcess dq RVA _ExitProcess
dq 0
user_table:
MessageBoxA dq RVA _MessageBoxA
dq 0
kernel_name db 'KERNEL32.DLL',0
user_name db 'USER32.DLL',0
_ExitProcess dw 0
db 'ExitProcess',0
_MessageBoxA dw 0
db 'MessageBoxA',0
but it displays a message box and also has external dependencies "kernel32.dll" and "user32.dll"
also tried this example from the FASM forum
format pe console
include 'win32ax.inc'
entry main
section '.data!!!' data readable writeable
strHello db 'Hello World !',13,10,0
strPause db 'pause',0
section '.txt' code executable readable
main:
; you can use crt functions or windows API.
cinvoke printf,strHello
cinvoke system,strPause; or import getc()
; or
; invoke printf,srtHello
; add esp, 4
; or use WriteFile and GetStdHandle APIs
push 0
call [ExitProcess]
section '.blah' import data readable
library kernel32,'kernel32.dll',\
msvcrt,'msvcrt.dll' ;; C-Run time from MS. This is always on every windows machine
import kernel32,\
ExitProcess,'ExitProcess'
import msvcrt,\
printf,'printf',\
system,'system'
but it depends on win32ax.inc and other imports
also
format PE console
include 'win32ax.inc'
.code
start:
invoke WriteConsole,<invoke GetStdHandle,STD_OUTPUT_HANDLE>,"Hello World !",13,0
invoke Sleep,-1
.end start
but requires "win32ax.inc" import
closest I could find without the win32ax from the FASM forum:
format pe64 console
entry start
STD_OUTPUT_HANDLE = -11
section '.text' code readable executable
start:
sub rsp,8*7 ; reserve stack for API use and make stack dqword aligned
mov rcx,STD_OUTPUT_HANDLE
call [GetStdHandle]
mov rcx,rax
lea rdx,[message]
mov r8d,message_length
lea r9,[rsp+4*8]
mov qword[rsp+4*8],0
call [WriteFile]
mov ecx,eax
call [ExitProcess]
section '.data' data readable writeable
message db 'Hello World!',0
message_length = $ - message
section '.idata' import data readable writeable
dd 0,0,0,RVA kernel_name,RVA kernel_table
dd 0,0,0,0,0
kernel_table:
ExitProcess dq RVA _ExitProcess
GetStdHandle dq RVA _GetStdHandle
WriteFile dq RVA _WriteFile
dq 0
kernel_name db 'KERNEL32.DLL',0
user_name db 'USER32.DLL',0
_ExitProcess db 0,0,'ExitProcess',0
_GetStdHandle db 0,0,'GetStdHandle',0
_WriteFile db 0,0,'WriteFile',0
but still requires the kernel32.dll and user32.dll
Any way to do this without any external DLLs at all? I know just the program fasm itself does it, and prints to the console, doesn't it?
Any way to do this without any external DLLs at all?
Under Windows: Definitely no!
Windows uses some methods (probably syscall) to enter the operating system, however, there are no official entry points.
This means that it is (unlikely but) possible that exactly the same program that shows the "Hello world" message box in the current Windows version will do something completely different after the next Windows update!
Because Microsoft is assuming that every Windows program is only calling the OS by using the .dll files that match the kernel version, they can do this.
I don't know about Windows 10, but an older Windows version (I don't remember if it was XP, Vista or 7) even simply assumed that an .exe file returns at once if it does not use any .dll file: The program was not even started in this case!
I know just the program fasm itself does it, and prints to the console
That is not the case, fasm is also using the kernel32 APIs.
FWIW kernel32 is loaded into the memory space of every process in Windows, so there is no penalty or overhead in using the kernel32 APIs.
You may like this Windows example in €ASM, which doesn't explicitly mention any DLL and doesn't require other external libraries.
Just save the source as "bluej.asm", assemble and link with euroasm bluej.asm and run as bluej.exe.
Nevertheless, you won't get away without using API functions imported from the default Windows system library "kernel32.dll".
bluej PROGRAM Format=PE, Entry=Start:
IMPORT GetStdHandle,WriteFile,ExitProcess
Start: PUSH -11 ; Param 1: standard output handle identificator.
CALL GetStdHandle; Return StdOutput handle in EAX.
PUSH 0 ; Param 5: no overlap.
PUSH Written ; Param 4: Address of a variable to store number of written bytes.
PUSH MsgSize ; Param 3: Number of bytes to write.
PUSH Msg ; Param 2: Address of text.
PUSH EAX ; Param 1: Output file handle.
CALL WriteFile ; System call.
PUSH 0 ; Errorlevel.
CALL ExitProcess ; System call.
Written DD 0
Msg DB "Hello, world!"
MsgSize EQU $ - Msg
ENDPROGRAM
What constitures as "dependency" to you? If you want to avoid even operating system DLL's, then you're probably out of luck. You can't rely on syscall numbers alone.
"no dependencies" can also mean "just using existing OS DLL's", such as ntdll, kernel32, etc., but without using 3rd party DLL's that may not be present, such as a specific version of the C runtime.
One method I would like to show is retrieving function pointers from the PEB. This is code that I've written and that I personally use, if I want to have shellcode that has no import section.
PebGetProcAddress works similarly to GetProcAddress, except that the DLL name and function name must be a hash, and the DLL must be loaded by using LoadLibrary.
This may not answer your question exactly, but I hope it gets you somewhat closer to your goal or help others who read it.
PebApi.asm
proc PebGetProcAddress ModuleHash:DWORD, FunctionHash:DWORD
local FirstEntry:DWORD
local CurrentEntry:DWORD
local ModuleBase:DWORD
local ExportDirectory:DWORD
local NameDirectory:DWORD
local NameOrdinalDirectory:DWORD
local FunctionCounter:DWORD
; Get InMemoryOrderModuleList from PEB
mov eax, 3
shl eax, 4
mov eax, [fs:eax] ; fs:0x30
mov eax, [eax + PEB.Ldr]
mov eax, [eax + PEB_LDR_DATA.InMemoryOrderModuleList.Flink]
mov [FirstEntry], eax
mov [CurrentEntry], eax
; Find module by hash
.L_module:
; Compute hash of case insensitive module name
xor edx, edx
mov eax, [CurrentEntry]
movzx ecx, word[eax + LDR_DATA_TABLE_ENTRY.BaseDllName.Length]
test ecx, ecx
jz .C_module
mov esi, [eax + LDR_DATA_TABLE_ENTRY.BaseDllName.Buffer]
xor eax, eax
cld
.L_module_hash:
lodsb
ror edx, 13
add edx, eax
cmp al, 'a'
jl #f
sub edx, 0x20 ; Convert lower case letters to upper case
##: dec ecx
test ecx, ecx
jnz .L_module_hash
; Check, if module is found by hash
cmp edx, [ModuleHash]
jne .C_module
; Get module base
mov eax, [CurrentEntry]
mov eax, [eax + LDR_DATA_TABLE_ENTRY.DllBase]
mov [ModuleBase], eax
; Get export directory
mov eax, [ModuleBase]
add eax, [eax + IMAGE_DOS_HEADER.e_lfanew]
mov eax, [eax + IMAGE_NT_HEADERS32.OptionalHeader.DataDirectoryExport.VirtualAddress]
add eax, [ModuleBase]
mov [ExportDirectory], eax
; Get name table
mov eax, [ExportDirectory]
mov eax, [eax + IMAGE_EXPORT_DIRECTORY.AddressOfNames]
add eax, [ModuleBase]
mov [NameDirectory], eax
; Get name ordinal table
mov eax, [ExportDirectory]
mov eax, [eax + IMAGE_EXPORT_DIRECTORY.AddressOfNameOrdinals]
add eax, [ModuleBase]
mov [NameOrdinalDirectory], eax
; Find function in export directory by hash
mov [FunctionCounter], 0
.L_functions:
mov eax, [ExportDirectory]
mov eax, [eax + IMAGE_EXPORT_DIRECTORY.NumberOfNames]
cmp eax, [FunctionCounter]
je .E_functions
; Compute hash of function name
xor edx, edx
mov esi, [NameDirectory]
mov esi, [esi]
add esi, [ModuleBase]
xor eax, eax
cld
.L_function_hash:
lodsb
test al, al
jz .E_function_hash
ror edx, 13
add edx, eax
jmp .L_function_hash
.E_function_hash:
; Check, if function is found by hash
cmp edx, [FunctionHash]
jne .C_functions
; Return function address
mov eax, [ExportDirectory]
mov eax, [eax + IMAGE_EXPORT_DIRECTORY.AddressOfFunctions]
add eax, [ModuleBase]
mov ebx, [NameOrdinalDirectory]
movzx ebx, word[ebx]
lea eax, [eax + ebx * 4]
mov eax, [eax]
add eax, [ModuleBase]
ret
.C_functions:
add [NameDirectory], 4
add [NameOrdinalDirectory], 2
inc [FunctionCounter]
jmp .L_functions
.E_functions:
; Function not found in module's export table
xor eax, eax
ret
.C_module:
; Move to next module, exit loop if CurrentEntry == FirstEntry
mov eax, [CurrentEntry]
mov eax, [eax + LIST_ENTRY.Flink]
mov [CurrentEntry], eax
cmp eax, [FirstEntry]
jne .L_module
; Module not found
xor eax, eax
ret
endp
PebApi.inc
macro pebcall modulehash, functionhash, [arg]
{
common
if ~ arg eq
reverse
pushd arg
common
end if
stdcall PebGetProcAddress, modulehash, functionhash
call eax
}
Example
PEB_User32Dll = 0x63c84283
PEB_MessageBoxW = 0xbc4da2be
; pebcall translates to a call to PebGetProcAddress and the call to the returned function pointer
pebcall PEB_User32Dll, PEB_MessageBoxW, NULL, 'Hello, World!', NULL, MB_OK
How to generate hashes for module names and function names
#define ROTR(value, bits) ((DWORD)(value) >> (bits) | (DWORD)(value) << (32 - (bits)))
DWORD ComputeFunctionHash(LPCSTR str)
{
DWORD hash = 0;
while (*str)
{
hash = ROTR(hash, 13) + *str++;
}
return hash;
}
DWORD ComputeModuleNameHash(LPCSTR str, USHORT length)
{
DWORD hash = 0;
for (USHORT i = 0; i < length; i++)
{
hash = ROTR(hash, 13) + (str[i] >= 'a' ? str[i] - 0x20 : str[i]);
}
return hash;
}

How to fix: (cannot have implicit far jump or call to near label) and (use a register assumed to ERROR)

I'm trying to create dll using VS 2017.
The dll will have one proc: symbol_count.
It asks to enter the string and then set symbol what is needed to count.
.def file
LIBRARY name
EXPORTS
symbol_count
Code:
.586
.model flat, stdcall
option casemap: none
include C:\masm32\include\windows.inc
include C:\masm32\include\user32.inc
include C:\masm32\include\msvcrt.inc
includelib C:\masm32\lib\msvcrt.lib
includelib C:\masm32\lib\user32.lib
.data
msg_string db 'Enter string: ', 0
msg_symbol db 'Enter symbol: ', 0
result db 'Count = %d', 0
str_modifier db '%s', 0
sym_modifier db '%c', 0
.data
string db ?
symbol db ?
DllEntry PROC hInstDLL:DWORD, reason:DWORD, reserved:DWORD
mov eax, 1
ret
DllEntry ENDP
symbol_count PROC
invoke crt_printf, OFFSET msg_string
invoke crt_scanf, OFFSET str_modifier, OFFSET string
invoke crt_printf, OFFSET msg_symbol
invoke crt_scanf, OFFSET sym_modifier, OFFSET symbol
xor esi, esi
xor ecx, ecx
mov ebx, OFFSET string
mov ecx, eax
mov al, symbol
loop1: <------------------------------------------ A2108
cmp byte ptr [ebx + ecx], 0
je endloop <------------------------------ A2107
cmp al, byte ptr [ebx + ecx]
jne next <-------------------------------- A2107
inc esi
next: <------------------------------------------- A2108
inc ecx
jmp loop1 <------------------------------- A2107
endloop: <---------------------------------------- A2108
invoke crt_printf, OFFSET result, esi
ret
symbol_count ENDP
End DllEntry
Here is the list of error messages, what a compiler gives to me: (
in the code, I marked the places where the compiler swears)
A2108 use of register assumed to ERROR
A2108 use of register assumed to ERROR
A2108 use of register assumed to ERROR
A2107 cannot have implicit far jump or call to near label
A2107 cannot have implicit far jump or call to near label
A2107 cannot have implicit far jump or call to near label
procedure argument or local not referenced : hInstDLL } all this points
procedure argument or local not referenced : reason } to DllEntry ENDP
procedure argument or local not referenced : reserved }
"You put your code into the .data section which may or may not cause some of the errors. The last 3 should just be warnings as you don't use the arguments." – #Jester

MASM console window creation troubles (maybe my stack frame??)

I'm having trouble with the following code snippet.
I'm compiling an asm file with a c++ file using visual studio express 2012.
I've tried debugging calculating the addresses of the stack pointers and memory locations (still learning assembly obviously), but can't find a fault. (As I'm new it's probably something obvious, but not obvious enough). The asm part is just a function called from the c++ file that is supposed to open a console window and print something in it. The SetConsoleTitleA works fine, it trips on the Writefile function. Here is the error itself :
First-chance exception at 0x00007FF8551A5B48 (KernelBase.dll) in
Troy2.exe: 0xC0000005: Access violation writing location
0x00007FF700000000.
Unhandled exception at 0x00007FF8551A5B48 (KernelBase.dll) in Troy2.exe:
0xC0000005:
Access violation writing location 0x00007FF700000000.
And the code is here :
extrn GetStdHandle: PROC
extrn WriteFile: PROC
extrn AllocConsole: PROC
extrn FreeConsole: PROC
extrn SetConsoleTitleA: PROC
extrn SetConsoleCursorPosition: PROC
extrn Sleep: PROC
extrn ExitProcess: PROC
extrn MessageBoxA:PROC
.data
consoletitle db 'Console', 0
prompt db 'Oronco-> ', 0
handle dd ?
.code
myfunction proc
call AllocConsole
sub rsp, 20h
lea rcx, consoletitle
call SetConsoleTitleA
add rsp, 20h
sub rsp, 20h
mov rcx, -11
call GetStdHandle
add rsp, 20h
mov handle, eax
sub rsp, 28h
lea rcx, handle
lea rdx, prompt
mov r8, SIZEOF prompt
mov r9, 0
mov dword ptr [rsp + 20h], 0
call WriteFile
add rsp, 28h
mov rcx, 2000
call Sleep
Call ExitProcess
myfunction endp
End
mov dword ptr [rsp + 20h], 0
is wrong. The last parameter has the type LPOVERLAPPED, which is here a 64-bit pointer. Change the line to
mov qword ptr [rsp + 20h], 0
Also,
lea rcx, handle
is wrong. WriteFile expects a value, not an address (pointer). Change it to
mov ecx, handle

Strange behaviour with a simple MASM32 program

I want to write a MASM program similar to the following C++ program :
#include <Windows.h>
#include <iostream>
typedef UINT (_stdcall *FuncPtr)(LPCSTR lpCmdLine, UINT uCmdShow);
int main(void)
{
HMODULE hDll = LoadLibrary(TEXT("Kernel32.dll"));
FuncPtr func_addr = reinterpret_cast<FuncPtr>(GetProcAddress(hDll, "WinExec"));
(*func_addr)("C:\\WINDOWS\\system32\\calc.exe", SW_SHOWDEFAULT);
FreeLibrary(hDll);
return (0);
}
As you can see, this code execute the microsoft calculator. I just want to do the same thing using MASM but the execution fails.
Here's the MASM source code :
.386
.model flat, stdcall
option casemap:none
include \masm32\include\windows.inc
include \masm32\include\user32.inc
include \masm32\include\kernel32.inc
include \masm32\include\masm32.inc
include \masm32\include\msvcrt.inc
includelib \masm32\lib\user32.lib
includelib \masm32\lib\kernel32.lib
includelib \masm32\lib\masm32.lib
includelib \masm32\lib\msvcrt.lib
.data
LpFileName db "kernel32.dll", 0
procName db "WinExec", 0
display db "addr_func = 0x%x", 0
.data?
hModule HMODULE ?
procAddr FARPROC ?
.code
start:
invoke LoadLibrary, offset LpFileName
mov hModule, eax
invoke GetProcAddress, hModule, ADDR procName
mov procAddr, eax
INVOKE crt_printf, ADDR display, procAddr
mov esi, procAddr
call esi
db "C:\WINDOWS\system32\calc.exe"
invoke FreeLibrary, hModule
invoke ExitProcess, NULL
end start
The crt_printf output is correct. The same address is printed like withe the C++ program. So the address passed to call is the same one. However the execution fails.
Here's a MASM32 code which works but this time the address of the function WinExec is hardcoded like this :
.386
.model flat, stdcall
option casemap:none
include \masm32\include\windows.inc
.code
start:
jmp _Debut
_Final:
TCHAR 233
dword 42424242h
_Suite:
mov esi, 779e304eh
call esi
jmp _Final
_Debut:
xor eax, eax
push eax
call _Suite
db "C:\WINDOWS\system32\calc.exe"
end start
See the line mov esi, 779e304eh. But dynamically, there is a problem. If I disassemble the code just above we can see that the order of bytes is reversed.
8EEH047E379
Maybe it's not the case dynamically and maybe I need a keyword in the following line (between the comma and procAddr):
mov esi, procAddr
I cannot find the solution. I'm lost. Can anyone help me?
Thanks a lot in advance for your help.
The execution fails because you are not passing it's parameters.
Here you just call the function without any arguments or rather with invalid arguments (because whatever is currently on the stack will be taken and the stack is corrupted in the process).
mov esi, procAddr
call esi
You should do
push SW_SHOWDEFAULT
push offset YourPathToCalc
mov esi, procAddr
call esi
In your samplecode this is what is done here implicitly
xor eax, eax
push eax ; uCmdShow
call _Suite ; Returnadress is the address of the commandline so this is bascially the "push path"
Another thing you are missing is, that when WinExec returns, it will start executing the path in your case so you need a jmp somewhere after the call.
And as Gunner pointed out, the path must be 0 terminated.
To add to the correct answer Devolus posted, your path to calc is not NULL terminated.
This
"C:\WINDOWS\system32\calc.exe" is not correct!
Instead it should be:
"C:\WINDOWS\system32\calc.exe", 0
Also, if you are going to put strings in the code section to use, you need to give them a label in order to use them and you need to jump over them otherwise the CPU will try to execute the bytes.
INVOKE crt_printf, ADDR display, procAddr
mov esi, procAddr
push SW_SHOWDEFAULT
push offset Calc
call esi
jmp #F
Calc db "C:\WINDOWS\system32\calc.exe", 0
##:
invoke FreeLibrary, hModule
invoke ExitProcess, NULL
* EDIT *
To convert that code to Assembly using MASM, all that is needed is this:
.data
LpFileName db "kernel32", 0
procName db "WinExec", 0
Calc db "calc", 0
.code
start:
invoke LoadLibrary, offset LpFileName
push eax
invoke GetProcAddress, eax, ADDR procName
push SW_SHOWDEFAULT
push offset Calc
call eax
call FreeLibrary
invoke ExitProcess, NULL
end start

Push an argument into stack?

I know that the first four arguments are in the register (RCX, RDX, R8, R9), and that additional arguments are pushed on the stack.
Question:
How to push an argument onto the stack? I tried with (push 0) but it does not work?
Code (MASM64)
extrn ExitProcess: PROC
extrn MessageBoxExA: PROC
.data
caption db '64-bit hello!', 0
message db 'Hello World!', 0
.code
Start PROC
sub rsp, 38h
mov rcx, 0 ; hWnd = HWND_DESKTOP
lea rdx, message ; LPCSTR lpText
lea r8, caption ; LPCSTR lpCaption
mov r9d, 0 ; uType = MB_OK
push 0 ; wLanguageId
call MessageBoxExA
mov ecx, eax
add rsp, 38h
call ExitProcess
Start ENDP
End
I'm know that MessageBox and MessageBoxEx work the same way, but im trying to use MessageBoxEx because its need one parameter to be passed (for learning purpose).
I know I've asked similar question, but it is more related to vb.net while this is not.
My assembly is a little rusty, but I was under the impression that all arguments went onto the stack (in reverse order) - I'd have thought you want to be pushing r8 and rdx in as well as the other arguments. Frankly though you might as well just keep doing lea rax, param and push rax for each of the arguments that are pointers.
The order in which the arguments are passed and whether they are passed in registers or on the stack (along with whether caller or callee is responsible for cleanup) is defined by the 'Calling Convention'.
What you are probably thinking of is STDCALL or CDECL, both are calling conventions used in 32-bit Windows that pass arguments on the stack in reverse order (right to left). x64 has moved to a FastCall calling convention where the arguments are passed in forward order (from left to right) and the first 4 arguments are passed in the registers RCX, RDX, R8 & R9. Any arguments beyond 4 are passed on the stack in the same left-to-right order. The original poster had the correct calling convention setup for x64 assembly with MASM. Also, the above responder who said the shadowspace valued subtracted from RSP should be 20h (32d) is correct. The shadow space is allowing space on the stack for the 4 arguments that are passed in by the registers in FastCall.
Changing the code above to:
extrn ExitProcess: PROC
extrn MessageBoxExA: PROC
.data
caption db '64-bit hello!', 0
message db 'Hello World!', 0
.code
Start PROC
sub rsp, 20h
mov rcx, 0 ; hWnd = HWND_DESKTOP
lea rdx, message ; LPCSTR lpText
lea r8, caption ; LPCSTR lpCaption
mov r9d, 0 ; uType = MB_OK
push 0 ; wLanguageId
call MessageBoxExA
mov ecx, eax
add rsp, 20h
call ExitProcess
Start ENDP
End
Works just fine in Visual Studio on a 64-bit machine

Resources