Hello world in NASM with LINK.EXE and WinAPI - winapi

Question

I'm trying to get a simple Hello world program in NASM to run.
I want to print to the console without using C-Libraries, interfacing directly with WinAPI.
I am using the Visual Studio provided LINK.EXE for linking.
Here's my code so far:
section .data
message: db 'Hello world!',10 ; 'Hello world!' plus a linefeed character
messageLen: db $-message ; Length of the 'Hello world!' string
global _start
extern GetStdHandle
extern WriteConsoleW
extern ExitProcess
section .text
_start:
; DWORD bytes;
mov rbp, rsp
sub rsp, byte 8
; hStdOut = GetStdHandle(STD_OUTPUT_HANDLE)
mov ecx, -11
call GetStdHandle
; WriteFile(hstdOut, message, length(message), &bytes, 0);
mov rcx, rax
mov rdx, message
mov r8, messageLen
lea r9, [rsp-4]
push 0
call WriteConsoleW
; ExitProcess(0)
mov rcx, 0
call ExitProcess
ret
Which I assemble and link like this:
nasm -f win64 .\ASM.ASM
link /entry:_start /nodefaultlib /subsystem:console .\ASM.obj "C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\um\x64\kernel32.lib" "C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\um\x64\user32.lib"
However when I run the resulting .exe file, I get nothing.
Some things I tried so far are
Using the decorated names (like _GetStdHandle#4), which resulted in the linker complaining about unresolved references
Not trying to print anything and calling Sleep, which resulted in the process sleeping indefinitely
Exiting with a different return code, which once again did nothing
What am I doing wrong?
EDIT: Fixed calling convention

Answer 1

There are three problems with your revised code. The first is:
message: db 'Hello world!',10 ; 'Hello world!' plus a linefeed character
messageLen: db $-message ; Length of the 'Hello world!' string
You defined messageLen to be a byte containing the length of the message and storing that value at the address of messageLen. You then do this:
mov r8, messageLen
That would move the address of label messageLen to r8. What you really should have done is define messageLen as an assembly time constant like this:
messageLen equ $-message ; Length of the 'Hello world!' string
The second problem is that you define the the string as a sequence of single byte characters:
message: db 'Hello world!',10 ; 'Hello world!' plus a linefeed character
There is nothing wrong with this, but to print them out you need to use the Ansi version of the function WriteConsole which is WriteConsoleA. Using WriteConsoleW printed the string as Unicode (UTF-16 on Windows 2000 and later, UTS-2 on NT4 and earlier versions of Windows).
The third problem is with regards to a mandatory 32 bytes of shadow space before the stack based parameter(s) are placed on the stack before making a function call. You also need to make sure the stack (RSP) is a 16-byte aligned value at the point of making a function call. These requirement can be found in the Microsoft 64-bit calling convention.
Code that would take this into account would look like this:
section .data
message: db 'Hello world!',10 ; 'Hello world!' plus a linefeed character
messageLen equ $-message ; Length of the 'Hello world!' string
global _start
extern GetStdHandle
extern WriteConsoleA
extern ExitProcess
section .text
_start:
; At _start the stack is 8 bytes misaligned because there is a return
; address to the MSVCRT runtime library on the stack.
; 8 bytes of temporary storage for `bytes`.
; allocate 32 bytes of stack for shadow space.
; 8 bytes for the 5th parameter of WriteConsole.
; An additional 8 bytes for padding to make RSP 16 byte aligned.
sub rsp, 8+8+8+32
; At this point RSP is aligned on a 16 byte boundary and all necessary
; space has been allocated.
; hStdOut = GetStdHandle(STD_OUTPUT_HANDLE)
mov ecx, -11
call GetStdHandle
; WriteFile(hstdOut, message, length(message), &bytes, 0);
mov rcx, rax
mov rdx, message
mov r8, messageLen
lea r9, [rsp-16] ; Address for `bytes`
; RSP-17 through RSP-48 are the 32 bytes of shadow space
mov qword [rsp-56], 0 ; First stack parameter of WriteConsoleA function
call WriteConsoleA
; ExitProcess(0)
; mov rcx, 0
; call ExitProcess
; alternatively you can exit by setting RAX to 0
; and doing a ret
add rsp, 8+8+32+8 ; Restore the stack pointer.
xor eax, eax ; RAX = return value = 0
ret