General structure for executing system commands from x86-64 assembly (NASM)? - macos

Question

I am trying to make some basic system calls in assembly (x86-64 in NASM on OSX), but have so far been unsuccessful.
The only examples I have seen on the web so far are for reading from stdin or writing to stdout, such as this:
global main
section .text
main:
call write
write:
mov rax, 0x2000004
mov rdi, 1
mov rsi, message
mov rdx, length
syscall
section .data
message: db 'Hello, world!', 0xa
length: equ $ - message
However, when I try to use that same pattern to make another system call, it doesn't work (it's saying Bus error: 10):
global main
section .text
main:
call mkdir
mkdir:
mov rax, 0x2000136 ; mkdir system command number
mov rdi, rax ; point destination to system command
mov rsi, directory ; first argument
mov rdx, 755 ; second argument
syscall
section .data
directory: db 'tmp', 0xa
What is the general structure for calling system commands (on OSX in NASM ideally)?
Basically what it seems like you're supposed to do is find your desired system call in here: http://www.opensource.apple.com/source/xnu/xnu-1504.3.12/bsd/kern/syscalls.master. So the "write" one looks like this:
4 AUE_NULL ALL { user_ssize_t write(int fd, user_addr_t cbuf, user_size_t nbyte); }
That is saying:
system call number: 4
number of arguments: 3 (file descriptor, memory address to string/buffer, length of buffer)
So I was beginning to think the general pattern was this:
rax: system call number
rdi: maybe? point to system call ("destination index"), but why the `1` in the write example?
rsi: first argument to system call ("source index", the string in this case)
rdx: second argument to system call
rcx: third argument (if necessary, but not in the system write case)
So then it's like you could do a direct mapping of any of the system commands. So mkdir:
136 AUE_MKDIR ALL { int mkdir(user_addr_t path, int mode); }
would be translated to:
rax: 0x20000136 ; 136 + 20000000
rdi: i dunno, maybe `rax`?
rsi: directory (first argument)
rdx: 755 (mode, second argument)
But yeah, that doesn't work.
What am I doing wrong? What is the general pattern of how to do this so I can test it out on any of the other system commands in syscalls.master? Can you describe the role the different registers play here too? That would help clarify a lot I think.

Answer 1

I believe OSX is following the standard SYSV ABI calling convention, at least your example certainly looks like that. Arguments go in the registers RDI, RSI, RDX, R10, R8, and R9, in order. System call number goes into RAX.
Let's look at write: int fd, user_addr_t cbuf, user_size_t nbyte
The assembly:
mov rdi, 1 ; fd = 1 = stdout
mov rsi, message ; cbuf
mov rdx, length ; nbyte
Now, for mkdir: user_addr_t path, int mode
Obviously you need to put path into rdi and mode into rsi.
mkdir:
mov rax, 0x2000136 ; mkdir system command number
mov rdi, directory ; first argument
mov rsi, 0x1ED ; second argument, 0x1ED = 755 octal
syscall
ret
Note you need ret and the end of mkdir subroutine, and you also need one so your main doesn't fall through into mkdir. Furthermore, you should probably use lea to load the directory argument, and use RIP-relative addressing, such as lea rdi, [rel directory].

Answer 2

You've got it almost right: You need 0x88 (dec 136) for the syscall number. The syscalls in syscall.master are in decimal. You ended up calling getsid (which is syscall 310).
For arguments, don't use syscalls.master since that gives you the kernel perspective which is a tad skewed (when it comes to argument names). You should use /usr/include/unistd.h for the prototypes, and usr/inclunde/sys/syscall.h for the numbers. syscalls.master comes in handy only in cases where the syscalls aren't exported to these files, and those are cases where the master files says NO_SYSCALL_STUB.
As for the ABI, it's the same as System V AMD64 ABI. http://people.freebsd.org/~obrien/amd64-elf-abi.pdf
You can see the system calls as libsystem does them:
otool -tV /usr/lib/system/libsystem_kernel.dylib | more
# seek to /^_mkdir:
_mkdir:
0000000000012dfc movl $0x2000088, %eax
0000000000012e01 movq %rcx, %r10
0000000000012e04 syscall
0000000000012e06 jae 0x12e0d
0000000000012e08 jmpq cerror_nocancel
0000000000012e0d ret
0000000000012e0e nop
0000000000012e0f nop
All the system calls essentially have the structure:
Arguments by this point have been put in RDI,RSI,... as per above
ABI
The system call # is loaded into EAX. The 0x2 implies POSIX
syscall. 0x1 would be a Mach Trap, 0x3 - arch specific, 0x4 -
Diagnostic syscalls
rcx saved into r10
syscall gets executed
<< kernel portion occurs, wherein execution goes into kernel mode through trap,
and the value of eax is used to i) get to system call table and ii) branch to correct
system call >>
kernel mode returns to user mode, past the syscall instruction
EAX now holds the syscall return value, so
that "jae" means if the syscall return value is >=0 - i.e. ok -
continue to the "ret" and return to the user
if not, jump to
cerror_nocancel which loads the value of errno and returns the -1 to
the user.

Answer 3

The Bus error: 10 error appears to be caused by an incorrect syscall number and no exit syscall.
; nasm -f macho64 mkdir.asm && ld -o mkdir mkdir.o && ./mkdir
%define SYSCALL_MKDIR 0x2000088
%define SYSCALL_EXIT 0x2000001
global start
section .text
start:
call mkdir
call exit
ret
mkdir:
mov rax, SYSCALL_MKDIR
mov rdi, directory
mov rsi, 0x1ED
syscall
exit:
mov rax, SYSCALL_EXIT
mov rdi, 0
syscall
section .data
directory: db 'tmp', 0
Summary of changes to the original code:
Renaming the main symbol to start
Changing the mkdir syscall number from 0x2000136 to 0x2000088
Changing the registry assignments
Changing the 0xa character to 0 in the directory variable (works without but results in an incorrect filename)
NASM
I also had to install version 2.10.09 of nasm:
brew install https://raw.githubusercontent.com/Homebrew/homebrew/c1616860c8697ffed8887cae8088ab39141f0308/Library/Formula/nasm.rb
brew switch nasm 2.10.09
This was due to:
No nacho64 support in /usr/bin/nasm
Latest brew version (2.11.08) results in this error: fatal: No section for index 2 offset 0 found