Self-modifying code on macOS v10.15 (Catalina) / x64 - macos

Question

As part as porting a Forth compiler, I'm trying to create a binary that allows for self-modifying code. The gory details are at https://github.com/klapauciusisgreat/jonesforth-MacOS-x64
Ideally, I create a bunch of pages for user definitions and call mprotect like so:
#define __NR_exit 0x2000001
#define __NR_open 0x2000005
#define __NR_close 0x2000006
#define __NR_read 0x2000003
#define __NR_write 0x2000004
#define __NR_mprotect 0x200004a
#define PROT_READ 0x01
#define PROT_WRITE 0x02
#define PROT_EXEC 0x04
#define PROT_ALL (PROT_READ | PROT_WRITE | PROT_EXEC)
#define PAGE_SIZE 4096
// https://opensource.apple.com/source/xnu/xnu-201/bsd/sys/errno.h
#define EACCES 13 /* Permission denied */
#define EINVAL 22 /* Invalid argument */
#define ENOTSUP 45 /* Operation not supported */
/* Assembler entry point. */
.text
.globl start
start:
// Use mprotect to allow read/write/execute of the .bss section
mov $__NR_mprotect, %rax // mprotect
lea user_defs_start(%rip), %rdi // Start address
and $-PAGE_SIZE,%rdi // Align at page boundary
mov $USER_DEFS_SIZE, %rsi // Length
mov $PROT_ALL,%rdx
syscall
cmp $EINVAL, %rax
je 1f
cmp $EACCES,%rax
je 2f
test %rax,%rax
je 4f // All good, proceed:
// must be ENOTSUP
mov $2,%rdi // First parameter: stderr
lea errENOTSUP(%rip),%rsi // Second parameter: error message
mov $8,%rdx // Third parameter: length of string
mov $__NR_write,%rax // Write syscall
syscall
jmp 3f
1:
mov $2,%rdi // First parameter: stderr
lea errEINVAL(%rip),%rsi // Second parameter: error message
mov $7,%rdx // Third parameter: length of string
mov $__NR_write,%rax // Write syscall
syscall
jmp 3f
2:
mov $2,%rdi // First parameter: stderr
lea errEACCES(%rip),%rsi // Second parameter: error message
mov $7,%rdx // Third parameter: length of string
mov $__NR_write,%rax // Write syscall
syscall
3:
// did't work -- then exit
xor %rdi,%rdi
mov $__NR_exit,%rax // syscall: exit
syscall
4:
// All good, let's get started for real:
.
.
.
.set RETURN_STACK_SIZE,8192
.set BUFFER_SIZE,4096
.set USER_DEFS_SIZE,65536*2 // 128 kiB ought to be enough for everybody
.bss
.balign 8
user_defs_start:
.space USER_DEFS_SIZE
However, I get an EACCES return value. I suspect that this is because of some security policy apple has set up, but I do not find good documentation.
Where is the source code for mprotect, and/or what are the methods to mark the data area executable and writable at the same time?
I found that compiling with
gcc -segprot __DATA rwx rwx
does mark the entire data segment rwx, so it must somehow be possible to do the right thing. But I would prefer to only make the area hosting Forth words executable, not the entire data segment.
I found a similar discussion here, but without any solution.

Answer 1

The segment that I want to 'unprotect' exec permission in has really two values describing its permissions:
the initial protection settings, which for __DATA I want rw-
the maximum protection (loosest) settings, which I want to be rwx.
So first I need to set the maxprot field to rwx. According to the ld manpage, this should be achieved by invoking gcc or ld with the flags -segprot __DATA rwx rw. However, a recent change made by Apple to the linker essentially ignores the maxprot value, and sets maxprot=initprot.
Thanks to Darfink, you can use this script to tweak the maxprot bits after the fact. I thought additional code signing with special entitlements was required, but it's not, at least for the __DATA segment. The __TEXT segment may need code signing with the com.apple.security.cs.disable-executable-page-protection entitlement.
Also see here for a few more details.
Looking at the larger picture, I should also point out that rather to unprotect pieces of an otherwise protected __DATA segment, it may be better to create a complete new data/code segment just for the self-modifying code with rwx permissions from the start. This allows still protecting the rest of data by the operating system, and requires no non-standard tools.