cedecl calling convention -- compiled asm instructions cause crash - debugging

Treat this more as pseudocode than anything. If there's some macro or other element that you feel should be included, let me know.
I'm rather new to assembly. I programmed on a pic processor back in college, but nothing since.
The problem here (segmentation fault) is the first instruction after "Compile function entrance, setup stack frame." or "push %ebp". Here's what I found out about those two instructions:
http://unixwiz.net/techtips/win32-callconv-asm.html
Save and update the %ebp :
Now that we're in the new function, we need a new local stack frame pointed to by %ebp, so this is done by saving the current %ebp (which belongs to the previous function's frame) and making it point to the top of the stack.
push ebp
mov ebp, esp // ebp « esp
Once %ebp has been changed, it can now refer directly to the function's arguments as 8(%ebp), 12(%ebp). Note that 0(%ebp) is the old base pointer and 4(%ebp) is the old instruction pointer.
Here's the code. This is from a JIT compiler for a project I'm working on. I'm doing this more for the learning experience than anything.
IL_CORE_COMPILE(avs_x86_compiler_compile)
{
X86GlobalData *gd = X86_GLOBALDATA(ctx);
ILInstruction *insn;
avs_debug(print("X86: Compiling started..."));
/* Initialize X86 Assembler opcode context */
x86_context_init(&gd->ctx, 4096, 1024*1024);
/* Compile function entrance, setup stack frame*/
x86_emit1(&gd->ctx, pushl, ebp);
x86_emit2(&gd->ctx, movl, esp, ebp);
/* Setup floating point rounding mode to integer truncation */
x86_emit2(&gd->ctx, subl, imm(8), esp);
x86_emit1(&gd->ctx, fstcw, disp(0, esp));
x86_emit2(&gd->ctx, movl, disp(0, esp), eax);
x86_emit2(&gd->ctx, orl, imm(0xc00), eax);
x86_emit2(&gd->ctx, movl, eax, disp(4, esp));
x86_emit1(&gd->ctx, fldcw, disp(4, esp));
for (insn=avs_il_tree_base(tree); insn != NULL; insn = insn->next) {
avs_debug(print("X86: Compiling instruction: %p", insn));
compile_opcode(gd, obj, insn);
}
/* Restore floating point rounding mode */
x86_emit1(&gd->ctx, fldcw, disp(0, esp));
x86_emit2(&gd->ctx, addl, imm(8), esp);
/* Cleanup stack frame */
x86_emit0(&gd->ctx, emms);
x86_emit0(&gd->ctx, leave);
x86_emit0(&gd->ctx, ret);
/* Link machine */
obj->run = (AvsRunnableExecuteCall) gd->ctx.buf;
return 0;
}
And when obj->run is called, it's called with obj as its only argument:
obj->run(obj);
If it helps, here are the instructions for the entire function call. It's basically an assignment operation: foo=3*0.2;. foo is pointing to a float in C.
0x8067990: push %ebp
0x8067991: mov %esp,%ebp
0x8067993: sub $0x8,%esp
0x8067999: fnstcw (%esp)
0x806799c: mov (%esp),%eax
0x806799f: or $0xc00,%eax
0x80679a4: mov %eax,0x4(%esp)
0x80679a8: fldcw 0x4(%esp)
0x80679ac: flds 0x806793c
0x80679b2: fsts 0x805f014
0x80679b8: fstps 0x8067954
0x80679be: fldcw (%esp)
0x80679c1: add $0x8,%esp
0x80679c7: emms
0x80679c9: leave
0x80679ca: ret
Edit: Like I said above, in the first instruction in this function, %ebp is void. This is also the instruction that causes the segmentation fault. Is that because it's void, or am I looking for something else?
Edit: Scratch that. I keep typing edp instead of ebp. Here are the values of ebp and esp.
(gdb) print $esp
$1 = (void *) 0xbffff14c
(gdb) print $ebp
$3 = (void *) 0xbffff168
Edit: Those values above are wrong. I should have used the 'x' command, like below:
(gdb) x/x $ebp
0xbffff168: 0xbffff188
(gdb) x/x $esp
0xbffff14c: 0x0804e481
Here's a reply from someone on a mailing list regarding this. Anyone care to illuminate what he means a bit? How do I check to see how the stack is set up?
An immediate problem I see is that the
stack pointer is not properly aligned.
This is 32-bit code, and the Intel
manual says that the stack should be
aligned at 32-bit addresses. That is,
the least significant digit in esp
should be 0, 4, 8, or c.
I also note that the values in ebp and
esp are very far apart. Typically,
they contain similar values --
addresses somewhere in the stack.
I would look at how the stack was set
up in this program.
He replied with corrections to the above comments. He was unable to see any problems after further input.
Another edit: Someone replied that the code page may not be marked executable. How can I insure it is marked as such?

The problem had nothing to do with the code. Adding -z execstack to the linker fixed the problem.

If push %ebp is causing a segfault, then your stack pointer isn't pointing at valid stack. How does control reach that point? What platform are you on, and is there anything odd about the runtime environment? At the entry to the function, %esp should point to the return address in the caller on the stack. Does it?
Aside from that, the whole function is pretty weird. You go out of your way to set the rounding bits in the fp control word, and then don't perform any operations that are affected by rounding. All the function does is copy some data, but uses floating-point registers to do it when you could use the integer registers just as well. And then there's the spurious emms, which you need after using MMX instructions, not after doing x87 computations.
Edit See Scott's (the original questioner) answer for the actual reason for the crash.

Related

How debuggers can trace the stack?

I was trying to implement a stack tracer, using stack pointers; RSP and RBP, but I think debuggers use an entirely different way to grab the return addresses, or maybe I am missing something. I can grab the return address of the last stack frame, but I can't get the others because I don't know the size of other stack frames, so I can't figure out how much bytes should I go back from stack frame, to get the return address. Are there anybody know which way do debuggers use to trace stack?
It is possible to trace the stack when the code uses frame pointers. In this case ebp/rbp is used as the frame pointer and functions begin with prologs and end with epilogs.
A typical prolog looks like this:
push rbp ; save previous frame pointer
mov rbp, rsp ; initialize this functions frame pointer
A typical epilog looks like this:
mov rsp, rbp ; restore the value of rsp
pop rbp ; restore previous frame pointer value from stack
retn
Thus in every place in a function rbp points to the stack position where the previous frame pointer is saved and rbp+8 contains the saved return address.
To get the called function a debugger should read [rbp+8] value and find a function to which this address belongs. This can be done by searching in debugging symbols.
Next it should read [rbp] value to get the frame pointer of the caller function. Continue this process until you find a toplevel function. This is typically a system library function that starts threads.

x86 assembler pushf causes program to exit

I think my real problem is I don't completely understand the stack frame mechanism so I am looking to understand why the following code causes the program execution to resume at the end of the application.
This code is called from a C function which is several call levels deep and the pushf causes program execution to revert back several levels through the stack and completely exit the program.
Since my work around works as expected I would like to know why using the pushf instruction appears to be (I assume) corrupting the stack.
In the routines I usually setup and clean up the stack with :
sub rsp, 28h
...
add rsp, 28h
However I noticed that this is only necessary when the assembly code calls a C function.
So I tried removing this from both routines but it made no difference. SaveFlagsCmb is an assembly function but could easily be a macro.
The code represents an emulated 6809 CPU Rora (Rotate Right Register A).
PUBLIC Rora_I_A ; Op 46 - Rotate Right through Carry A reg
Rora_I_A PROC
sub rsp, 28h
; Restore Flags
mov cx, word ptr [x86flags]
push cx
popf
; Rotate Right the byte and save the FLAGS
rcr byte ptr [q_s+AREG], 1
; rcr only affects Carry. Save the Carry first in dx then
; add 0 to result to trigger Zero and Sign/Neg flags
pushf ; this causes jump to end of program ????
pop dx ; this line never reached
and dx, CF ; Save only Carry Flag
add [q_s+AREG], 0 ; trigger NZ flags
mov rcx, NF+ZF+CF ; Flag Mask NZ
Call SaveFlagsCmb ; NZ from the add and CF saved in dx
add rsp, 28h
ret
Rora_I_A ENDP
However if I use this code it works as expected:
PUBLIC Rora_I_A ; Op 46 - Rotate Right through Carry A reg
Rora_I_A PROC
; sub rsp, 28h ; works with or without this!!!
; Restore Flags
mov ah, byte ptr [x86flags+LSB]
sahf
; Rotate Right the byte and save the FLAGS
rcr byte ptr [q_s+AREG], 1
; rcr only affects Carry. Save the Carry first in dx then
; add 0 to result to trigger Zero and Sign/Neg flags
lahf
mov dl, ah
and dx, CF ; Save only Carry Flag
add [q_s+AREG], 0 ; trigger NZ flags
mov rcx, NF+ZF+CF ; Flag Mask NZ
Call SaveFlagsCmb ; NZ from the add and CF saved in dx
; add rsp, 28h ; works with or without this!!!
ret
Rora_I_A ENDP
Your reported behaviour doesn't really make sense. Mostly this answer is just providing some background not a real answer, and a suggestion not to use pushf/popf in the first place for performance reasons.
Make sure your debugging tools work properly and aren't being fooled by something into falsely showing a "jump" to somewhere. (And jump where exactly?)
There's little reason to mess around with 16-bit operand size, but that's probably not your problem.
In Visual Studio / MASM, apparently (according to OP's comment) pushf assembles as pushfw, 66 9C which pushes 2 bytes. Presumably popf also assembles as popfw, only popping 2 bytes into FLAGS instead of the normal 8 bytes into RFLAGS. Other assemblers are different.1
So your code should work. Unless you're accidentally setting some other bit in FLAGS that breaks execution? There are bits in EFLAGS/RFLAGS other than condition codes, including the single-step TF Trap Flag: debug exception after every instruction.
We know you're in 64-bit mode, not 32-bit compat mode, otherwise rsp wouldn't be a valid register. And running 64-bit machine code in 32-bit mode wouldn't explain your observations either.
I'm not sure how that would explain pushf being a jump to anywhere. pushf itself can't fault or jump, and if popf set TF then the instruction after popf would have caused a debug exception.
Are you sure you're assembling 64-bit machine code and running it in 64-bit mode? The only thing that would be different if a CPU decoded your code in 32-bit mode should be the REX prefix on sub rsp, 28h, and the RIP-relative addressing mode on [x86flags] decoding as absolute (which would presumably fault). So I don't think that could explain what you're seeing.
Are you sure you're single-stepping by instructions (not source lines or C statements) with a debugger to test this?
Use a debugger to look at the machine code as you single-step. This seem really weird.
Anyway, it seems like a very low-performance idea to use pushf / popf at all, and also to be using 16-bit operand-size creating false dependencies.
e.g. you can set x86 CF with movzx ecx, word ptr [x86flags] / bt ecx, CF.
You can capture the output CF with setc cl
Also, if you're going to do multiple things to the byte from the guest memory, load it into an x86 register. A memory-destination RCR and a memory-destination ADD are unnecessarily slow vs. load / rcr / ... / test reg,reg / store.
LAHF/SAHF may be useful, but you can also do without them too for many cases. popf is quite slow (https://agner.org/optimize/) and it forces a round trip through memory. However, there is one condition-code outside the low 8 in x86 FLAGS: OF (signed overflow). asm-source compatibility with 8080 is still hurting x86 in 2019 :(
You can restore OF from a 0/1 integer with add al, 127: if AL was originally 1, it will overflow to 0x80, otherwise it won't. You can then restore the rest of the condition codes with SAHF. You can extract OF with seto al. Or you can just use pushf/popf.
; sub rsp, 28h ; works with or without this!!!
Yes of course. You have a leaf function that doesn't use any stack space.
You only need to reserve another 40 bytes (align the stack + 32 bytes of shadow space) if you were going to make another function call from this function.
Footnote 1: pushf/popf in other assemblers:
In NASM, pushf/popf default to the same width as other push/pop instructions: 8 bytes in 64-bit mode. You get the normal encoding without an operand-size prefix. (https://www.felixcloutier.com/x86/pushf:pushfd:pushfq)
Like for integer registers, both 16 and 64-bit operand-size for pushf/popf are encodeable in 64-bit mode, but 32-bit operand size isn't.
In NASM, your code would be broken because push cx / popf would push 2 bytes and pop 8, popping 6 bytes of your return address into RFLAGS.
But apparently MASM isn't like that. Probably a good idea to use explicit operand-size specifiers anyway, like pushfw and popfw if you use it at all, to make sure you get the 66 9C encoding, not just 9C pushfq.
Or better, use pushfq and pop rcx like a normal person: only write to 8 or 16-bit partial registers when you need to, and keep the stack qword-aligned. (16-byte alignment before call, 8-byte alignment always.)
I believe this is a bug in Visual Studio. I'm using 2022, so it's an issue that's been around for a while.
I don't know exactly what is triggering it, however stepping over one specific pushf in my code had the same symptoms, albeit with the code actually working.
Putting a breakpoint on the line after the pushf did break, and allowed further debugging of my app. Adding a push ax, pop ax before the pushf also seemed to fix the issue. So it must be a Visual Studio issue.
At this point I think MASM and debugging in Visual Studio has pretty much been abandoned. Any suggestions for alternatives for developing dlls on Windows would be appreciated!

unknown "ptr" variable in golang asm code

Recently I just begin to read the source code of atomic.LoadUint64, then I got an unknown variable "ptr" in the following asm code:
TEXT runtime∕internal∕atomic·Load64(SB), NOSPLIT, $0-12
MOVL ptr+0(FP), AX
TESTL $7, AX
JZ 2(PC)
MOVL 0, AX // crash with nil ptr deref
MOVQ (AX), M0
MOVQ M0, ret+4(FP)
EMMS
RET
I couldn't find the declaration of this variable, and any documents about this variable, can anyone tell me about it?
A Quick Guide to Go's Assembler
Symbols
The FP pseudo-register is a virtual frame pointer used to refer to
function arguments. The compilers maintain a virtual frame pointer and
refer to the arguments on the stack as offsets from that
pseudo-register. Thus 0(FP) is the first argument to the function,
8(FP) is the second (on a 64-bit machine), and so on. However, when
referring to a function argument this way, it is necessary to place a
name at the beginning, as in first_arg+0(FP) and second_arg+8(FP).
(The meaning of the offset—offset from the frame pointer—distinct from
its use with SB, where it is an offset from the symbol.) The
assembler enforces this convention, rejecting plain 0(FP) and
8(FP). The actual name is semantically irrelevant but should be used
to document the argument's name. It is worth stressing that FP is
always a pseudo-register, not a hardware register, even on
architectures with a hardware frame pointer.
ptr, in ptr+0(FP), is a name for the first argument to the function. The argument is probably a pointer.

0xbffff8a8: aam $-0x8 error when saving the base pointer

I am currently following an introductory course in microelectronics and assembly programming in Uni. At the beginning of every function, I'm saving the caller's base pointer by pushing it onto the stack. Given the following function, I get an error:
.globl my_func
.globl _my_func
my_func:
_my_func:
pushl %ebp
movl %esp,%ebp
movl 4(%esp),%ebx
subl $1,%ebx
movl %ebx,%eax
ret
0xbffff8a8: aam $-0x8 <-EXC_BAD_ACCESS (code=2, address=0xbffff8a8)
I've figured out this is a memory exception, I just don't understand why it's being thrown. When I skip the first two instructions in the function (the base pointer saving), the function runs well. And before you point it out -- yes, I know the function is pointless and slow, I'm just trying to learn how the instructions work, and how to use the stack and registers.
I'm assembling it for IA32 on an Intel Mac with OSX10.9 using LLVM5.1
You need to reset the stack pointer at the end of the function, either explicitly or by popping a register to match what you pushed at the start of the function, otherwise when you return it will be to an invalid address:
popl %ebp ; restore stack pointer to its original value
ret

GCC with the -fomit-frame-pointer option

I'm using GCC with the -fomit-frame-pointer and -O2 options. When I looked through the assembly code it generated,
push %ebp
movl %esp, %ebp
at the start and pop %ebp at the end is removed. But some redundant subl/addl instructions to esp is left in - subl $12, %esp at the start and addl $12, %esp at the end.
How will I be able to remove them as some inline assembly will jmp to another function before addl is excecuted.
You probably don't want to remove those -- that's usually the code that allocates and deallocates your local variables. If you remove those, your code will trample all over the return addresses and such.
The only safe way to get rid of them is not to use any local variables. Even in macros. And be really careful about inline functions, as they often have their own locals that'll get put in with yours. You may want to consider explicitly disabling function inlining for that section of code, if you can.
If you're absolutely sure that the adds and subs aren't needed (and i mean really, really sure), on my machine GCC apaprently does some stack manipulation to keep the stack aligned at 16 byte boundaries. You may be able to say "-mpreferred-stack-boundary=2", which will align to 4-byte boundaries -- which x86 processors like to do anyway, so no code is generated to realign it. Works on my box with my GCC; int main() { return 0; } turned into
main:
xorl %eax, %eax
ret
but the alignment code looked different to start with...so that may not be the problem for you.
Just so you're warned: optimization causes a lot of weird stuff like that to happen. Be careful with hand-coded assembler language and optimized <insert-almost-any-language-here> code, especially when you're doing something as unstructured as a jump from the middle of one function into another.
I solved the problem by giving a function prototype, then defining it manually like this:
void my_function();
asm (
".globl _my_function\n"
"_my_function:\n\t"
/* Assembler instructions go here */
);
Later I also wanted the function to be exported, so I added this at the end of the source file:
asm (
".section .drectve\n\t"
".ascii \" -export:my_function\"\n"
);
How will I be able to remove them as some inline assembly will jmp to another function before addl is executed.
This will corrupt your stack, that caller expects the stack pointer
to be corrected on function return. Does the other function return
by ret instruction? What exactly do you try to achieve? maybe there's another solution possible?
Please, show us the lines around the function call (in the caller) and your
entry/exit part of your function in question.

Resources