Functions and the Stack

By the end of this lesson you can write a function that other code can call safely, including one that calls itself. The rules for that are the System V AMD64 ABI, the calling convention Linux compilers use. Learn it once and you can read any compiled C program.

call, ret and the stack

The stack is a region of memory that grows toward lower addresses, with rsp pointing at its top. push x does rsp -= 8 and then stores x at [rsp]; pop x loads from [rsp] and then does rsp += 8.

• call label pushes the address of the next instruction, then jumps to label.
• ret pops an address from the stack and jumps to it.

So ret only works if rsp points at the return address, which means that whatever a function pushes it must pop before returning. One push too many and ret jumps to a garbage address.

The calling convention

• Arguments, in order: rdi, rsi, rdx, rcx, r8, r9. Beyond six, the stack.
• Return value: rax.
• Caller-saved (a function may destroy them): rax, rcx, rdx, rsi, rdi, r8 to r11.
• Callee-saved (a function must give them back unchanged): rbx, rbp, rsp, r12 to r15.

Watch the fourth argument: functions take it in rcx, but the syscall convention from lesson 1 uses r10, because syscall itself destroys rcx.

A leaf function, one that calls nothing, can usually work entirely in caller-saved registers and needs no stack at all:

; max2(a, b): rdi = a, rsi = b, returns the larger signed value in rax
max2:
    mov rax, rdi
    cmp rsi, rax
    jle .done
    mov rax, rsi
.done:
    ret

Surviving your own call

A function that calls another one has a problem: the callee may destroy every caller-saved register, including the argument it still needs. The standard fix is to park values in callee-saved registers, saving the caller's old contents on the stack first. Here is Fibonacci, with two recursive calls and one value that has to wait out the first:

; fib(n): rdi = n, returns fib(n) in rax. fib(0) = 0, fib(1) = 1.
fib:
    cmp rdi, 2
    jb .small               ; n is 0 or 1: the answer is n itself
    push rbx                ; we are about to use rbx and r12, so save them
    push r12
    mov rbx, rdi            ; n must survive the calls below
    lea rdi, [rbx - 1]
    call fib                ; rax = fib(n - 1)
    mov r12, rax            ; park it while computing the other half
    lea rdi, [rbx - 2]
    call fib                ; rax = fib(n - 2)
    add rax, r12
    pop r12                 ; restore in the reverse order of the pushes
    pop rbx
    ret
.small:
    mov rax, rdi
    ret

Each active call keeps its own rbx value on the stack while the deeper call reuses the register. The base case returns before touching the stack, and every path that pushed pops exactly what it pushed. A function with no base case recurses until the stack runs into unmapped memory and the process dies with a segfault.

Locals work the same way: sub rsp, 16 reserves two 8-byte slots at [rsp] and [rsp + 8], and add rsp, 16 gives them back before ret.

One more rule matters when you call C code: rsp must be a multiple of 16 at the moment of the call, because SSE instructions inside libc fault otherwise. Your own functions in this course never need it, but a compiler pads frames for exactly this reason.

Common mistakes

• Using rbx or r12 without saving it. Your function works, then its caller's value is gone and the failure shows up somewhere else.
• Unbalanced pushes. ret pops a data value instead of the return address.
• Missing base case, or one that is off by one for the smallest input.
• Overwriting rdi before you have used it to set up the next call.

Your exercise

The starter's main parses a digit, keeps it in rbx across the call, and prints the result with a ready-made print_uint routine. It checks your ABI discipline indirectly: it reads rbx after your function returns. Fill in fact, and treat the smallest inputs with care.