Stack Frames and Calling Conventions

This chapter explains how a function call is realized on the stack and in registers: the stack frame, prologue/epilogue, and calling convention.

Why This Matters

The stack frame is where automatic variables, return addresses, and saved registers live. Understanding it is essential for reading assembly, debugging crashes, and writing correct low-level or FFI code.

Prerequisites

• c.cpu.1 — CPU registers.

Core Concept

When a function is called:

1. The caller places arguments in registers/stack (per the calling convention). 2. The caller executes call, pushing the return address. 3. The callee runs a prologue: saves the frame pointer, sets up the new frame, and reserves stack space for locals. 4. The callee body runs. 5. The callee runs an epilogue: restores the stack/frame pointer and returns to the caller.

The stack frame is the region between the frame pointer and stack pointer holding the function's locals and saved state.

Example (x86-64 System V)

int add(int a, int b)
{
    int sum = a + b;
    return sum;
}

add:
    push rbp            ; save caller's frame pointer
    mov  rbp, rsp       ; set frame pointer
    mov  DWORD PTR [rbp-4], edi   ; a (arg in edi) -> local
    mov  DWORD PTR [rbp-8], esi   ; b (arg in esi) -> local
    mov  edx, DWORD PTR [rbp-4]
    mov  eax, DWORD PTR [rbp-8]
    add  eax, edx
    mov  DWORD PTR [rbp-12], eax  ; sum -> local
    mov  eax, DWORD PTR [rbp-12]  ; return value in eax
    pop  rbp            ; restore caller's frame pointer
    ret

At -O2 much of this (the locals and frame pointer) is optimized away.

How It Works

The stack grows downward on x86/ARM. push decrements SP and writes; pop reads and increments SP. call pushes the return address and jumps; ret pops it and jumps back.

Variations

Frame pointer omission

With -fomit-frame-pointer (default at -O2), the compiler uses SP-relative addressing and may omit rbp, freeing it for general use. Backtraces then rely on DWARF call frame info.

Red zone (x86-64 System V)

There is a 128-byte "red zone" below the stack pointer that functions may use without adjusting SP; it is ABI-specific.

Common Mistakes

• Confusing the caller's frame with the callee's frame.
• Assuming arguments are always on the stack (they are often in registers).

Undefined Behavior

• Stack overflow (too-deep recursion or huge locals) is UB at the C level.

Portability

• Stack direction, frame layout, and conventions are ABI-specific.

Under the Hood

The prologue/epilogue are emitted by the compiler for every function. At optimization, the compiler may keep variables entirely in registers and emit no frame at all.

Practical Usage

• Read push rbp/mov rbp,rsp to identify function prologues in assembly.
• Use backtrace and frame inspection in GDB to understand crashes.

Exercises

1. Compile a function at -O0 and identify its prologue/epilogue. 2. Trace a call with si (step instruction) in GDB, watching the stack pointer. 3. Compare the same function at -O2 and observe frame-pointer omission.

Deep Challenge

Draw the full stack layout during a nested call (caller frame, return address, callee frame, locals, saved registers) on x86-64 System V, and explain how the epilogue unwinds it.

Related Concepts

• c.cpu.1 — registers.
• c.build.calling-convention — ABI.
• c.core.18 — recursion.

References

• System V AMD64 ABI, ARM AAPCS.

Verification

• Stack-frame layout and conventions are ABI-specific. HARDWARE/ABI
• No example was executed during generation unless noted. `Execution not

verified.`

Progress

• [ ] Read
• [ ] Understand
• [ ] Complete examples
• [ ] Complete exercises
• [ ] Complete deep challenge

Concept checkboxes

• [ ] Stack frame layout
• [ ] Prologue/epilogue
• [ ] Return address
• [ ] Frame pointer omission

Mastery levels