ABI and Calling Conventions

This chapter explains the ABI (application binary interface) and the calling conventions it includes. The ABI is what makes separately compiled code — and code from different languages — interoperate.

Why This Matters

The C *language* is standardized; the *binary* interface is not. The ABI determines how arguments are passed, how structs are returned, how names are mangled, and how the stack is aligned. Getting the ABI wrong corrupts data or crashes, even when the source looks correct.

Prerequisites

• c.build.1 — the compilation pipeline.

Core Concept

ABI

The ABI is the complete contract between compiled units. It includes:

• size and alignment of each type;
• calling conventions;
• name mangling (for C++);
• struct layout and padding rules;
• register usage and stack alignment;
• how variadic functions and large returns work.

The C standard leaves these to the implementation; the ABI fixes them for a specific platform.

Calling convention

A calling convention specifies:

• where arguments go (registers first, then stack);
• who cleans the stack (caller or callee);
• which registers are caller-saved vs. callee-saved;
• where the return value goes;
• stack alignment at call boundaries.

Examples

x86-64 System V (Linux/macOS, simplified)

• Integer/pointer args: RDI, RSI, RDX, RCX, R8, R9, then stack.
• Floating args: XMM0–XMM7.
• Return value: RAX (integer), XMM0 (floating).
• Stack is 16-byte aligned before a call.

Windows x64

• Integer args: RCX, RDX, R8, R9, then stack.
• Return: RAX.
• The conventions differ from System V in several details.

These are platform-specific facts, not ISO C. PLATFORM-SPECIFIC

How It Works

The compiler generates code that obeys the ABI: it moves arguments into the right registers/stack slots, saves callee-saved registers, and reads the return value from the right register. The linker and loader assume all objects follow the same ABI.

Variations

Calling conventions within one platform

x86 (32-bit) had several conventions (cdecl, stdcall, fastcall); x86-64 mostly unified to System V (Unix) and Microsoft x64 (Windows).

Struct returns

Small structs may be returned in registers; large structs may be returned via a hidden pointer argument (sret). This is ABI-defined.

Common Mistakes

• Assuming a struct's layout matches between two compilers or languages.
• Mixing calling conventions (e.g., calling a stdcall function as cdecl).
• Assuming the ABI is the same across OSes on the same CPU.

Undefined Behavior

• Calling a function through a pointer of an incompatible type is UB (and can

violate the ABI at the machine level).

Portability

• The ABI is platform- and compiler-specific. Cross-language FFI (c.ffi.*)

depends critically on matching the ABI.

Under the Hood

The ABI is realized in the generated assembly: argument moves, stack adjustments, register saves, and return-value placement. c.cpu.6 shows concrete examples.

Practical Usage

• When writing FFI, use the platform's C ABI as the common denominator.
• Use sizeof/_Alignof/offsetof (not assumptions) when layout matters.
• Respect extern "C" (C++) to avoid name mangling (c.ffi.cpp).

Exercises

1. Write a tiny function and compile to assembly; identify where each argument is placed. 2. Write a function that returns a large struct and observe the sret pattern. 3. Compare the assembly of a simple call on Linux vs. Windows (or two architectures) if available.

Deep Challenge

Explain the full sequence of a function call on x86-64 System V: argument placement, stack alignment, prologue, and return. Then explain how a variadic function's va_list interacts with the register save area.

Related Concepts

• c.cpu.2 — stack frames.
• c.ffi.1 — ABI as interop foundation.
• c.build.5 — executable formats.

References

• System V AMD64 ABI; Microsoft x64 calling convention; platform ABI docs.

Verification

• ABI and calling conventions are platform/compiler-specific. PLATFORM-SPECIFIC
• No example was executed during generation unless noted. `Execution not

verified.`

Progress

• [ ] Read
• [ ] Understand
• [ ] Complete examples
• [ ] Complete exercises
• [ ] Complete deep challenge

Concept checkboxes

• [ ] ABI definition
• [ ] Calling conventions
• [ ] Register vs. stack arguments
• [ ] Struct return (sret)
• [ ] Platform differences

Mastery levels