Part 6 — Debugging and Optimization

UB-Based Optimization and Compiler Assumptions

This chapter explains how compilers exploit undefined behavior to optimize — and why UB is therefore far more dangerous than "the hardware does something weird."

Why This Matters

When you write UB, the compiler is allowed to assume it never happens. It can then delete your bounds checks, reorder your code, or remove entire branches. This is why UB programs behave *differently under optimization*, not just "incorrectly."

Prerequisites

c.object.6 — definedness classification.
c.opt.1 — optimizer basics.

Core Concept

A compiler optimizes under the assumption that the program is defined. If the program contains UB, the compiler's reasoning is unsound, and the generated code may do anything — including things impossible in the source.

The classic example:

int table[4];

int lookup(int i)
{
    if (i < 0 || i >= 4) return -1;
    return table[i];   /* the compiler may assume i is in bounds */
}

Because table[i] would be UB if i were out of bounds, the compiler may assume the check i < 0 || i >= 4 is always false and delete it. The "check" provides no protection against UB.

Examples

Signed overflow enabling elimination

int f(int x)
{
    if (x > 0 && x + 1 > 0)   /* compiler assumes x+1 doesn't overflow */
        return 1;
    return 0;
}

The compiler may assume x + 1 never overflows, so x > 0 implies x + 1 > 0 and the whole function returns 1.

Uninitialized read enabling branch removal

int f(void)
{
    int x;
    if (x) return 1;   /* reading x is UB, so the branch is unreachable */
    return 0;
}

The compiler may assume the if (x) is unreachable and return 0 always.

How It Works

Optimization passes (range analysis, dead-code elimination, alias analysis) use the assumption "no UB" to prune possibilities. A signed overflow or out-of-bounds access becomes a license to assume a path is impossible.

Variations

Sanitizers vs. UB-based optimization

UBSan inserts checks that make the UB *detectable*, but it changes the code. Without it, the optimizer exploits the UB.

The "as-if" rule

The compiler may do anything as long as observable behavior is preserved *for defined programs*. UB is outside the contract.

Common Mistakes

Adding a bounds check *after* an out-of-bounds access and expecting it to

help.

Assuming UB manifests as a crash or garbage rather than silently changing

control flow.

Reasoning about C as if it were "portable assembly."

Undefined Behavior

This chapter is about UB. Any UB can be exploited by the optimizer; the

examples illustrate specific cases.

Portability

The *exploitation* of UB is implementation/optimization dependent, which is

why UB programs vary across compilers and flags.

Under the Hood

Range analysis propagates invariants; if an operation is UB outside a range, the range is assumed. Dead-code elimination removes paths that would require UB. Alias analysis (strict aliasing) assumes non-aliasing and reorders.

Practical Usage

Write only defined C.
Run UBSan/ASan to find UB before it bites under optimization.
Never put a bounds check *after* the access that can be UB.

Exercises

1. Write the table[i] example and compare assembly at -O2 vs. -O0. 2. Write a signed-overflow example and observe the optimized result. 3. Use UBSan to catch the UB in each example.

Deep Challenge

Construct a program where an out-of-bounds access causes the optimizer to remove a security-relevant check, and explain the exact reasoning. Then show the correct, defined version and prove the check survives optimization.

c.ub.catalog — common UB.
c.obj.aliasing — strict aliasing.
c.debug.ubsan — detecting UB.

References

LLVM/GCC blog posts and docs on UB and optimization; WG14 papers.

Verification

Compilers are documented to assume UB does not occur. VERIFIED
No example was executed during generation unless noted. `Execution not

verified.`

Progress

[ ] Read
[ ] Understand
[ ] Complete examples
[ ] Complete exercises
[ ] Complete deep challenge

Concept checkboxes

[ ] UB assumption in optimization
[ ] Bounds-check elimination
[ ] Signed-overflow exploitation
[ ] Uninitialized-read exploitation

Mastery levels

Concept	Current level (0–8)	Target level
c.opt.ub	0	7

← Prev Next →