Control Flow Isolation Across Module Boundaries with MPK and IBT

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Existing software-based fault isolation (SFI) techniques suffer from significant performance overhead due to hardware independence and instrumentation proliferation, failing to effectively contain security breaches across module boundaries in computer programs.

Innovation Solution

A hybrid approach combining Intel Memory Protection Keys (MPK) and Indirect Branch Tracking (IBT) introduces data dependencies for branch control flow integrity, using prologues at valid targets to enforce access control and restrict execution to authorized domains, reducing overhead and improving fault isolation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If software-based fault isolation (SFI) is implemented using traditional instrumentation techniques, then security breach containment is improved, but performance overhead increases significantly due to instrumentation proliferation at multiple call sites

Engineering Contradiction:
Improvesecurity breach containmentVSAvoidperformance overhead
Core Design Contradiction:
ReliabilityVSProductivity

Solution Approach 1:

The patent segments the fault isolation mechanism into two parts: (1) hardware-based indirect branch tracking that provides coarse-grained control flow validation, and (2) selective instrumentation only at critical call sites that invoke sensitive subroutines. This segmentation reduces the total number of instrumentation points while maintaining security, as the hardware layer handles general control flow validation and software instrumentation focuses only on high-risk boundaries.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary layer between the control flow validation mechanism and the instrumentation code. This intermediary uses hardware-supported indirect branch tracking to validate control flow targets before allowing execution, reducing the need for extensive software-based checks at every call site. The hardware intermediary handles routine validation, allowing software instrumentation to be minimized.

Inventive Principle:
Principle #24Intermediary (Mediator)

2Reliability

If instrumentation is added at every call site to enforce access control, then fault isolation security is improved, but device complexity increases due to proliferation of access control checks

Engineering Contradiction:
Improvefault isolation securityVSAvoidinstrumentation complexity
Core Design Contradiction:
ReliabilityVSDevice complexity

Solution Approach 1:

The patent implements a universal access control mechanism using hardware-based indirect branch tracking that can validate control flow targets across multiple call sites simultaneously. This single hardware mechanism serves multiple security functions: validating indirect jumps, validating function calls, and enforcing domain boundaries. By making the control flow validation universal and hardware-based, the patent reduces the need for separate instrumentation at each call site, thereby reducing overall system complexity while maintaining comprehensive security.

Inventive Principle:
Principle #6Universality (Multi-functionality)

3Productivity

If hardware-based control flow validation is implemented alone, then performance overhead is reduced, but security effectiveness decreases due to lack of fine-grained access control

Engineering Contradiction:
Improveperformance overheadVSAvoidsecurity effectiveness
Core Design Contradiction:
ProductivityVSReliability

Solution Approach 1:

The patent applies local quality by combining hardware-based validation (providing general control flow integrity) with selective software instrumentation at specific call sites that invoke sensitive subroutines. The software instrumentation adds fine-grained access control checks locally at critical boundaries where security is most needed, while relying on the hardware layer for general control flow validation. This localized enhancement of security at critical points maintains overall security effectiveness without requiring universal instrumentation.

Inventive Principle:
Principle #3Local quality

Data Source

PatentUS12450059B2Method for control flow isolation with protection keys and indirect branch tracking
Publication Date: 2025.10.21 ORACLE INT CORP
  • US12450059B2 patent drawing
  • US12450059B2 patent drawing
  • US12450059B2 patent drawing

AI summary

Herein is innovative control flow integrity (CFI) based on code generation techniques that instrument data protection for access control of subroutines invoked across module boundaries. This approach is counterintuitive because, even though code is stored separately from data, access control to the data is used to provide access control to the code. In an embodiment, an instrumentation computer generates, at the beginning of a subroutine that is implemented in machine instructions, a prologue that contains: a first instruction of the subroutine that indicates that the first instruction is a target of a control flow branch and a second instruction of the subroutine that verifies that a memory address is accessible. Generated in the machine instructions are instruction(s) that, when executed by a processor, cause the memory address to have limited accessibility. Some code generation may be performed at the start of runtime by a loader or a dynamic linker.