Hypervisor ROP Detection via LBR Virtualization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current malware detection systems within virtual machines are ineffective in detecting return-oriented programming (ROP) attacks, as malicious applications can evade detection by reading and altering the Last Branch Recording (LBR) functionality, thereby thwarting the monitoring and prevention of ROP exploits.

Innovation Solution

A ROP exploit detection system operating within a hypervisor intercepts function calls and sets breakpoints on memory pages, analyzes Last Branch Records (LBRs) to identify violations of programming invariants, and manages permissions to prevent the detection process from being altered or disabled by the executing application, ensuring accurate detection of ROP attacks.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If malware detection systems analyze Last Branch Records (LBRs) to detect ROP attacks, then detection capability is improved, but malicious applications can read and alter LBR functionality to evade detection

Engineering Contradiction:

ImproveROP attack detection capabilityVSAvoidApplication's ability to evade detection by altering LBR functionality

Core Design Contradiction:

ReliabilityVSObject-affected harmful factors

Solution Approach 1:

The hypervisor acts as an intermediary between the malicious application and the LBR functionality. By virtualizing the LBR registers and intercepting read attempts through hypervisor callbacks, the system prevents applications from directly accessing or altering LBR functionality, thereby maintaining detection reliability while blocking evasion techniques

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent moves the detection capability from the application level to the hypervisor level, creating a hierarchical separation. The hypervisor maintains virtual LBR registers and controls access to actual LBR functionality, adding a dimensional layer of protection that prevents applications from interfering with detection mechanisms

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Measurement precision

If breakpoints are set on memory pages to monitor application behavior, then detection accuracy is improved, but applications can detect and alter breakpoint functionality to avoid detection

Engineering Contradiction:

ImproveApplication behavior monitoring accuracyVSAvoidApplication's ability to detect and alter breakpoint functionality

Core Design Contradiction:

Measurement precisionVSObject-affected harmful factors

Solution Approach 1:

The hypervisor serves as an intermediary that manages breakpoint functionality. When applications attempt to read or alter breakpoint-related registers, the hypervisor intercepts these operations through callback routines, preventing evasion while maintaining monitoring accuracy

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system performs preliminary protection by setting up hypervisor-level intercepts and permission controls before applications can detect or alter breakpoint functionality. This preemptive measure ensures that monitoring accuracy is maintained throughout application execution

Inventive Principle:

Principle #10Preliminary action

3Reliability

If LBR functionality is enabled for monitoring, then ROP detection is improved, but applications can detect LBR functionality is turned on and turn it off to avoid detection

Engineering Contradiction:

ImproveROP attack detection capabilityVSAvoidApplication's ability to adapt by disabling LBR functionality

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The hypervisor performs preliminary action by intercepting attempts to modify LBR functionality before they can take effect. Through permission controls and callback routines, the system prevents applications from disabling LBR functionality, ensuring continuous detection capability

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The hypervisor acts as an intermediary that controls access to LBR functionality. By managing virtual LBR registers and intercepting modification attempts, the system maintains detection reliability while preventing applications from adapting to evade detection

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS10621338B1Method to detect forgery and exploits using last branch recording registers

Publication Date: 2020.04.14 MAGENTA SECURITY HOLDINGS LLC

AI summary

A method for detecting a ROP attack comprising processing of an object within a virtual machine managed by a virtual machine monitor (VMM), intercepting an attempted execution by the object of an instruction, the instruction stored on a page in memory that is accessed by the virtual machine, responsive to determining the page includes instructions corresponding to one of a predefined set of function calls, (i) inserting a first transition event into the memory at a starting address location of a function call, and (ii) setting a permission of the page to be execute only, and responsive to triggering the first transition event, halting, by the VMM, the processing of the object and analyzing, by logic within the VMM, content of last branch records associated with the virtual machine to determine whether the processing of the object displays characteristics of a ROP attack is shown.