Hypervisor Memory Permission Enforcement for Kernel Code Protection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing technologies, such as SMAP and SMEP, are not universally available and may allow kernel access or execution of application code through alternate memory mapping, posing security risks in protecting application code from kernel mode execution.

Innovation Solution

A method and system that utilize a hypervisor to detect allocated physical memory pages for an application and request setting these pages to an executable-by-user mode in the hypervisor's page tables, preventing kernel mode execution.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If SMAP or SMEP technologies are used to prevent kernel mode execution of application code, then application code execution during kernel mode is blocked, but these technologies are not universally available and may allow kernel access through alternate memory mapping

Engineering Contradiction:

Improveapplication code protectionVSAvoidhardware compatibility

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The patent introduces a hypervisor as an intermediary layer between the application code and the kernel mode execution environment. The hypervisor intercepts memory access requests and enforces execution permissions, preventing kernel mode execution of application code without requiring hardware-specific features like SMAP or SMEP. This mediator approach achieves reliable code protection while maintaining broad hardware compatibility.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If kernel mode execution of application code is prevented through hardware features, then security is improved, but device complexity increases due to reliance on specific hardware capabilities

Engineering Contradiction:

ImprovesecurityVSAvoidhardware dependency

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The hypervisor serves as a software-based intermediary that implements security policies without relying on hardware-specific features. By managing memory permissions and intercepting execution requests at the virtualization layer, the system achieves high security while avoiding the complexity of hardware feature dependencies.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces hardware-based security mechanisms (SMAP/SMEP) with a software-based hypervisor implementation. This substitution eliminates the need for specific hardware capabilities, reducing device complexity and hardware dependency while maintaining or enhancing security through software-controlled memory permission management.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Object-affected harmful factors

If application code is protected from kernel mode execution, then malicious code execution is prevented, but the system requires additional layers of protection increasing complexity

Engineering Contradiction:

Improvemalicious code executionVSAvoidprotection layers

Core Design Contradiction:

Object-affected harmful factorsVSDevice complexity

Solution Approach 1:

The hypervisor acts as a single comprehensive intermediary that consolidates multiple protection functions into one layer. It manages memory permissions, intercepts execution requests, and enforces security policies, replacing the need for multiple separate protection mechanisms and reducing overall system complexity while effectively preventing malicious code execution.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11429412B2Guest protection from application code execution in kernel mode

Publication Date: 2022.08.30 RED HAT ISRAEL

AI summary

Systems and methods are disclosed for securing an application running on a guest. An example method includes detecting, by a guest running on a virtual machine, that a set of physical memory pages is allocated to an application. The virtual machine runs on a hypervisor, and the application runs on the guest. During runtime, the guest may send a request to the hypervisor to set the set of physical memory pages to an executable-by-user mode in the hypervisor's page tables.