Nested Enclave Virtual Machine Security via Segmentation

The patent implements nested enclaves where a root enclave contains child enclaves, allowing hierarchical protection structures. The root enclave executes privileged instructions and manages child enclaves, while child enclaves provide protection for specific applications or system components. This nesting enables system-level software to be protected through a hierarchy of enclave layers, each providing security boundaries.

The patent divides system software into separate enclaves that can be individually protected. Instead of protecting the entire system as one unit, the system is segmented into multiple isolated enclave regions, each with its own security context. This segmentation allows different parts of system software to have appropriate protection levels and enables granular security management.

The patent introduces a memory mapping mechanism that acts as an intermediary between enclaves. The processor's memory management unit translates enclave virtual addresses to physical addresses, enabling controlled memory sharing. This intermediary layer allows enclaves to access shared memory regions while maintaining security boundaries, as the translation process enforces access controls and isolation policies.

The patent creates a unified memory management system that serves multiple functions: it provides isolation between enclaves, enables controlled sharing of memory regions, and supports both user-mode and kernel-mode operations. The same memory management infrastructure handles private enclave memory, shared memory regions, and inter-enclave communication buffers, making the system versatile for different protection scenarios.

The patent implements dynamic privilege management where enclaves can transition between different privilege levels. Child enclaves can execute privileged instructions under the control of the root enclave, which manages the privilege transitions. This dynamic approach allows the system to provide privileged instruction execution when needed while maintaining security through controlled transitions and oversight by the root enclave.

The root enclave performs self-service by managing its own child enclaves and handling privileged operations. The root enclave executes privileged instructions to create, destroy, and manage child enclaves, and can perform operations that require higher privilege levels. This self-service capability allows the enclave system to provide privileged functionality without requiring external trust, as the root enclave autonomously manages its security context.