Emulating OS Rings to Detect Malicious Code Behavior
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer networks and systems are vulnerable to attacks through return-oriented programming (ROP) and other malicious code execution methods, where attackers exploit existing program code to gain control without injecting new code, posing a challenge in effectively detecting and mitigating such threats.
Innovation Solution
The solution involves emulating the first and second rings of an operating system, where the second ring has greater access, and executing code payloads in an emulated environment to observe and analyze their behavior without actually performing system functions, using a separate layer to emulate system calls and interrupts, and instrumenting both layers to detect suspect behavior.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If code payloads are executed in a real operating system environment, then the behavior of the code can be observed and analyzed, but the host platform is at risk from malicious code actions
Solution Approach 1:
The patent introduces an emulator as an intermediary layer between the code payload and the host platform. The emulator executes the code payload in a simulated environment, allowing behavior observation without direct execution on the host system. This mediator isolates the host from harmful effects while preserving the ability to observe code behavior through the emulation layer.
Solution Approach 2:
The patent creates a virtual copy of the operating system environment through emulation. Instead of executing code directly on the real OS, the system executes it on a virtualized copy that mimics the behavior of the real system. This copying approach allows safe observation of code behavior while protecting the original system from malicious actions.
2Adaptability or versatility
If the emulator fully performs system functions, then the code executes more realistically, but the detection capability is reduced because the emulator cannot distinguish real from emulated behavior
Solution Approach 1:
The emulator implements a partial emulation strategy where only certain system functions are fully performed while others are simulated or discarded. This partial action approach maintains enough realism for code execution while introducing detectable patterns that help identify malicious behavior. The emulator selectively performs actions based on detection needs rather than fully implementing all system functions.
Solution Approach 2:
The patent implements feedback mechanisms where the emulator monitors its own operations and the code payload's behavior. By observing patterns in system call sequences, execution flows, and resource access, the emulator can detect anomalies that indicate malicious code. This feedback loop enables detection capability while maintaining execution realism through controlled emulation.
3Productivity
If system calls and interrupts are actually performed, then the code executes with full functionality, but the host system is exposed to malicious actions
Solution Approach 1:
The emulator acts as an intermediary that intercepts system calls and interrupts before they reach the host system. It processes these calls in the virtualized environment, performing necessary functions while blocking harmful actions from reaching the real system. This mediator approach preserves code execution functionality while protecting the host from malicious effects.
Solution Approach 2:
The patent extracts the system call and interrupt handling from the host system and relocates it to the emulator layer. By taking out these critical system interactions from the host environment and processing them in the controlled emulation layer, the system maintains full code functionality while isolating the host from potential malicious actions.
Data Source
AI summary
A system, method and media are shown for emulating potentially malicious code involving emulating a first ring of an operating system, emulating a second ring of the operating system, where the second ring has greater access to system resources than the first ring and where the first and second rings are separately emulated, executing a code payload in the emulated first ring, checking the behavior of the executing code payload for suspect behavior, and identifying the code payload as malicious code if suspect behavior is detected. Some examples emulate the second ring by operating system or microarchitecture functionality such that the second ring emulation returns results to the executing code payload, but does not actually perform the functionality in a host platform. Some examples execute the code payload in the emulated first shell at one or more offsets.