Shellcode Execution Detection for DEP Bypass Exploits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing exploit detection techniques are limited in effectiveness as they fail to detect malicious shellcode that bypasses data execution prevention (DEP) by using memory attribute change functions, allowing exploits to execute shellcode undetected.
Innovation Solution
A system and method for detecting exploits by monitoring memory blocks during dynamic analysis, identifying suspicious shellcode, dumping memory blocks to find a candidate entry point, and executing the shellcode using a CPU or full system emulator to determine its malicious nature, generating a verdict based on the execution results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data execution prevention (DEP) is used to prevent shellcode execution, then security is improved, but exploit detection capability deteriorates because exploits can bypass DEP using memory attribute change functions
Solution Approach 1:
The system performs preliminary monitoring of memory attribute change functions before shellcode execution can occur. By detecting calls to VirtualProtect, VirtualAlloc, or other memory attribute change functions, the system identifies exploits attempting to bypass DEP protections before they can successfully execute malicious shellcode.
Solution Approach 2:
The system introduces an intermediary detection layer that monitors memory attribute change functions as intermediaries between the exploit and the shellcode execution. This intermediary monitoring mechanism detects the exploitation attempt itself, separating the detection of the bypass mechanism from the detection of the actual malicious payload execution.
2Difficulty of detecting and measuring
If memory attribute change functions are monitored to detect exploits, then exploit detection capability is improved, but system complexity increases
Solution Approach 1:
The system implements a universal monitoring framework that handles multiple types of memory attribute change functions through a single detection mechanism. The same monitoring infrastructure detects various bypass techniques (VirtualProtect, VirtualAlloc, RtlMoveMemory, etc.) without requiring separate detection logic for each function, reducing overall system complexity.
Solution Approach 2:
The detection system segments the monitoring process into distinct components: identifying memory attribute change function calls, extracting shellcode from affected memory regions, and analyzing the shellcode for malicious behavior. This segmentation allows each component to be optimized independently and simplifies the overall system architecture.
3Measurement precision
If shellcode is executed in emulators for analysis, then detection accuracy is improved, but analysis time increases
Solution Approach 1:
The system performs preliminary static analysis of shellcode before emulator execution to identify obvious malicious patterns and filter out benign code. This preliminary filtering reduces the number of shellcode samples that require time-consuming emulator execution, thereby reducing overall analysis time while maintaining high detection accuracy for malicious payloads.
Solution Approach 2:
The system applies partial emulator execution rather than complete execution analysis. By executing only the critical portions of shellcode or using lightweight emulator modes for initial analysis, the system achieves sufficient detection accuracy without the full time cost of complete execution tracing, balancing accuracy and performance.
Data Source
AI summary
Detection of an exploit including shellcode is disclosed. Memory blocks are monitored during dynamic analysis of a sample to identify a memory block including suspicious shellcode. The memory block is dumped in memory to identify a candidate shellcode entry point associated with the suspicious shellcode. The suspicious shellcode is executed based on the candidate shellcode entry point to determine whether the suspicious shellcode is malicious. A verdict is generated regarding the sample based on results of executing the suspicious shellcode.