Shellcode Detection via Memory Attribute Change Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing exploit detection techniques are limited in effectiveness as they fail to detect malicious shellcode that bypasses data execution prevention (DEP) by using memory attribute change functions, allowing exploits to execute shellcode undetected.
Innovation Solution
A system and method for detecting exploits by monitoring memory blocks during dynamic analysis, identifying suspicious shellcode, dumping memory blocks to find a candidate entry point, and executing the shellcode using a CPU or full system emulator to determine its malicious nature, generating a verdict based on the execution results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data execution prevention (DEP) is used to prevent shellcode execution, then security is improved, but exploit detection capability deteriorates because exploits can bypass DEP using memory attribute change functions
Solution Approach 1:
The system performs preliminary monitoring of memory attribute change functions before shellcode execution can occur. By detecting calls to VirtualProtect, VirtualAlloc, or other memory attribute change functions, the system can identify exploits attempting to bypass DEP protections and prevent shellcode execution in advance.
Solution Approach 2:
The system introduces an intermediary detection layer that monitors memory attribute change functions as intermediaries between the exploit and the shellcode execution. This intermediary monitoring mechanism detects the memory protection changes without directly executing the shellcode, thereby maintaining both DEP effectiveness and exploit detection capability.
2Difficulty of detecting and measuring
If memory attribute change functions are monitored to detect exploits, then exploit detection is improved, but system complexity increases due to additional monitoring and analysis mechanisms
Solution Approach 1:
The monitoring system is designed to handle multiple exploit types and memory attribute change functions through a unified framework. The same monitoring mechanism detects various memory protection changes (VirtualProtect, VirtualAlloc, RtlProtectMemory, etc.) and analyzes different shellcode patterns, reducing the need for separate detection systems for each exploit variant.
Solution Approach 2:
The system creates a virtual copy of the memory space and executes shellcode in this virtual environment rather than the actual system memory. This virtualization allows the monitoring system to analyze shellcode behavior without directly interacting with the real system, simplifying the monitoring mechanism while improving detection accuracy.
3Measurement precision
If shellcode is executed in a virtual environment for analysis, then detection accuracy is improved, but execution time increases due to emulation overhead
Solution Approach 1:
The system performs partial execution analysis by monitoring specific memory attribute change functions and shellcode patterns rather than executing the entire shellcode in a full virtual environment. This selective execution approach maintains sufficient detection accuracy while significantly reducing the time required for analysis compared to complete virtualization.
Data Source
AI summary
Detection of an exploit including shellcode is disclosed. Memory blocks are monitored during dynamic analysis of a sample to identify a memory block including suspicious shellcode. The memory block is dumped in memory to identify a candidate shellcode entry point associated with the suspicious shellcode. The suspicious shellcode is executed based on the candidate shellcode entry point to determine whether the suspicious shellcode is malicious. A verdict is generated regarding the sample based on results of executing the suspicious shellcode.