Fake Exception Handler Detection via Stack Pointer Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods fail to effectively detect and prevent fake exception handlers resulting from malicious stack buffer overflows that overwrite Structured Exception Handling (SEH) records, as attackers can circumvent conventional techniques by creating fake exception registration records or exploiting compiler optimizations.
Innovation Solution
A fake exception handler detection system monitors the operating system's exception processing logic, scanning the first bytes of identified exception handlers for a return instruction and analyzing preceding instructions to determine if they modify the stack pointer, thereby distinguishing between legitimate and fake exception handlers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional SEH detection techniques (checking Next pointer termination value or checking for FinalExceptHandler) are used, then exception handling security is improved, but the system becomes vulnerable to fake exception record chains that circumvent these checks
Solution Approach 1:
The patent applies preliminary action by analyzing the exception handler code structure before execution occurs. Specifically, it scans the first N bytes of the exception handler to detect the presence of a return instruction and analyzes preceding instructions for stack pointer modifications that would shrink the stack. This preliminary analysis prevents fake exception handlers from executing by identifying their characteristic patterns before they can transfer control to malicious shellcode.
2Object-affected harmful factors
If stack buffer overflow protection mechanisms are implemented, then security against malicious code execution is improved, but the complexity of the system increases
Solution Approach 1:
The patent applies partial action by focusing the security analysis on only the critical first N bytes of the exception handler rather than analyzing the entire handler code. This partial scanning approach efficiently detects the characteristic patterns of fake exception handlers (return instruction within first N bytes combined with stack pointer shrinking) without requiring comprehensive analysis of all exception handling code, thus maintaining low system complexity while achieving high security effectiveness.
3Measurement precision
If exception handler validation is performed by scanning first bytes for return instructions and analyzing stack pointer modifications, then detection accuracy of fake exception handlers is improved, but the processing time for exception handling increases
Solution Approach 1:
The patent applies partial action by limiting the scanning operation to only the first N bytes of the exception handler rather than analyzing the complete handler code. This selective analysis approach maintains high detection accuracy for fake exception handlers while minimizing the processing time overhead, as the vast majority of legitimate exception handlers do not contain return instructions within the first N bytes, allowing for rapid identification and rejection of malicious handlers.
Data Source
AI summary
Fake exception handlers resulting from malicious stack buffer overflows that overwrite an exception handling record on the stack are detected. The operating system exception processing logic is monitored. Responsive to an exception occurring, an exception handler to be called by the monitored operating system exception processing logic is identified. A specific number of the first bytes of the identified exception handler are scanned to determine whether a return instruction is present therein. Instructions of the identified exception handler that are positioned prior to the return instruction are analyzed to determine whether they modify the value of the stack pointer so as to shrink the stack. The identified exception handler is adjudicated as being fake, responsive to determining that a return instruction is present in the first specific number of bytes of the exception handler and/or that the instructions positioned prior to the return instruction shrink the stack.