Detecting Return-to-LIBC Buffer Overflows via Dynamic Disassembly
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Return-to-LIBC attacks exploit buffer overflow vulnerabilities by overwriting return addresses to execute malicious library functions, compromising system security, as existing methods fail to differentiate between legitimate and malicious call modules.
Innovation Solution
The method hooks critical OS functions, stalls calls, and checks if the initiating call module was called by a call instruction; if not, it takes protective action to prevent malicious execution by terminating the call to the critical OS function.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing buffer overflow protection methods are used, then system security is partially improved, but they fail to differentiate between legitimate and malicious call modules, leading to false positives and inability to detect RLIBC attacks
Solution Approach 1:
The patent applies local quality by examining the specific calling convention (call instruction vs. jump/return instruction) at the local level where the function is invoked. Instead of applying a blanket protection method to all function calls, the system specifically checks whether the call module was invoked using a proper call instruction, which is the local characteristic that distinguishes legitimate from malicious calls in RLIBC attacks.
Solution Approach 2:
The patent uses inversion by reversing the traditional detection approach. Instead of trying to detect malicious characteristics directly, the system checks for the absence of legitimate characteristics (proper call instruction). By verifying that a call module was NOT called by a call instruction, the system identifies RLIBC attacks, effectively detecting threats by looking for what should NOT be present.
2Productivity
If no protection is applied, then system performance is maintained, but the system becomes vulnerable to RLIBC attacks and buffer overflow exploits
Solution Approach 1:
The patent applies preliminary action by performing the security check before the critical OS function executes. The system hooks the function call and checks whether the call module was properly invoked by a call instruction before allowing execution to proceed. This preemptive verification prevents RLIBC attacks from succeeding while maintaining system performance for legitimate calls.
Solution Approach 2:
The patent uses an intermediary approach by introducing a hook function as a mediator between the call module and the critical OS function. This intermediary layer intercepts the function call, performs the security check regarding call instruction usage, and then decides whether to allow the call to proceed. This mediator enables security verification without significantly impacting the performance of legitimate operations.
3Measurement precision
If comprehensive security checks are implemented, then detection accuracy is improved, but system complexity increases due to hooking and stalling mechanisms
Solution Approach 1:
The patent applies taking out by extracting the security verification logic from the main execution flow and implementing it as a separate hook function. This extracted security check mechanism handles the complexity of detecting RLIBC attacks independently, while the main system continues to operate with minimal disruption. The complex detection logic is isolated in the hook, reducing the complexity burden on the overall system.
Data Source
AI summary
A method makes use of the fact that call modules, such as APIS, making calls to a critical operating system (OS) function are typically called by a call instruction while, in contrast, a RLIBC attack typically uses call modules that are jumped to, returned to, or invoked by some means other than a call instruction. The method includes stalling a call to critical OS function and checking to ensure that the call module making the call to the critical OS function was called by a call instruction. If it is determined that the call module making the call to the critical OS function was not called by a call instruction, the method further includes taking protective action to protect a computer system.