Direct Syscall Hooking for Malicious Call Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing security solutions fail to detect malicious direct system calls (syscalls) effectively, allowing malware to bypass conventional methods and evade detection.
Innovation Solution
Deploy a hooked version of the syscall in the computer's memory, analyze the return address and extracted features to classify calls as benign or malicious, generating alerts for potential threats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional security solutions are used to detect malicious syscalls, then the system maintains simplicity and compatibility, but detection effectiveness fails to identify malicious direct syscalls
Solution Approach 1:
The patent introduces a hooking mechanism as an intermediary layer between user-mode applications and kernel-mode syscalls. This hook allows the security system to intercept and analyze syscall parameters without modifying the core syscall implementation, thereby improving detection effectiveness while maintaining system compatibility and minimizing complexity
Solution Approach 2:
The patent segments the syscall detection process into distinct components: parameter extraction, hashing, comparison against known malicious patterns, and alert generation. This modular approach improves detection reliability while allowing each component to be optimized independently, managing overall system complexity
2Speed
If direct syscall calls are analyzed in real-time, then malicious calls can be identified promptly, but the analysis complexity increases due to examining return addresses and multiple features
Solution Approach 1:
The patent transforms complex syscall parameter analysis into simplified hash value comparisons. By converting syscall parameters into hash values and comparing against pre-computed malicious patterns, the system achieves fast real-time detection without the computational overhead of detailed parameter analysis, balancing speed and complexity
Solution Approach 2:
The patent performs preliminary hashing of syscall parameters at the hook level before full analysis. This preliminary action filters out benign calls early in the process, allowing only suspicious calls to undergo more detailed analysis, thereby maintaining high detection speed while reducing overall analysis complexity
Data Source
AI summary
Methods, storage systems and computer program products implement embodiments of the present invention for protecting a computer by first deploying in a memory of the computer a hooked version of a syscall used by an operating system kernel of the computer A notification of a call to the hooked version of the syscall from a user mode of the computer is received from the hooked version of the syscall, the notification including a return address in the memory and a set of features extracted from the call. The return address and the received features are analyzed so as to classify the call as benign or malicious, and an alert is generated for the computer upon classifying the new call as malicious.