Hook-Based Malicious Activity Detection During File Unpacking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for detecting malicious software in computer systems are ineffective in identifying threats during the unpacking process, as they either result in false positives or fail to detect threats in time to prevent malicious payload execution, due to timing issues and the cyclic advancement of packers/encryptors and unpackers/decryptors.
Innovation Solution
A method that involves inserting a hook function into processes to suspend system calls at optimal times during initial execution stages, allowing for attribute detection and comparison with known malicious signatures, and subsequent blocking or notification of potentially malicious processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If file scanning with signature-based detection is used, then known threats can be detected, but unknown threats packed with new packers/encryptors cannot be detected in time
Solution Approach 1:
The system performs preliminary actions by intercepting system calls during the unpacking process itself, before the malicious payload executes. By placing hooks on system calls made during unpacking (such as creating mutexes, registry keys, or network connections), the system can detect and block unknown threats at the moment they are being unpacked, rather than waiting for signature-based detection after execution begins.
Solution Approach 2:
The system introduces an intermediary layer by inserting hook functions that intercept and monitor system calls during the unpacking process. These hooks act as mediators between the unpacking process and the detection system, allowing real-time monitoring of unpacking behavior without requiring the unpacked code to be fully executed or signed before detection can occur.
2Reliability
If proactive detection by intercepting system calls is used, then unknown threats can be blocked, but false positives increase due to triggering on legitimate software behaviors
Solution Approach 1:
The system applies local quality by differentiating between types of system calls based on their specific characteristics and contexts. Rather than treating all system calls uniformly, the system analyzes the specific parameters, timing, and patterns of each call to determine whether it represents legitimate software behavior or malicious unpacking activity, allowing selective blocking that reduces false positives.
Solution Approach 2:
The system utilizes parameter changes by monitoring specific parameters of system calls during unpacking, such as the timing, frequency, and specific values of parameters passed to functions like CreateMutex, RegOpenKey, or network connection calls. By analyzing changes in these parameters against established patterns of legitimate behavior, the system can distinguish between benign operations and malicious activity more accurately.
3Object-generated harmful factors
If memory scanning is performed after unpacking, then known threats can be detected with low false positives, but the malicious payload may have already executed
Solution Approach 1:
The system performs preliminary detection actions by intercepting and monitoring system calls during the unpacking process itself, before the unpacked malicious payload has a chance to execute. This preliminary monitoring allows the system to identify and block unknown threats at the critical moment when they are being unpacked, eliminating the timing problem where detection occurs after execution begins.
Solution Approach 2:
The system maintains continuous monitoring during the unpacking process by keeping hooks active on system calls throughout the unpacking sequence. This continuous action ensures that no critical moment is missed, allowing real-time detection and blocking of unknown threats as they are unpacked, rather than performing discrete scans at fixed intervals after unpacking completes.
Data Source
AI summary
A method of detecting and blocking malicious activity of processes in computer memory during unpacking of a file after the code and data contained in the file are unpacked is described. The method includes inserting a hook function into one or more un-assessed processes running in the computer memory. A hook is then placed on one or more system calls carried out by the one or more un-assessed processes; the one or more system calls determining an optimal time period in which to detect malicious activity in the un-assessed processes. During the optimal time period the one or more system calls carried out by the one or more un-assessed processes are suspended and attributes of the one or more un-assessed processes are detected and the likely maliciousness of the one or more un-assessed processes is determined from the attributes.