Emulator Malicious App Detection via API Return Value Checks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Sophisticated malicious applications can detect execution in emulators by checking API call return values, exploiting differences in functionality between real-world and emulated environments, thus avoiding detection.
Innovation Solution
The method involves detecting API calls and checks on variable return values by malicious applications, identifying invalid hardcoded parameters and non-standard checks to determine if an application is malicious, even if it conceals its behavior.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If an emulator is used to execute and monitor applications for malicious behavior, then security detection capability is improved, but sophisticated malicious applications can detect the emulated environment through API call checks and avoid exhibiting malicious behavior
Solution Approach 1:
Instead of waiting for malicious applications to exhibit malicious behaviors, the system inverts the detection approach by proactively identifying applications that perform environment-detecting API calls. The emulator monitors for specific API call patterns (such as checking return values from environment-identifying APIs) and flags applications that perform such checks, regardless of whether they exhibit traditional malicious behaviors. This inversion transforms the detection paradigm from behavior-based to pattern-based, effectively countering evasion strategies.
Solution Approach 2:
The system performs preliminary detection of environment-detecting API calls before malicious behaviors can be concealed. By monitoring and analyzing API call patterns during the initial execution phase in the emulator, the system identifies suspicious applications early in the quarantine period, before they have an opportunity to adapt their behavior or hide their malicious intent. This preliminary action prevents sophisticated malware from successfully evading detection through behavioral adaptation.
2Measurement precision
If the emulator monitors all application behaviors to detect malicious payloads, then detection thoroughness is improved, but detection complexity and resource consumption increase
Solution Approach 1:
The system extracts and focuses monitoring efforts on specific, high-value API call patterns that are indicative of environment-detection attempts. Rather than analyzing all application behaviors equally, the emulator identifies and extracts particular API calls (such as those that query system information, check environment variables, or validate return values in suspicious ways) for intensified monitoring. This selective extraction reduces the overall monitoring complexity while maintaining high detection precision for sophisticated malware.
Solution Approach 2:
The detection system applies different monitoring intensities to different parts of the application execution process. Specifically, it applies enhanced monitoring and analysis to API call sequences and return value checks, while using standard monitoring for other application behaviors. This localized quality enhancement focuses computational resources on the most critical detection points, reducing overall system complexity while improving detection effectiveness for environment-aware malware.
Data Source
AI summary
Detecting a malicious application executing in an emulator based on a check made by the malicious application after making an API call. In one embodiment, a method may include executing an application in an emulator that emulates a real-world computing environment. The method may also include detecting, in the application, an API call configured to accept a parameter and return a variable return value to a return address in the application. The method may further include detecting, at the return address, a check to be performed on the variable return value returned by the API call. The method may also include, in response to the detecting of the check, determining that the application is malicious. The method may further include performing a security action on the malicious application to prevent the malicious application from executing in the real-world computing environment.