Inverted Byte-Sequence Search for Obfuscated Malware Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing malware detection methods are inefficient in identifying disguised or obfuscated malicious software due to the time-consuming process of determining if a program is malware, as adversaries frequently disguise or obfuscate malware to evade detection.
Innovation Solution
A binary search engine utilizing an inverted index of byte sequences to efficiently search and identify malware by generating signatures and fuzzy hashes, and employing machine learning models to enhance detection accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional malware detection methods are used, then detection accuracy may be maintained, but detection time increases significantly and productivity decreases
Solution Approach 1:
The system performs preliminary actions by pre-computing and storing byte sequence signatures and fuzzy hashes in inverted indices during an indexing phase. When malware detection is needed, the system queries these pre-computed indices rather than analyzing files from scratch, dramatically reducing detection time while maintaining accuracy.
Solution Approach 2:
The patent replaces traditional mechanical file-by-file analysis with a database query-based system using inverted indices. Instead of sequentially examining file contents, the system substitutes this with efficient index lookups and set operations, transforming the detection mechanism from linear scanning to logarithmic or constant-time queries.
2Reliability
If malware is heavily disguised or obfuscated, then adversaries can evade detection, but this increases the complexity of detection methods required
Solution Approach 1:
The system employs multiple parameter representations of the same malware: exact byte sequence signatures for precise matching, and fuzzy hashes that tolerate obfuscation. By changing the parameters of comparison (from exact matches to similarity-based matches), the system maintains reliability against disguised malware without requiring overly complex analysis tools.
Solution Approach 2:
The detection system uses a composite approach combining multiple detection techniques: inverted indices for exact signature matching, fuzzy hash computation for similarity detection, and machine learning models for pattern recognition. This composite methodology handles obfuscated malware effectively while keeping individual components relatively simple.
3Measurement precision
If traditional signature-based detection is used, then false positives may be reduced, but detection coverage decreases when malware is obfuscated
Solution Approach 1:
The inverted index structure serves multiple functions: it stores exact byte sequence signatures for precise detection, supports fuzzy hash lookups for obfuscated malware, and provides the foundation for machine learning model training. This multi-functional system maintains precision for known malware while adapting to detect disguised variants.
Solution Approach 2:
The patent introduces fuzzy hashes as an intermediary between exact signatures and full file analysis. Fuzzy hashes act as a mediator that captures essential malware characteristics while being tolerant of obfuscation, bridging the gap between precise signature matching and adaptable detection of modified malware.
Data Source
AI summary
Techniques for searching an inverted index associating byte sequences of a fixed length and files that contain those byte sequences are described herein. Byte sequences comprising a search query are determined and searched in the inverted index. In some examples, training data for training machine learning (ML) model(s) may be created using pre-featured data from the inverted index. In various examples, training data may be used to retrain the ML model until the ML model meets a criterion. In some examples, the trained ML model may be used to perform searches on the inverted index and classify files.