Binary Search Engine Using Inverted Byte Index for Malware Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current malware detection methods are inefficient in identifying disguised or obfuscated malicious software due to the time-consuming process of determining if a program is malware, as malware authors often disguise their programs to evade detection.
Innovation Solution
A binary search engine utilizing an inverted index of byte sequences to search for and identify malware by generating signatures and fuzzy hashes, allowing for efficient detection of malicious files through the analysis of byte sequences and logical operators within a corpus of files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional malware detection methods are used to analyze programs, then detection accuracy can be maintained, but the process becomes time-consuming and inefficient
Solution Approach 1:
The system pre-computes and stores byte sequence signatures and fuzzy hashes in inverted indices during an offline phase. When malware detection is needed, the system performs rapid lookups in these pre-built indices rather than analyzing entire programs from scratch, dramatically reducing detection time while maintaining accuracy.
Solution Approach 2:
The patent replaces traditional mechanical program analysis with a data structure-based lookup system using inverted indices. Instead of sequentially examining program instructions, the system substitutes this with efficient byte sequence matching and signature comparison operations that leverage pre-computed data structures.
2Adaptability or versatility
If malware authors disguise or obfuscate their programs, then they can evade detection, but this increases the complexity of determining if a program is malware
Solution Approach 1:
The system transforms the detection approach by changing parameters from high-level program behavior analysis to low-level byte sequence patterns. By operating at the byte level and using multiple hash functions (MD5, SHA1, SHA256), the system can detect disguised malware through pattern matching rather than relying on recognizable program structures.
Solution Approach 2:
The patent combines multiple detection techniques into a composite system: traditional signature-based detection, fuzzy hashing with multiple algorithms, and inverted index technology. This composite approach maintains detection effectiveness against disguised malware while managing complexity through integrated architecture.
3Productivity
If the system analyzes byte sequences and generates signatures for every file, then malware detection efficiency improves, but the computational resources and processing requirements increase
Solution Approach 1:
The system segments the file analysis process into distinct phases: offline pre-computation of byte sequences and signature generation, and online rapid lookup using inverted indices. This segmentation allows resource-intensive operations to be performed once during indexing, while detection operations consume minimal resources by leveraging pre-computed data structures.
Solution Approach 2:
The system performs preliminary analysis by pre-computing byte sequences, fuzzy hashes, and signatures during an offline indexing phase. This preliminary action stores results in inverted indices so that subsequent malware detection requires minimal computational resources, achieving high productivity without continuous high energy consumption.
Data Source
AI summary
Techniques for searching an inverted index associating byte sequences of a fixed length and files that contain those byte sequences are described herein. Byte sequences comprising a search query are determined and searched in the inverted index. In some examples, training data for training machine learning model(s) may be created using pre-featured data from the inverted index. In various examples, training data may be used to retrain a ML model until the ML model meets a criterion. In some examples, the trained ML model may be used to perform searches on the inverted index and classify files.