Binary Search Engine Inverted Index for Malware Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Malware detection is time-consuming and challenging due to the disguising and obfuscation techniques used by malware authors, making it difficult to determine if a program is malicious using existing detection tools.
Innovation Solution
A binary search engine and inverted index system that searches for byte sequences in files, generates signatures, and creates fuzzy hashes to efficiently identify malware by associating byte sequences with file identifiers and evaluating search results using logical operators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional malware detection tools are used, then detection accuracy can be maintained, but detection time increases significantly due to obfuscation techniques
Solution Approach 1:
The system pre-generates all possible byte sequences of a specified length (e.g., 4-byte sequences) and creates an inverted index mapping each byte sequence to the files containing it before malware detection is needed. This preliminary indexing allows rapid lookup during actual detection without time-consuming analysis of obfuscated malware
Solution Approach 2:
The inverted index serves as an intermediary data structure that bridges the search query (byte sequences from malware samples) and the file database. Instead of directly scanning files for malware patterns, the system uses the inverted index to quickly retrieve candidate files containing specific byte sequences, dramatically reducing detection time while maintaining accuracy
2Reliability
If comprehensive file scanning is performed to ensure accurate malware detection, then detection reliability improves, but processing speed decreases
Solution Approach 1:
The system segments the malware detection process into distinct phases: (1) extracting byte sequences from malware samples, (2) querying the inverted index for candidate files, and (3) validating results. This segmentation allows each phase to be optimized independently, with the inverted index handling rapid candidate retrieval to maintain high throughput while reliability is ensured in the validation phase
3Measurement precision
If detailed analysis of malware byte sequences is performed to improve detection precision, then measurement precision improves, but device complexity increases
Solution Approach 1:
Instead of implementing complex real-time analysis algorithms, the system creates a simplified copy of the file system structure in the form of an inverted index. This index copies only the essential mapping information (byte sequence to file identifiers) needed for detection, reducing computational complexity while maintaining precise matching capability through direct lookup
Data Source
AI summary
Techniques for searching an inverted index associating byte sequences of a fixed length and files that contain those byte sequences are described herein. Byte sequences comprising a search query are determined and searched in the inverted index. In some examples, the inverted index may be distributed across multiple computers and the search may be performed in parallel. In some examples, a search query may be submitted as expressions comprising query language or regular expressions that are interpreted as search terms, transformed into byte sequences, and searched for in the inverted index. In some examples, an automatic notification request for a search query may be processed and notifications may be sent based on a default or preferred frequency and method.