Binary Search Engine Using Inverted Byte Index for Malware Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Malware detection is hindered by the difficulty in identifying disguised or obfuscated malicious software, leading to time-consuming processes in determining if a program is malware.
Innovation Solution
A binary search engine utilizing an inverted index that associates byte sequences with files, enabling efficient searching, signature generation, and fuzzy hash creation to identify malware by processing byte sequences and evaluating search results through a syntax tree, thereby facilitating quicker detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional malware detection methods are used to identify disguised or obfuscated malicious software, then detection accuracy may be maintained, but the process becomes time-consuming and less efficient
Solution Approach 1:
The patent segments malware detection into multiple independent components: creating n-gram byte sequences from file contents, building inverted indexes for rapid lookup, generating signatures from suspicious patterns, and creating fuzzy hashes for variant detection. Each component operates independently and can be processed in parallel, dramatically improving detection speed while maintaining accuracy against obfuscated malware
Solution Approach 2:
The patent performs preliminary actions by pre-processing file contents into n-gram byte sequences and pre-building inverted indexes that map byte sequences to file identifiers. These preliminary structures are created before actual detection queries, enabling rapid lookup and comparison during the detection phase without time-consuming real-time analysis of raw file contents
2Reliability
If comprehensive malware analysis is performed to ensure accurate detection, then detection reliability improves, but the time required to determine if a program is malware increases
Solution Approach 1:
The patent applies partial action by not analyzing every byte of every file in detail. Instead, it extracts and analyzes only relevant n-gram byte sequences that appear in the inverted index, generating signatures and fuzzy hashes for suspicious patterns. This partial analysis approach maintains high detection reliability by focusing on discriminative features while avoiding time-consuming exhaustive analysis of entire files
3Adaptability or versatility
If byte sequences are extracted and processed to generate signatures and fuzzy hashes, then malware identification capability improves, but the complexity of the detection system increases
Solution Approach 1:
The patent introduces several intermediary structures to bridge the gap between raw file contents and malware detection: n-gram byte sequences as intermediaries between files and signatures, inverted indexes as intermediaries between byte sequences and file identifiers, and fuzzy hashes as intermediaries for detecting malware variants. These intermediaries simplify the overall system architecture by breaking down complex detection tasks into manageable lookup and comparison operations
Data Source
Figure 1a
Figure 1b
Figure 1c
AI summary
Techniques for searching an inverted index associating byte sequences of a fixed length and files that contain those byte sequences are described herein. Byte sequences comprising a search query are determined and searched in the inverted index. In some examples, the inverted index may be distributed across multiple computers and the search may be performed in parallel. In some examples, a search query may be submitted as expressions comprising query language or regular expressions that are interpreted as search terms, transformed into byte sequences, and searched for in the inverted index. In some examples, an automatic notification request for a search query may be processed and notifications may be sent based on a default or preferred frequency and method.