Sandbox Log Indexing for Malware Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The large number of program files that need to be analyzed for malware detection poses a significant challenge due to resource and time constraints with conventional analysis methods, making it impractical to efficiently detect malware in millions or tens of millions of files.
Innovation Solution
The system indexes sandbox logs using a distance function that measures dissimilarity between log files by creating an index tree, where logs are placed into buckets based on sequence lengths, and uses pruning strategies to efficiently search for similar behavior, allowing for the identification of malware patterns across large datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional analysis methods are used to detect malware in millions of files, then detection accuracy can be maintained, but resource consumption and analysis time become impractical
Solution Approach 1:
The patent segments the large-scale malware detection problem into multiple phases: (1) log collection and indexing phase where logs are organized into buckets by sequence length, (2) distance computation phase where dissimilarity is calculated only for relevant log pairs, and (3) search phase where pruning eliminates unnecessary comparisons. This segmentation allows the system to handle millions of files by processing them in manageable chunks rather than attempting exhaustive analysis of all files simultaneously.
Solution Approach 2:
The patent performs preliminary actions by pre-computing and storing distance metrics between log files in an index structure before actual malware detection queries are executed. Logs are pre-processed to extract sequence features and compute dissimilarity scores, which are then stored for rapid retrieval. This preliminary indexing enables fast comparison during detection without re-computing distances from scratch, significantly reducing computational resources during production use.
2Measurement precision
If all log files are analyzed in detail for malware detection, then detection precision is improved, but analysis time increases significantly
Solution Approach 1:
The patent applies local quality by computing distance metrics selectively based on local characteristics of log files. Instead of performing exhaustive detailed analysis on all log pairs, the system computes distances only for logs that fall into the same or adjacent buckets based on sequence length. This localized computation focuses analytical resources on the most promising candidates while skipping obviously dissimilar logs, thereby maintaining detection precision for relevant cases while dramatically reducing overall analysis time.
Solution Approach 2:
The patent implements partial action by performing detailed analysis only on a subset of log files that are deemed relevant based on preliminary filtering. The system computes distance metrics and performs detailed comparison only for logs within a certain distance threshold or within the same bucket, rather than analyzing all possible log pairs. This partial analysis approach maintains sufficient precision for malware detection by focusing on the most likely matches while avoiding wasteful computation on clearly dissimilar logs.
3Reliability
If an exhaustive search is performed across all log files to find similar behavior, then detection completeness is improved, but computational complexity becomes unmanageable
Solution Approach 1:
The patent segments the search space by organizing logs into buckets based on sequence length characteristics. This segmentation transforms the exhaustive search problem into multiple smaller, manageable sub-searches within each bucket. The computational complexity is reduced from O(n²) for exhaustive comparison to approximately O(k²) where k is the average bucket size, making the system scalable to millions of files while maintaining detection completeness within each segment.
Solution Approach 2:
The patent performs preliminary organization of logs into buckets and pre-computation of distance metrics before the actual search process. This preliminary action creates an indexed structure that enables efficient retrieval and comparison. During the search phase, the system only needs to query relevant buckets and compare pre-computed distances, avoiding the need to perform exhaustive comparisons from scratch and thereby reducing computational complexity while maintaining completeness.
Solution Approach 3:
The patent implements a threshold-based approach where detailed search is performed only on logs that meet certain criteria (e.g., within a distance threshold or in the same bucket). This partial search action maintains detection completeness for relevant cases by ensuring all logs within the threshold are examined, while avoiding the computational burden of examining all possible log pairs. The system achieves sufficient completeness for malware detection without the full complexity of exhaustive search.
Data Source
AI summary
Systems and methods index and search log files created after execution of binaries. A plurality of log files each have one or more sequences. An index tree is created for the log files. A first log file is placed into a bucket of the index tree according to the lengths of the one or more sequences of the first log file. Remaining logs files are placed the index tree according to their respective sequence lengths. Each log becomes a representative in the bucket or associated with a representative in the bucket. The index tree can be searched, where an incurred distance and a remaining distance is maintained during the search. Nodes are pruned based, at least in part, on the incurred distance and the remaining distance.