Dynamic Decimation of Token N-grams for Code Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for identifying code similarity and detecting malware or related code variants in binaries and text samples face challenges in achieving rapid bulk processing speed and library capacity, limiting their effectiveness in addressing the need for efficient malware detection and code provenance analysis.
Innovation Solution
The proposed solution involves generating 'fingerprints' or 'DNA' strings through lossy compression of binary code, which are then processed using n-gram based scoring and dynamic decimation to efficiently identify similarities, allowing for scalable and efficient comparison of code variants and malware detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional n-gram based similarity detection methods are used, then code similarity can be identified, but processing speed and library capacity are insufficient for rapid bulk processing
Solution Approach 1:
The patent segments the code into fixed-length substrings called n-grams, where each n-gram represents a contiguous sequence of characters or tokens. This segmentation transforms the code into a collection of discrete units that can be processed independently and efficiently, enabling rapid bulk processing while maintaining similarity detection capability through statistical comparison of n-gram distributions
Solution Approach 2:
The patent changes the parameter of n-gram length from variable to fixed, and introduces a decimation factor to control the density of n-grams retained for comparison. By adjusting these parameters, the system can optimize between processing speed (higher decimation) and similarity detection precision (lower decimation), resolving the contradiction through parameter tuning
2Measurement precision
If all n-grams are retained for comparison, then similarity detection precision is maintained, but memory requirements and processing overhead increase significantly
Solution Approach 1:
The patent extracts only a subset of n-grams from the full set by applying a decimation factor, which selectively retains certain n-grams while discarding others. This extraction process reduces the quantity of data stored in memory and processed during comparison, while the extracted subset maintains sufficient statistical properties for accurate similarity detection
Solution Approach 2:
The patent applies partial action by retaining only a fraction (1/F where F is the decimation factor) of the total n-grams for comparison. This partial retention is sufficient to maintain similarity detection precision while dramatically reducing memory usage and processing overhead, especially for large code bases
3Productivity
If dynamic decimation with variable factors is applied, then processing efficiency improves, but system complexity increases
Solution Approach 1:
The patent introduces dynamic decimation where the decimation factor F varies based on the size and characteristics of the code being processed. For larger code bases, higher decimation factors are applied to maintain efficiency, while smaller codes use lower factors to preserve precision. This dynamic adjustment optimizes processing efficiency across different scenarios without requiring fundamentally different systems
Data Source
AI summary
An apparatus for identifying related code variants or text samples includes processing circuitry configured to execute instructions for receiving query binary code, processing the query binary code to generate one or more query code fingerprints comprising compressed representations of respective functional components of the query binary code, generating token sequence n-grams of the fingerprints, hashing the n-grams, partitioning samples by length to compare selected samples based on length, and identifying similarity via dynamic decimation of token sequence n-grams.