Scalable Software Sample Clustering via Multi-Stage Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Clustering software samples based on their features is challenging due to the large number of features and varying similarities, leading to scalability issues in processing time, storage space, and computation efficiency.
Innovation Solution
A method that disassembles software samples, applies hashing (using SPP-NOP and mini-hashing), and performs similarity calculations to reduce the number of comparisons, employing stride hashing and grouping to efficiently cluster software samples, thereby reducing storage and CPU cycles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional clustering methods are used to process software samples with many features, then clustering accuracy can be maintained, but processing time increases significantly and scalability is reduced
Solution Approach 1:
The patent segments the clustering process into multiple stages: (1) computing first hash results for all software samples, (2) computing second hash results for selected samples, (3) computing third hash results for further refinement, and (4) performing similarity calculations only on reduced candidate sets. This segmentation allows the system to maintain clustering accuracy while dramatically reducing processing time by avoiding exhaustive comparisons of all samples.
Solution Approach 2:
The patent applies preliminary hashing operations to software samples before the actual clustering process. By computing first, second, and third hash results in advance, the system pre-processes the data to create compact representations that enable faster similarity calculations later, thus reducing the time required for the main clustering operation.
2Loss of information
If all features of software samples are stored and processed, then comprehensive clustering analysis is achieved, but storage space requirements increase significantly
Solution Approach 1:
The patent creates hash-based copies of software sample features instead of storing the complete feature sets. The first, second, and third hash results serve as compact representations that preserve the essential information needed for clustering while occupying minimal storage space, thus maintaining feature completeness information while reducing storage requirements.
Solution Approach 2:
The patent transforms the feature representation from raw feature data to hashed values through multiple hashing stages. This parameter transformation compresses the feature space significantly, allowing the system to retain the essential clustering information in a much more space-efficient format.
3Measurement precision
If similarity calculations are performed on all software sample pairs, then accurate clustering results are obtained, but CPU cycle consumption becomes prohibitive for large datasets
Solution Approach 1:
The patent segments the similarity calculation process by first identifying candidate pairs using hash result comparisons, then performing detailed similarity calculations only on these reduced candidate sets. This segmentation avoids the computationally prohibitive exhaustive comparison of all sample pairs while maintaining calculation accuracy for the relevant subsets.
Solution Approach 2:
The patent performs similarity calculations partially by limiting the scope to candidate pairs identified through hash-based filtering rather than calculating similarities for all possible pairs. This partial action is sufficient to achieve accurate clustering results without the excessive computational cost of complete pairwise comparison.
4Productivity
If multiple hashing stages are applied to software samples, then computational efficiency and storage reduction are achieved, but system complexity increases
Solution Approach 1:
The patent segments the hashing process into distinct stages (first hash, second hash, third hash), where each stage serves a specific purpose in reducing the data size and identifying candidate pairs. This segmentation makes the complex multi-stage hashing process more manageable and allows for optimized implementation of each individual hashing stage.
Data Source
AI summary
Systems, methods, and software can be used to cluster software codes in a scalable manner. In some aspects, a computer-implemented method comprises: obtaining a plurality of software samples; computing one or more first hash results for each of the plurality of software samples; computing one or more second hash results for each of the plurality of software samples based on the one or more first hash results, wherein an amount of the one or more second hash results is less than an amount of the one or more first hash results; determining a similarity output based on the one or more second hash results of two of the plurality of software samples; and clustering the plurality of software samples based on the similarity output to generate one or more software sample clusters.