String Clustering Using Hash Tables for Scalable Object Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional malware, spam, and document classification approaches face challenges with high false positive rates and scalability issues when dealing with large numbers of objects, leading to inefficient performance and memory consumption.
Innovation Solution
A novel technique using hash tables and rolling hash functions to efficiently group strings into clusters by determining an evaluation function between pairs of strings, allowing for scalable object classification in anti-malware, anti-spam, and data leakage prevention systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional malware classification based on behaviors is used, then classification can be performed, but false positive identification rate increases
Solution Approach 1:
The patent transforms malware classification from behavior-based parameters to signature-based parameters using hash tables. By changing the classification parameter from dynamic behavior analysis to static signature matching, the system achieves more reliable classification with reduced false positives, as signatures provide definitive identification criteria.
Solution Approach 2:
The patent creates simplified copies of malware characteristics through hash signatures. Instead of analyzing complete malware behaviors, the system generates hash copies of malware signatures that can be rapidly compared and matched, maintaining classification reliability while eliminating false positives associated with behavior analysis.
2Reliability
If suffix tree or largest common substrings techniques are used for clustering, then object classification can be achieved, but performance degrades and memory consumption increases when dealing with very large numbers of objects
Solution Approach 1:
The patent replaces the mechanical suffix tree data structure with a hash table-based system. This substitution eliminates the O(n*m) time complexity and high memory consumption of suffix trees, achieving O(1) average time complexity for signature matching while maintaining clustering accuracy through hash-based signature comparison.
Solution Approach 2:
The patent changes the clustering parameter from comparing entire strings or substrings to comparing hash signatures. This parameter transformation reduces the computational burden from linear string comparison to constant-time hash lookup, dramatically improving processing speed for large-scale object clustering while preserving accuracy.
3Reliability
If suffix tree or largest common substrings techniques are used for clustering, then object classification can be achieved, but memory consumption becomes very large when dealing with very large numbers of objects
Solution Approach 1:
The patent creates compact hash signature copies of malware characteristics instead of storing complete malware samples or suffix trees. Each malware object is represented by a fixed-size hash signature, reducing memory consumption from storing entire malware binaries to storing only their hash representations, while maintaining sufficient information for accurate clustering.
Solution Approach 2:
The patent transforms the memory storage parameter from storing complete string data to storing hash signatures. This parameter change reduces memory requirements from O(n*m) where n is the number of objects and m is the average object size, to O(n*k) where k is the fixed hash signature size, enabling scalable clustering of very large object sets.
Data Source
AI summary
One embodiment relates to an apparatus configured to efficiently group a set of strings into clusters of related strings. Data storage is configured to store computer-readable code and data, and a processor is configured to access the data storage and to execute said computer-readable code. Computer-readable code is configured to receive the set of strings, determine an evaluation function between pairs of strings in said set, and group the strings into clusters, wherein determining the evaluation function between pairs of strings utilizes hash tables. Another embodiment relates to a computer-implemented method of efficiently grouping a set of strings into clusters of related strings based on rules of inference. Other embodiments and features are also disclosed.