Data Set Signatures Using DAFSA to Reduce Matching Collisions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data signature methods face challenges in identifying distinct signatures that minimize false positives and effectively match data sets, particularly in contexts like malicious file detection, where slight changes can lead to missed detections, and in scenarios with high collision rates among vast data subsets.
Innovation Solution
A computing system generates and selects distinct data signatures by iteratively compressing data portions to identify complex segments, using deterministic acyclic finite state automata (DAFSA) structures to store and compare candidate signatures, and removing non-unique signatures to ensure uniqueness, thereby reducing false positives and improving matching accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If hashing is used to generate data signatures, then the signature generation is simple and fast, but the likelihood of matching changes significantly with minor data changes leading to missed detections
Solution Approach 1:
The patent divides the data set into multiple portions and selects specific portions as signatures, rather than using the entire data set or a simple hash of it. This segmentation allows the signature to be more robust to minor changes while still being distinctive.
Solution Approach 2:
The patent changes the parameter of signature representation from a fixed hash value to a selected portion of the data set. This parameter change allows the signature to maintain reliability under transformation while preserving ease of comparison.
2Reliability
If complex portions of data sets are selected as signatures, then false positives are reduced, but the complexity of identifying and storing distinct signatures increases
Solution Approach 1:
The patent performs preliminary actions by pre-identifying and storing signatures in a data structure before actual matching occurs. This preliminary organization reduces the complexity during the matching phase while maintaining high reliability in false positive reduction.
Solution Approach 2:
The patent introduces a data structure as an intermediary between the raw data sets and the signature matching process. This intermediary structure organizes signatures in a way that reduces complexity while maintaining the ability to distinguish complex data portions effectively.
3Speed
If data signatures are used to match data sets quickly, then comparison speed is improved, but collision rates increase among vast data subsets
Solution Approach 1:
The patent applies local quality by selecting specific portions of data sets as signatures rather than using uniform hashing across all data. This localized approach to signature selection increases distinctness and reduces collisions while maintaining fast comparison speed.
Solution Approach 2:
The patent changes the parameter of signature representation from hash values to actual data portions, which inherently have higher entropy and distinctness. This parameter change reduces collision rates while preserving the speed advantage of signature-based matching over full data comparison.
4Measurement precision
If distinct signatures are identified to reduce false positives, then detection accuracy is improved, but the time and computational resources required increase
Solution Approach 1:
The patent performs the computationally intensive task of identifying distinct signatures as a preliminary action, storing the results in a data structure for future use. This preliminary computation, while time-consuming, is performed once and then reused, reducing the time loss in subsequent detection operations.
Solution Approach 2:
The patent creates copies of signature information in a structured data format that can be quickly accessed and compared. This copying approach allows the system to invest computational resources upfront in creating accurate signature representations that can then be efficiently reused for multiple comparisons.
Data Source
AI summary
Examples relate to identifying signatures for data sets. In one example, a computing device may: for each of a plurality of first data sets, obtain a data set signature; generate a first data structure for storing each data set signature that is distinct from each other data set signature; for each of a plurality of second data sets, obtain at least one data subset; generate a second data structure for storing each data subset; remove, from the first data structure, each data set signature that matches a data subset included in the second data structure; and for each data set signature removed from the first data structure, identify each first data set from which the data set signature was obtained; and for each identified first data set, obtain a new data set signature.