File Clustering via Minhash and LSH for Forensic Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In high-dimensional data spaces, identifying similarities between files across multiple devices is challenging due to the curse of dimensionality, leading to inefficient data organization and protection, especially in enterprise environments with large volumes of files and diverse attributes.
Innovation Solution
A system and method for file clustering that extracts characteristics from files across multiple memory devices, uses algorithms like Minhash and Locality Sensitive Hashing to identify similarities, and generates visual representations for analysis, while also protecting sensitive data by tagging files based on reference sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pairwise comparison of files is performed to identify similarities, then complete set of similar matches can be obtained, but the time complexity increases to O(n^2) which is not practical for large numbers of files
Solution Approach 1:
The patent segments the file comparison process into multiple stages: first extracting key characteristics (hashes, keywords, metadata) from files, then performing clustering based on these segmented features rather than comparing entire files pairwise. This segmentation reduces the problem from O(n^2) full comparisons to O(n) characteristic extraction followed by efficient clustering operations.
Solution Approach 2:
The patent introduces an intermediary representation layer between files and comparison operations. Instead of directly comparing files, the system extracts file characteristics (hashes, keywords, metadata) as intermediary features, then performs clustering on these features. This intermediary layer enables efficient similarity detection without requiring exhaustive pairwise file comparisons.
2Ease of operation
If files are organized using common data organization strategies in high-dimensional spaces, then data can be structured, but the curse of dimensionality causes all objects to appear sparse and dissimilar, making these strategies inefficient
Solution Approach 1:
The patent extracts only the most relevant characteristics from files (hashes, keywords, metadata attributes) rather than using all possible features. This extraction process removes irrelevant dimensions that contribute to the curse of dimensionality, creating a reduced feature space where similarity detection remains reliable and organization strategies remain effective.
Solution Approach 2:
The patent transforms the high-dimensional file data into a different parameter space by extracting specific characteristics (hashes, keywords, metadata) and using these transformed parameters for clustering. This parameter transformation changes the problem from operating in high-dimensional sparse space to operating in a lower-dimensional dense space where common organization strategies remain effective.
3Loss of information
If the number of file characteristics (content hashes, keywords, regular expression attributes) increases to capture more file properties, then more comprehensive file analysis is achieved, but the dimensionality increases making similarity detection more challenging
Solution Approach 1:
The patent performs preliminary action by extracting and organizing file characteristics (hashes, keywords, metadata) before the clustering process. This preliminary extraction creates a structured representation of file properties that can be efficiently processed, avoiding the need to handle high-dimensional raw data during the similarity detection phase.
Solution Approach 2:
The patent addresses dimensionality by changing the dimensional space in which clustering occurs. Instead of clustering in the original high-dimensional space of all possible file attributes, the system projects files into a lower-dimensional space defined by selected characteristics (hashes, keywords, metadata), making the clustering problem tractable while preserving essential similarity information.
Data Source
AI summary
A system and method for file clustering, multi-drive forensic analysis and protection of sensitive data. Multiple memory devices can store files. A module can extract characteristics from the stored files, identify similarities between the files based on the extracted characteristics and generate file clusters based on the identified similarities. A visual representation of the file clusters, which can be generated to show the identified similarities among the files, can be displayed by a user interface module.