Storage System File Similarity Detection Using LSH Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in identifying and managing similarities among large, frequently updated, and varied files, leading to complex and time-consuming processes, especially when dealing with vast numbers of files.
Innovation Solution
The implementation of a storage system that employs hash functions, such as Locality Sensitive Hashing (LSH), to detect duplicate and similar files by applying hash formulas on data blocks, using a similarity database based on graph structures like Neo4J to calculate similarity scores between files, and employing mechanisms like Jaccard similarity coefficient and Levenshtein distance to determine file similarity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional file comparison methods are used to identify similar files, then accuracy in detecting duplicate blocks is maintained, but the process becomes extremely time-consuming and complex when dealing with vast numbers of large files
Solution Approach 1:
The patent divides large files into smaller blocks and applies hashing to these blocks individually. This segmentation allows the system to process files in manageable chunks rather than comparing entire files at once, dramatically reducing the time and computational resources required while maintaining detection accuracy
Solution Approach 2:
The patent replaces traditional mechanical file comparison methods with hash function-based detection. Instead of directly comparing file contents block-by-block, the system uses cryptographic hash functions to generate fingerprints of blocks, enabling rapid identification of similar or duplicate blocks without exhaustive comparison
2Productivity
If large chunks are used for comparison to reduce processing overhead, then processing speed improves, but smaller identical contents may be missed
Solution Approach 1:
The system segments files into blocks of optimized size that balance processing efficiency with detection sensitivity. This segmentation strategy ensures that even small identical contents are captured as separate blocks while maintaining manageable processing units
Solution Approach 2:
The patent applies hashing to more blocks than strictly necessary by using a carefully chosen block size that ensures coverage of all potential duplicate regions. This partial redundancy in block processing guarantees that small identical contents are not missed while keeping the overall processing load acceptable
3Reliability
If comprehensive file similarity analysis is performed on all files in the storage system, then security threats are thoroughly identified, but the complexity and resource requirements increase significantly
Solution Approach 1:
The patent replaces complex comprehensive file analysis with efficient hash-based similarity detection. The hash functions provide a mathematical foundation for identifying similar files without requiring intricate comparison algorithms, significantly reducing system complexity while maintaining detection completeness
Solution Approach 2:
The system automatically builds and maintains a database of hash values for all files in the storage system. This self-service mechanism enables continuous security monitoring without manual intervention, allowing comprehensive threat identification to scale automatically with the storage system size
Data Source
AI summary
A method for securing a storage system, the method may include detecting, by cybersecurity engine, a suspected file that is stored in the storage system; autonomously searching for one of more files that fulfill one or more similarity criteria in relation to the suspected file; wherein the one or more similar files are stored in a storage system; finding, by the similarity engine, the one or more similar files using a similarity data structure that associates a group of files stored in the storage system with (a) inter-file similarity scores and (b) block hash values shared between files; and responding to the finding, wherein the responding comprises sending the one of more similar files to the cybersecurity engine.