Bloom Filter Deduplication for Randomized File Paths
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data sharing methods are inefficient and costly, particularly for smaller entities, due to slow data transfer, lack of control over data usage, and ineffective deduplication of unstructured file paths, leading to storage waste and latency in accessing updated data.
Innovation Solution
Implementing a data exchange platform with cloud computing services that enables secure data sharing without copying, using bloom filters for efficient deduplication of unstructured file paths, and role-based access control to manage data access and deduplicate files based on metadata and bloom filters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional data sharing methods are used, then data can be transferred between parties, but the transfer is slow and costly particularly for smaller entities
Solution Approach 1:
The patent uses bloom filters as compact data structures that copy only essential deduplication information rather than transferring complete datasets. This allows rapid determination of data uniqueness without moving large amounts of data, significantly reducing transfer time and costs while maintaining accuracy in identifying duplicate files across distributed systems.
2Adaptability or versatility
If data is shared with multiple parties, then data accessibility is improved, but control over data usage is lost
Solution Approach 1:
The patent introduces bloom filters as an intermediary mechanism that enables multiple parties to access and query data without direct exposure to the underlying datasets. The bloom filter structure acts as a controlled interface that allows versatility in data querying while maintaining reliability through centralized management of the filter structure and its parameters, ensuring data usage control is preserved even as accessibility increases.
3Productivity
If files are ingested without deduplication, then all incoming files are processed, but storage space is wasted on duplicate files
Solution Approach 1:
The patent applies preliminary deduplication action by computing bloom filters for incoming files before actual file ingestion occurs. This preliminary step quickly identifies duplicate files using the bloom filter matching mechanism, allowing the system to reject duplicates early in the pipeline and proceed only with unique files, thereby maintaining high ingestion throughput for legitimate files while eliminating storage waste from duplicates.
4Measurement precision
If deduplication is performed on unstructured file paths, then duplicate detection is attempted, but the process is inefficient and causes latency
Solution Approach 1:
The patent transforms the deduplication problem by changing the parameter representation from complete unstructured file paths to compact bloom filter signatures. This parameter transformation allows the system to work with fixed-size data structures regardless of the original path complexity, enabling efficient bitwise operations for duplicate detection that maintain high accuracy while dramatically reducing processing time and computational overhead.
Data Source
AI summary
Techniques for deduplicating files to be ingested by a database are disclosed. A bloom filter may be built for each of a first set of files to be ingested into a data exchange to generate a set of bloom filters, wherein each of the set of bloom filters is built with a number of hash functions that is based on a desired false positive rate. The set of bloom filters may be stored in the metadata storage of the data exchange. In response to receiving a set of candidate files to be ingested, identifying using the set of bloom filters, candidate files from the set of candidate files that are duplicative of a file in the first set of files and pruning from the set of candidate files, each candidate file identified as being duplicative of a file in the first set of files using the set of bloom filters.