Deduplication Cache Using Bitmap Keys for Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently caching and deduplicating large volumes of files, requiring significant storage space and being computationally expensive, especially when dealing with billions of files where structural similarities and limited domains are not optimally leveraged for compression.
Innovation Solution
A system utilizing a server with processors and a database that identifies unique source identifiers and densely populated fields, partitions data into significant and less significant bits, generates keys, and uses bitmaps to determine duplicate files, optimizing storage and query efficiency by minimizing memory usage and enabling high-volume insertion and lookup tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional hash tables are used to store and deduplicate files, then file deduplication can be achieved, but memory usage becomes excessively high and storage space requirements become astronomical
Solution Approach 1:
The patent segments files into fixed-size chunks and further divides each chunk into variable-length tokens based on content boundaries. This segmentation allows the system to process and store only unique tokens rather than entire files, dramatically reducing memory usage while maintaining deduplication accuracy through hierarchical indexing structures.
Solution Approach 2:
The system changes the representation parameters of file data by converting binary file content into standardized token sequences with metadata attributes. By transforming files into structured token representations with configurable parameters (token size, chunk size, encoding format), the system optimizes storage efficiency while preserving the ability to accurately identify duplicates.
2Quantity of substance
If generalized compression schemes are applied to reduce storage space, then memory usage decreases, but the schemes are not clever enough to determine optimal conversions automatically and make cached files more difficult to index
Solution Approach 1:
The system performs preliminary actions by pre-processing files into standardized token formats during the caching phase, organizing data into indexed structures with metadata before queries are executed. This preliminary organization enables efficient retrieval and indexing operations while maintaining compressed storage, as the data is already structured for optimal query performance.
Solution Approach 2:
The patent introduces token sequences as an intermediary representation between raw file content and the indexing system. These tokens serve as mediators that preserve essential file characteristics for indexing while reducing data redundancy, allowing the system to maintain both compression efficiency and indexing capability without direct manipulation of original file formats.
3Reliability
If every file is independently stored until processing is complete, then data integrity is maintained, but astronomical quantities of storage are required
Solution Approach 1:
The system merges identical file chunks and tokens across multiple files by storing only unique segments with references to their sources. By combining common data elements into shared storage locations and using pointer-based references, the system maintains data integrity for all files while eliminating redundant storage, reducing total storage volume from astronomical to manageable levels.
Data Source
AI summary
A method for caching and deduplicating a plurality of received segments of data is disclosed. The method comprises identifying a value of a first data field in each segment acting as a unique source identifier; and identifying a value of a second data field in each segment, the second data field being densely populated by values in the plurality of segments. The value of the second data field is partitioned into a first partition comprising more significant bits and a second partition comprising less significant bits. A key is generated based on values of the first data field and the first partition. A database entry associates the first key with a bitmap, the bitmap having a length based on the number of possible values a bitmap of equal length to the second partition could validly take. Single bits of the bitmap are set corresponding to received segments, to enable deduplication.