Storage Deduplication Using XOR Difference and Run-Length Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication techniques face challenges in efficiently identifying and removing similar data while maintaining accuracy, particularly in storage applications where the difference between data sets is localized, leading to increased processing time and potential biases in bit significance.
Innovation Solution
The proposed solution involves performing an exclusive OR operation on data sets, followed by run-length encoding, and storing the encoded data with position information when the encoded size is below a predetermined threshold, allowing for high-speed deduplication by managing similarity and difference effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional deduplication techniques are used to identify and remove similar data, then data duplication is reduced, but processing time increases and accuracy decreases when dealing with localized differences between data sets
Solution Approach 1:
The patent segments data comparison into fixed-size blocks and uses XOR operations to identify differences only in specific blocks rather than comparing entire data sets. This segmentation allows rapid identification of similar data by focusing only on differing portions, resolving the contradiction between accuracy and processing time.
Solution Approach 2:
The patent changes the parameter of data representation by using XOR results and run-length encoding to transform data into a compact difference representation. This parameter transformation enables efficient comparison and storage of data differences, improving both accuracy in identifying similar data and reducing processing time.
2Quantity of substance
If XOR operation and run-length encoding are used to compress difference data, then storage efficiency improves, but device complexity increases
Solution Approach 1:
The patent uses copying by storing difference data in a simplified encoded format rather than storing complete data sets. The XOR-based difference representation and run-length encoding create a compact copy of the essential information, reducing storage requirements while maintaining the ability to reconstruct original data through simple XOR operations.
3Measurement precision
If complete data comparison is performed to ensure accurate deduplication, then deduplication accuracy is maintained, but processing speed decreases
Solution Approach 1:
The patent applies preliminary action by performing XOR operations on data blocks before full comparison, and using run-length encoding to pre-process difference data. This preliminary processing identifies obvious similarities and differences early, allowing the system to quickly determine whether complete comparison is necessary, thus maintaining precision while improving speed.
Data Source
AI summary
An information storage device includes a memory and a processor coupled to the memory. The memory is configured to store therein a plurality of data. The processor is configured to receive a write request of writing first data to the memory. The processor is configured to perform an exclusive OR operation on the first data and second data of the plurality of data to obtain first difference data. The processor is configured to encode the first difference data by a run-length encoding to obtain encoded data. The processor is configured to determine whether a first size of the encoded data is smaller than a predetermined threshold value. The processor is configured to store, upon determining that the first size is smaller than the predetermined threshold value, the encoded data in the memory in association with first position information indicating a position of the second data.