Encoding-Based Deduplication for Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage systems face inefficiencies due to the high cost and increasing speed of memory technology, necessitating complex methodologies like deduplication and compression to maximize storage efficiency.
Innovation Solution
A computer-implemented method that encodes candidate data portions, identifies similarities and differences with target data portions, and combines measurements to determine if symbol-based deduplication or compression is necessary, using techniques like Lempel-Ziv-Welch encoding and XOR operations to optimize storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If memory technology speed increases, then data processing capability improves, but memory cost increases
Solution Approach 1:
The patent segments data into fixed-size chunks and further divides them into smaller sub-chunks for processing. This segmentation enables efficient comparison and deduplication operations, allowing the system to maximize the utilization of high-speed memory while reducing the amount of data that needs to be stored, thereby mitigating the cost increase associated with faster memory technology.
Solution Approach 2:
The patent changes the parameter of data representation by encoding data chunks into fixed-length binary representations and using hash functions to generate unique identifiers. This parameter transformation enables efficient storage and retrieval operations in high-speed memory, optimizing the balance between memory speed and cost.
2Quantity of substance
If deduplication and compression methodologies are employed, then storage efficiency improves, but system complexity increases
Solution Approach 1:
The patent performs preliminary encoding of data into fixed-size chunks and generates hash values before actual storage operations. This preliminary processing organizes data in a standardized format that simplifies subsequent deduplication and compression operations, reducing the complexity of the overall storage system while improving storage efficiency.
Solution Approach 2:
The patent creates simplified representations of data through hashing and encoding, where complex data structures are transformed into compact binary forms. These copied representations enable efficient comparison and deduplication without requiring the system to manipulate the original complex data structures, thereby reducing system complexity.
3Quantity of substance
If data is encoded and compared to identify redundancies, then storage space is reduced, but processing time increases
Solution Approach 1:
The patent performs partial processing by comparing only fixed-size chunks of data rather than entire data sets. This approach identifies redundancies efficiently by focusing on manageable portions of data, reducing the overall processing time while still achieving significant storage space reduction through deduplication.
Solution Approach 2:
The patent replaces complex mechanical comparison operations with mathematical hash functions and binary encoding. This substitution enables rapid identification of data redundancies through computational operations that are much faster than traditional byte-by-byte comparison methods, reducing processing time while maintaining storage efficiency.
Data Source
AI summary
A method, computer program product, and computing system for encoding a candidate data portion to generate an encoded candidate data portion; identifying one or more portion similarities between the encoded candidate data portion and an encoded target data portion to position the one or more portion similarities with respect to the encoded target data portion, thus generating one or more portion similarity measurements; identifying one or more portion differences between the encoded candidate data portion and the encoded target data portion to generate one or more portion difference measurements; and combining the one or more portion similarity measurements and the one or more portion difference measurements to generate a candidate similarity measurement for the candidate data portion.