Removable Storage Deduplication via Commonality Factoring Tags
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in optimizing storage capacity and data throughput performance while minimizing costs, particularly in distributed storage systems where redundant data storage occurs frequently.
Innovation Solution
A data storage system employing a commonality factoring (CF) module to assign CF tags to data patterns, and a deduplication module to determine if these patterns are already stored, thereby eliminating redundant data transfers by referencing existing data patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in distributed storage systems without deduplication, then storage capacity and data throughput are maximized, but redundant data storage occurs frequently leading to wasted storage space and increased costs
Solution Approach 1:
The system performs preliminary actions by calculating checksums of data blocks before storage and maintaining a checksum index in advance. When data is written, the checksum is computed and stored in the index before the actual data is written to the storage medium, enabling rapid identification of duplicates without reading existing data.
Solution Approach 2:
The patent introduces a checksum index as an intermediary structure between the data storage and the deduplication decision-making process. This index acts as a mediator that stores checksum values and their corresponding location information, allowing the system to efficiently determine whether data is already stored without direct comparison of all data blocks.
2Loss of energy
If data deduplication is implemented using checksum indexing, then redundant data storage is reduced, but additional processing steps and memory usage are required for checksum calculation and index maintenance
Solution Approach 1:
The patent extracts the deduplication decision-making process from the main data storage operation. By separating the checksum calculation and index lookup steps from the actual data write operation, the system can perform deduplication checks independently and efficiently, reducing the complexity impact on the main storage throughput.
Solution Approach 2:
Instead of copying entire data blocks for comparison, the system creates and stores only the checksum copies (fingerprints) of data blocks in the index. This copying approach reduces the amount of data that needs to be stored and processed, significantly reducing memory usage and processing complexity while maintaining deduplication effectiveness.
3Productivity
If traditional data storage methods are used, then data transfer speed is maintained, but storage expenses increase due to redundant data
Solution Approach 1:
The system performs partial action by only calculating and storing checksums for data blocks that are actually written to storage, rather than processing all possible data. This selective approach reduces the overhead of deduplication processing while maintaining effective redundancy elimination, balancing performance and cost benefits.
Data Source
AI summary
An apparatus and associated methodology contemplate a data storage system having a removable storage device operably transferring data between the data storage system and another device via execution of a plurality of input/output (I/O) commands. A commonality factoring (CF) module executing computer instructions stored in memory assigns a CF tag to a data pattern in the transferred data. A deduplication module executing computer instructions stored in memory determines if the data pattern corresponding to the CF tag is previously stored in the removable storage device.