Assisted Mainframe Data De-duplication for Tape Backup Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data de-duplication techniques in mainframe environments are ineffective due to the frequent changes in data, which disrupt the ability to identify duplicate blocks, resulting in suboptimal storage reduction and high storage requirements in mainframe tape libraries.
Innovation Solution
The Assisted Mainframe Data De-duplication (AMDD) technology preprocesses data by rearranging and padding it to minimize the impact of changes, allowing for better identification of duplicate blocks and improving data de-duplication efficiency by integrating with existing tape-on-disk solutions and de-duplication engines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional data de-duplication is used in mainframe environments, then storage space is reduced for static data, but effectiveness drops significantly when data changes occur
Solution Approach 1:
The patent segments data into fixed-size blocks and processes each block independently through hash functions. This segmentation allows the system to identify duplicate blocks even when other parts of the data change, maintaining de-duplication effectiveness for mainframe environments where data changes occur frequently.
Solution Approach 2:
The patent changes the parameter of data representation by applying hash functions to transform original data blocks into fixed-size hash values. This parameter transformation enables efficient duplicate detection regardless of data changes, as identical original blocks will always produce the same hash value even when other blocks in the dataset change.
2Reliability
If data is frequently updated in mainframe tape libraries, then data integrity is maintained, but duplicate block identification becomes ineffective
Solution Approach 1:
The patent implements continuous de-duplication processing by maintaining a rolling hash window that continuously processes data blocks as they are read from the data stream. This continuous processing ensures that duplicate blocks are identified in real-time during backup operations, maintaining high de-duplication efficiency even when data is frequently updated.
Solution Approach 2:
The patent performs preliminary hashing of data blocks during the backup process itself, before data is written to storage. By calculating hash values in advance during the data streaming phase, the system can identify and eliminate duplicates before they consume storage space, maintaining both data integrity and processing efficiency.
3Quantity of substance
If block-level de-duplication is implemented, then storage compression is improved across multiple files, but processing complexity increases
Solution Approach 1:
The patent extracts only the essential identifying feature of each data block by computing its hash value, rather than processing or comparing the entire block content. This extraction approach dramatically reduces processing complexity while maintaining effective duplicate detection across multiple files and data streams.
Solution Approach 2:
The patent introduces hash values as an intermediary representation between the original data blocks and the de-duplication logic. Instead of directly comparing large data blocks which would be computationally expensive, the system uses these intermediary hash values for efficient comparison and duplicate identification across multiple files.
Data Source
AI summary
Data de-duplication (“de-dupe”) allows IT organizations to replace aging tape systems with disk-based backup solutions and minimize the storage allocated to backup and data protection. The effectiveness of de-dupe technology is dependent on the data being protected. Data streams with little data repetitiveness provide disappointing results when processed through a block-level de-dupe engine. To avoid this problem, Assisted Mainframe De-Dupe (AMDD) technology can insure that filesystem block-level de-dupe products efficiently de-dupe tape backup streams received from IBM™ and/or compatible mainframes. By pre-processing backup tape volumes before sending the data to storage, AMDD insures that large amounts of unchanged data lines up on de-dupe block boundaries each time the data is sent to the de-dupe process engine. By providing “well-behaved” data to the de-dupe process, AMDD can improve the effectiveness of de-dupe processing and substantially reduce the storage used to backup critical mainframe DASD resources.