Data De-Duplication Using Linked Lists and Lightweight Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems face challenges in efficiently storing and retrieving large amounts of data due to high storage costs, mechanical limitations in storage devices, and the need for specialized hardware to manage data de-duplication using complex hash functions, which increases computational complexity and time for identifying duplicative data blocks.
Innovation Solution
The implementation of data processing methods that support data de-duplication and compression using lighter-weight hash functions, allowing for software-based hash processing without specialized hardware, and utilizing linked lists and tables to efficiently manage data storage and retrieval, reducing storage requirements and processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If complex hash functions (e.g., MD-5) with 128-bit hash values are used to minimize collisions and enable data de-duplication, then data de-duplication accuracy is improved, but specialized hardware is required which increases device complexity and cost
Solution Approach 1:
The patent uses simpler, computationally lighter hash functions that can be implemented in software without specialized hardware, sacrificing some collision resistance for ease of implementation and reduced hardware requirements
Solution Approach 2:
The patent introduces an intermediary verification step where potential duplicate blocks are first identified by hash value matching, then verified by comparing a portion of the actual data blocks themselves, avoiding the need for complex hardware-based hash functions
2Quantity of substance
If data de-duplication is implemented to reduce storage requirements, then storage efficiency is improved, but the process of retrieving and comparing each previously stored block with matching hash values increases time consumption
Solution Approach 1:
The patent extracts only a portion of the data block (e.g., first few bytes or a specific segment) for comparison purposes when verifying potential duplicates, rather than comparing entire blocks, thereby reducing comparison time while maintaining de-duplication effectiveness
Solution Approach 2:
The patent performs partial comparison of data blocks (comparing only a portion rather than the entire block) to verify duplicates, which is sufficient for most de-duplication purposes but significantly faster than full block comparison
3Adaptability or versatility
If data is broken into small blocks for storage and de-duplication, then storage flexibility and de-duplication granularity are improved, but the overhead for tracking and managing each block increases device complexity
Solution Approach 1:
The patent uses a universal hash value table structure that serves multiple functions: storing hash values, tracking block locations, and enabling quick lookup for de-duplication, reducing the need for separate management structures for each function
Data Source
AI summary
Data processing methods and apparatus for efficiently storing and retrieving data, e.g., blocks of data, to and from memory. The data processing includes, e.g., techniques such as using linked lists and/or tables for tracking duplicate data blocks received for storage, the use of lossless data compression, and de-duplication based on comparing hash values, compressed data block sizes, and/or bit by bit comparisons of the block of data to be stored and previously stored blocks of data.