Duplicate Data Estimation via Fingerprint Database
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage systems face inefficiencies due to duplicate data storage, which wastes space and requires significant time and money to implement data de-duplication technologies, with administrators needing tools to estimate potential savings before investment.
Innovation Solution
A duplicate estimation application computes fingerprints of data elements and logs them in a database, identifying unique entries to estimate storage savings from de-duplication, allowing administrators to assess the efficiency of data de-duplication techniques without upgrading the storage system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data de-duplication functionality is implemented, then storage space efficiency is improved, but system cost and complexity increase due to requiring upgrades or replacements
Solution Approach 1:
The patent creates a virtual copy of the storage system's data structure (the fingerprint database) that can be scanned without modifying the actual storage system. This allows estimation of de-duplication savings through fingerprint hashing and comparison, enabling administrators to assess potential benefits before implementing costly upgrades or replacements
Solution Approach 2:
The patent performs preliminary analysis by scanning the existing data structure and building a fingerprint database before any de-duplication implementation. This preliminary action estimates the potential space savings and provides administrators with information to make informed decisions about whether to proceed with upgrades or replacements
2Quantity of substance
If data de-duplication functionality is implemented, then storage space efficiency is improved, but implementation time and cost increase
Solution Approach 1:
The patent works with a virtual representation of the data structure rather than implementing de-duplication directly in the storage system. By scanning existing data and creating fingerprints in a database, the system can estimate savings quickly without the time-consuming process of actual de-duplication implementation
Solution Approach 2:
The patent performs a preliminary scan and fingerprinting operation that can be completed relatively quickly compared to full de-duplication implementation. This preliminary action provides administrators with estimation data to make informed decisions about whether the time and cost investment in de-duplication functionality is warranted
3Quantity of substance
If administrators upgrade storage systems to enable data de-duplication, then storage space efficiency is improved, but investment return becomes uncertain without estimation tools
Solution Approach 1:
The patent creates a virtual copy of the storage data structure that can be analyzed to extract information about duplicate data patterns. By building a fingerprint database from this virtual copy, the system can provide administrators with concrete information about potential space savings, making investment decisions more informed and reducing uncertainty
Solution Approach 2:
The patent provides feedback to administrators in the form of estimation results that show the potential space savings from de-duplication. This feedback loop allows administrators to see the expected return on investment before committing to upgrades or replacements, reducing uncertainty in decision-making
Data Source
AI summary
The present invention provides a system and method for estimating duplicate data in a storage system. A duplicate estimation application executes on a client of a storage system selects an element from an intended destination such as, e.g., a data store of the storage system. If the element is a file (or other data container), the application reads data from the file and computes a fingerprint of the read data. The computed fingerprint is then logged in a fingerprint database, which is illustratively stored on a storage device connected to the client executing the application. This process repeats until the entire file (or other data container) has been read and fingerprinted. Once all elements have been scanned, fingerprinted and recorded, the application identifies any unique entries within the fingerprint database. Utilizing this information, the application computes an estimated space savings that may be realized by employing a data de-duplication technique.