File System Data Deduplication via Hash-Based Instance Repository
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication techniques rely on databases or in-memory indices, requiring a two-step process and being limited in size and speed, which increases storage costs and management complexity for enterprises with exponentially growing data.
Innovation Solution
A method using fingerprint generation to determine duplicate data objects by creating a location and name portion from hash values, storing instances in a conventional file system directory structure, eliminating the need for separate databases or indices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If databases or in-memory indices are used to store indexed data for deduplication, then data can be retrieved with unique names and locations, but the system requires a two-step process (instance lookup followed by retrieval based on location/pointer information) which increases complexity and reduces speed
Solution Approach 1:
The patent extracts the indexing function directly into the file system structure itself. Instead of using a separate database or in-memory index to store location information, the file system directory structure and file names encode the indexing information directly. This eliminates the need for a separate lookup step and reduces system complexity by integrating the index into the storage structure.
Solution Approach 2:
The file system structure serves multiple functions simultaneously: it stores data files, provides indexing through directory names and file names, and enables deduplication through hash-based naming conventions. This multi-functionality eliminates the need for separate indexing mechanisms and reduces the two-step process to a single direct access operation.
2Device complexity
If conventional file system arrangements are used to store instances, then storage costs are reduced and management is simplified, but the ability to efficiently determine duplicates and retrieve data may be compromised
Solution Approach 1:
The patent segments the data storage into a standard file system structure with directories and files, while embedding deduplication information in the file names and directory structure. This segmentation allows the use of simple, well-understood file system constructs while maintaining efficient duplicate detection through hash-based naming conventions that enable direct lookup without complex indexing.
3Quantity of substance
If data is stored using traditional deduplication methods with databases or indices, then data can be deduplicated, but storage costs increase and management complexity increases for enterprises with exponentially growing data
Solution Approach 1:
The file system structure performs deduplication services autonomously through its own mechanisms. By using hash-based naming conventions and storing indexing information directly in the file system structure, the system self-manages duplicate detection and data retrieval without requiring external database management systems or complex indexing mechanisms. This self-service approach reduces management complexity while maintaining effective deduplication.
Data Source
AI summary
A data deduplication method providing direct look up and storage in an instance repository (IR). The method includes receiving a data object and processing the data object to generate a fingerprint that includes a location component, which defines a file location within the IR such as by first using a hash function to create a hash for the data object and parsing the hash value into sub-strings defining sub-directories of the IR. The method includes determining whether the data object is a duplicate by verifying the presence of a file in the IR at the file location. Determining if the data is unique involves performing a system call on the IR providing the location component as the file path. The method includes, when a file is not in the IR, updating the IR to store the data object as a file at the file location defined by the location component.