Global Single Instance Indexing for Backup Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges in performing fast searches across large volumes of backup data due to limitations in indexing technology, such as the number of entries that index engines can handle, leading to inefficiencies in disaster recovery operations.
Innovation Solution
The implementation of global single instance indexing techniques, which generate a content hash for each data item, use it as an indexer ID, and maintain a reference count in an index database, allowing for efficient storage and retrieval of data items across multiple backup images while avoiding duplicate entries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional indexing technology is used to index backup data items, then fast search capability is provided, but the number of entries in the index database is limited (e.g., 6 million or 30 million entries)
Solution Approach 1:
The patent merges multiple index entries that reference the same data item into a single index entry by using content hash as the indexer ID. Instead of creating separate entries for each backup image containing the same file, the system consolidates them into one entry with a resource list that tracks all backup images, thereby eliminating the entry limit constraint while maintaining fast search capability.
Solution Approach 2:
The single index entry structure serves multiple functions: it stores the data item identifier (content hash), maintains a resource list of all backup images containing the item, tracks reference counts for each backup image, and enables both fast searching and efficient retrieval. This multi-functional design replaces the need for multiple separate index entries.
2Quantity of substance
If multiple inter-connected machines (federated indexers) are dedicated for indexing to overcome the entry limit, then the number of indexable entries increases, but system complexity and cost increase
Solution Approach 1:
The patent consolidates the indexing function into a single index database structure rather than distributing it across multiple federated indexers. By using content hash-based single instance indexing, the system achieves scalable entry capacity without requiring multiple interconnected indexing machines, thereby reducing system complexity while maintaining the ability to handle large numbers of backup data items.
3Reliability
If periodic snapshots are taken at high frequency for backup, then operational recoverability is improved, but the volume of backup data generated increases
Solution Approach 1:
The patent eliminates duplicate index entries for identical data items across multiple backup snapshots by using content hash as the indexer ID. When the same file is backed up in multiple periodic snapshots, only one index entry is created, and the resource list tracks all snapshot references. This merging approach maintains full operational recoverability while significantly reducing the index database size proportional to the number of duplicate items.
Data Source
AI summary
Techniques for global single instance indexing for backup data are disclosed. In one particular exemplary embodiment, the techniques may be realized as a method for indexing backup data. The method may comprise generating a content hash for an item being backed up. The item may be associated with a resource name and the content hash may be used as an identifier in backup and indexing. The method may further comprise saving an entry for the item in an index database. The entry may comprise a resource list which may comprise the resource name and a reference count associated with the resource name. Moreover, the method may comprise setting a value for the reference count.