Backup Indexing via External Hash Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup systems face inefficiencies in indexing large numbers of files due to resource-intensive processes and significant performance degradation when using hash-based identifiers in search engines like Elasticsearch, particularly as the number of documents increases.
Innovation Solution
The method involves generating external files containing file hashes for already-indexed and new backup files, comparing these files to determine changes, and updating the search engine index by adding or deleting entries for newly added or deleted files, without relying on hash-based identifiers, thus improving performance by only ingesting changed data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If hash-based identifiers are used in Elasticsearch to avoid indexing duplicate files, then duplicate detection capability is improved, but indexing performance degrades significantly as the number of documents increases
Solution Approach 1:
The patent segments the duplicate detection process into two independent parts: (1) generating and storing hash identifiers outside Elasticsearch during data ingestion, and (2) performing duplicate detection by comparing these external hash values before ingestion. This separation removes the performance burden from Elasticsearch while maintaining duplicate detection capability.
Solution Approach 2:
The patent introduces an intermediary mechanism (external hash storage and comparison system) that mediates between the incoming data and Elasticsearch. This intermediary handles the computationally intensive hash-based duplicate detection outside the search engine, allowing Elasticsearch to focus solely on indexing operations.
2Reliability
If all files in large backups are indexed to ensure completeness, then data coverage is improved, but system resource consumption increases excessively
Solution Approach 1:
The patent performs preliminary duplicate detection by comparing file hashes against previously indexed files before actually ingesting data into Elasticsearch. This preliminary action identifies and filters out duplicates in advance, ensuring data coverage is maintained while avoiding the waste of resources on redundant indexing operations.
Solution Approach 2:
Instead of indexing all files without discrimination, the patent applies partial action by selectively indexing only those files that pass the duplicate detection filter. This approach maintains sufficient data coverage for search operations while significantly reducing the excessive resource consumption associated with indexing every single file.
3Measurement precision
If unique hash identifiers are generated for each file to enable duplicate detection, then duplicate file identification is improved, but indexing speed decreases as the index grows larger
Solution Approach 1:
The patent extracts the hash identifier generation and duplicate detection functionality from the Elasticsearch indexing pipeline. By taking out this computationally intensive operation and performing it externally before ingestion, the system maintains precise duplicate file identification while preserving indexing speed, as Elasticsearch no longer bears the burden of hash-based duplicate detection during the indexing process.
Data Source
AI summary
An apparatus, method, and system for updating a file index in a search engine in a data backup system to reflect file changes introduced in a new backup is disclosed. The operations comprise: generating a first external file, the first external file comprising file hashes for files already indexed in a file index in a search engine of a data backup storage system that are not associated with a deleted status; generating a second external file, the second external file comprising file hashes for files in a new backup; determining one or more file changes introduced in the new backup based on a comparison between the first external file and the second external file; and updating the file index in the search engine to reflect the one or more file changes introduced in the new backup.