Backup Indexing via Metadata Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing backup systems are inefficient in generating and searching backups due to the need for computationally expensive processes like crawling, which consume resources and hinder quick data retrieval and restoration.
Innovation Solution
A system that utilizes persistent storage and a backup orchestrator to generate backups, identify allocation schemes, extract system metadata, and create indexes for efficient storage and search functionality, allowing for the generation and storage of backups in a searchable format without crawling the contents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional backup systems crawl through backup contents to enable search functionality, then search capability is achieved, but computational resources and time are excessively consumed
Solution Approach 1:
The system performs preliminary indexing during the backup creation process by extracting file metadata (names, paths, sizes, timestamps) and building search indexes before the backup is completed. This preliminary action eliminates the need for computationally expensive crawling operations during search, as the index is already prepared and available for immediate querying.
Solution Approach 2:
Instead of analyzing the actual backup data during search operations, the system creates and uses a separate index structure that copies essential metadata information about the backup contents. This index serves as a lightweight representation that enables fast search operations without requiring access to or processing of the full backup data.
2Quantity of substance
If backups are stored without indexing, then storage space is optimized, but data retrieval and restoration become inefficient
Solution Approach 1:
The backup data is logically segmented into two components: the actual backup data and a separate index structure. The index contains segmented metadata information (file names, paths, sizes, timestamps) organized for efficient querying. This segmentation allows the system to maintain compact backup storage while providing fast retrieval through the indexed metadata without requiring duplication of the full backup data.
3Adaptability or versatility
If the system extracts and indexes all metadata from backups, then search functionality is enhanced, but the backup generation process becomes more complex
Solution Approach 1:
The backup generation process automatically extracts file metadata and builds indexes without requiring manual intervention or complex external processing. The system uses the existing file system metadata that is already available during the backup process, eliminating the need for additional complex analysis or processing steps. The indexing is performed as a natural byproduct of the backup creation process itself.
Data Source
AI summary
A backup manager for providing backup services includes persistent storage and a backup orchestrator. The persistent storage includes protection policies. The backup orchestrator generates a backup for a client based on the protection policies; identifies a portion of the backup that includes an allocation scheme; extracts system metadata from the backup using the allocation scheme; generates an index for the backup using the system metadata; and stores the backup and the index in backup storage.