Backup Search via Distributed Catalog Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing devices face data loss issues due to persistent storage failures, and existing backup systems struggle with data retrieval and search efficiency, especially when backups are stored in a non-standard format without metadata, leading to increased computational costs for searching.
Innovation Solution
A remote agent provides data protection services by instantiating a backup agent in virtual machines, generating a backup data processing schema, and storing backups in backup storage, while maintaining a catalog for efficient search services, thus distributing the computational load and standardizing backup processing across a distributed system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If backups are stored in a nonstandard format without metadata, then storage flexibility is improved, but data retrieval efficiency deteriorates
Solution Approach 1:
The system performs preliminary actions by generating a backup data catalog during the backup process itself. The catalog is created in advance and stored alongside the backup data, containing metadata that describes the backup contents. This preliminary catalog generation enables efficient future retrieval operations without requiring time-consuming analysis of the backup data when retrieval is needed.
2Reliability
If a centralized backup system processes all metadata, then data protection consistency is improved, but computational cost increases
Solution Approach 1:
The system segments the metadata processing workload by distributing catalog generation to individual backup agents that operate locally at each backup source. Each backup agent independently generates its own catalog entry, eliminating the need for a centralized system to process all metadata. This segmentation maintains data protection consistency through standardized catalog schemas while significantly reducing the computational burden on any single system component.
3Reliability
If backup metadata is processed centrally, then data protection consistency is improved, but device complexity increases
Solution Approach 1:
The backup system implements self-service by enabling each backup agent to autonomously generate its own catalog data locally. The backup agents independently process their own metadata and create catalog entries without requiring centralized processing or coordination. This self-service approach simplifies the overall system architecture by eliminating complex centralized metadata processing components while maintaining data protection consistency through standardized catalog schemas.
Data Source
AI summary
A remote agent for providing data protection services to virtual machines includes persistent storage for storing a backup data catalog and a backup manager. The backup manager instantiates a backup agent in a new virtual machine of the virtual machines; obtains a backup data package associated with the new virtual machine using the backup agent, the backup data package specifies assets of the new virtual machine; generates a backup data processing schema for the new virtual machine based on the backup data package; initiates a backup generation for the new virtual machine to store a backup for the new virtual machine in backup storage; processes, using the backup agent, metadata associated with the backup using backup data processing schema to obtain backup data processing schema compliant catalog data; and provides search services for the backup using the backup data processing schema compliant catalog data and the backup data catalog.