Source-Side Metadata Indexing for Backup Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data backup storage systems face challenges in scalable metadata synchronization, particularly after data changes, leading to inefficiencies in real-time reflection of data changes and excessive workloads on backup appliances due to duplicate metadata storage and inefficient indexing processes.
Innovation Solution
Implementing a metadata bus that enables load balancing for CRUD operations and distributes extract and transform operations across clients, allowing real-time indexing and eliminating duplicate metadata storage by transferring transformed metadata directly to a search cluster, reducing the workload on backup appliances and enabling efficient file granularity indexing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata is stored and synchronized between backup software and search software in a conventional data backup storage system, then data protection and search capability are provided, but synchronization becomes challenging and inefficient as the amount of data grows
Solution Approach 1:
The patent segments the centralized metadata storage and processing into distributed metadata nodes that operate independently across the network. Each metadata node handles a portion of the metadata independently, eliminating the single-point bottleneck and enabling parallel processing of metadata operations across multiple nodes.
Solution Approach 2:
The patent introduces a metadata bus as an intermediary component that facilitates communication and coordination between metadata nodes and search nodes. The metadata bus acts as a mediator that receives metadata change notifications from backup nodes and distributes them to appropriate search nodes, enabling efficient synchronization without direct peer-to-peer communication overhead.
2Loss of information
If metadata synchronization is performed after data change to reflect real-time changes, then searchable metadata reflects current data state, but the metadata node becomes non-scalable and workload concentrates on backup appliance
Solution Approach 1:
The patent divides the metadata management function into multiple independent metadata nodes distributed across the network. Each metadata node independently tracks and synchronizes a portion of the metadata, enabling the system to scale horizontally by adding more nodes as data grows, rather than relying on a single non-scalable metadata node.
Solution Approach 2:
The patent implements self-service mechanisms where metadata nodes autonomously generate and publish their own metadata change notifications to the metadata bus without requiring centralized coordination. This self-service approach eliminates synchronization bottlenecks and enables each node to independently maintain real-time metadata accuracy for its managed data portion.
3Reliability
If duplicate metadata is stored and indexed in conventional systems, then data protection is ensured, but excessive workload is generated on backup appliances
Solution Approach 1:
The patent extracts the metadata storage and indexing function from the backup appliance and places it in separate dedicated metadata nodes and search nodes. This extraction eliminates the excessive workload on backup appliances by offloading metadata processing to specialized components that are optimized for their specific functions.
Solution Approach 2:
The patent creates multi-functional metadata nodes that can serve multiple purposes: storing metadata, indexing metadata for search, and publishing change notifications. This multi-functionality consolidates what would otherwise require separate components, reducing overall system complexity and workload distribution overhead.
Data Source
AI summary
A method, apparatus, and system for storing indexed metadata associated with backed up data is provided. The operations comprising: for each first data item stored at a first client device to be backed up, extracting, at the first client device, an associated first metadata item; transforming, at the first client device, the first metadata item based on a predefined metadata definition; associating, at the first client device, a first client identifier and a backup identifier with the transformed first metadata item; and transferring the transformed first metadata item from the first client device to a metadata bus of a search cluster.