Multi-tiered Backup Indexing for Scalable Storage Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Information management systems face bottlenecks in data processing due to communication overhead, particularly during backup and restore operations, especially when dealing with a large number of client devices, leading to reduced resource availability and scalability issues.
Innovation Solution
Implementing a distributed backup management system where clients store job metadata locally and communicate directly with cloud or network storage, reducing the burden on computing resources and enabling scalable operations by minimizing communication with the storage manager.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If clients communicate with the storage manager during backup operations, then backup management and control are maintained, but communication bottlenecks occur when large numbers of clients operate simultaneously
Solution Approach 1:
The patent segments the backup management function by introducing a tiered indexing structure where index data is distributed across multiple storage locations (first storage location for primary indexes, second storage location for secondary indexes). This segmentation allows clients to access index data in parallel without all communication funnelling through the storage manager, thereby maintaining control while increasing throughput.
Solution Approach 2:
The patent introduces an intermediary indexing layer that mediates between clients and the storage manager. The index data stored in the tiered structure acts as an intermediary, allowing clients to perform backup operations by accessing pre-computed index information without requiring continuous communication with the storage manager, thus reducing communication bottlenecks while maintaining management control.
2Adaptability or versatility
If the information management system supports more client devices, then system coverage and utility increase, but resource availability (processor and storage) decreases due to communication overhead
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing index data in a tiered structure before backup operations commence. This pre-computed index information is cached in accessible storage locations, allowing clients to perform backup operations efficiently without requiring real-time resource-intensive processing during the actual backup, thus enabling more clients to be supported with reduced per-client resource consumption.
3Ease of operation
If centralized backup management is maintained through storage manager communication, then system control and coordination are ensured, but scalability to large numbers of clients is limited
Solution Approach 1:
The patent introduces another dimension to the storage architecture by creating a multi-layered indexing system with first and second storage locations. This dimensional expansion of the storage hierarchy allows the system to maintain centralized control through the storage manager while enabling scalable client access through the distributed index structure, effectively adding a spatial dimension to resource organization that supports both control and scalability.
Data Source
AI summary
Certain embodiments disclosed herein reduce or eliminate a communication bottleneck at the storage manager by reducing communication with the storage manager while maintaining functionality of an information management system. In some implementations, a client obtains information for enabling a secondary storage job (e.g., a backup or restore) from a storage manager and stores the information (which may be referred to as job metadata) in a local cache. The client may then reuse the job metadata for multiple storage jobs reducing the frequency of communication with the storage manager. When a configuration of the information management system changes, or the availability of resources changes, the storage manager can push updates to the job metadata to the clients. Further, a client can periodically request updated job metadata from the storage manager ensuring that the client does not rely on out-of-date job metadata.