Hierarchical Index Trees for Distributed Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data protection and recovery methods are inadequate for real-time data management in enterprise environments, as they fail to keep pace with rapid data growth, geographically distributed operations, and 24/7 data center requirements, leading to significant costs and downtime due to incomplete data capture, manual processes, and inability to recover fine-grained data objects efficiently.
Innovation Solution
A data management system (DMS) that provides continuous data protection and real-time data management services by creating distributed object storage, indexing active objects with hierarchical indexing trees, allowing for efficient search and recovery of objects across a distributed network, and enabling real-time data replication and distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional tape backup devices are used for periodic data storage, then data protection is provided, but data recovery time and application downtime increase significantly
Solution Approach 1:
The patent implements continuous indexing of data objects with their metadata (including timestamps, locations, and hierarchical relationships) as data changes occur in real-time. This preliminary indexing action eliminates the need for time-consuming manual search and recovery operations later, as the indexed structure enables rapid location and restoration of specific objects without requiring full backup restoration or manual tape searching.
2Reliability
If volume snapshot tools are used to capture data at a particular time, then data can be reintroduced after failure, but fine-grained application data cannot be recovered without reintroducing the entire backup volume
Solution Approach 1:
The patent segments data into individual objects with unique identifiers and indexes them independently, rather than treating data as monolithic volumes. Each object can be individually located, accessed, and recovered through the hierarchical index structure. This segmentation enables fine-grained recovery of specific application objects without requiring restoration of entire backup volumes, significantly improving recovery efficiency and reducing downtime.
3Reliability
If manual data recovery processes are used, then data can be recovered after failure, but the process is tedious and time-consuming involving multiple manual steps
Solution Approach 1:
The patent implements an automated feedback mechanism where the indexing system continuously monitors data changes, updates metadata, and maintains hierarchical indices. When data recovery is needed, the system automatically queries the index structure, locates the required objects based on search criteria, and facilitates rapid recovery without manual intervention. This feedback loop eliminates tedious manual steps and transforms recovery into an automated, efficient process.
4Reliability
If storage based snapshot is used to capture data, then data can be stored for later use, but critical data changes after backup may result in incomplete updates being captured
Solution Approach 1:
The patent implements continuous indexing and monitoring of data objects as they change, rather than capturing data at discrete snapshot points. The system continuously tracks metadata, timestamps, and hierarchical relationships in real-time, ensuring that even critical data changes are captured and indexed immediately. This continuous action eliminates gaps in data capture and ensures complete, consistent data records are maintained for recovery operations.
Data Source
AI summary
A data management system or “DMS” provides data services to data sources associated with a set of application host servers. The data management system typically comprises one or more regions, with each region having one or more clusters. A given cluster has one or more nodes that share storage. When providing continuous data protection and data distribution, the DMS nodes create distributed object storage to provide the necessary real-time data management services. The objects created by the DMS nodes are so-called active objects. The distributed object store can be built above raw storage devices, a traditional file system, a special purpose file system, a clustered file system, a database, and so on. According to the present invention, the DMS active object store provides an indexing service to the active objects. In an illustrative embodiment, any object property that has a given attribute is indexed and, as a result, the attribute becomes searchable. The DMS provides hierarchical distributed indexing using index trees to facilitate searching in a highly efficient manner.