Virtual Access Point in Time via Parallel MapReduce Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data protection systems face challenges in efficiently creating and managing virtual access points in time for data recovery, particularly in distributed storage systems, where continuous data protection and rollback capabilities are essential but often hindered by the complexity of journaling and data replication processes.
Innovation Solution
A storage system is implemented with a production site and a replication site, utilizing a highly parallel process and MapReduce operations to generate virtual access points in time (PIT) by creating metadata objects that describe the location and offset of data objects, allowing for concurrent processing and merging of disk and metadata objects to reconstruct data at any point in time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional journaling and data replication processes are used to create virtual access points in time, then data protection and rollback capabilities are provided, but the process becomes complex and inefficient
Solution Approach 1:
The patent segments the virtual access point creation process into multiple independent parallel operations using MapReduce framework. The Map phase processes different data objects concurrently across multiple nodes, creating metadata objects that describe locations and offsets. The Reduce phase merges these metadata objects to form the complete virtual access point. This segmentation eliminates the sequential complexity of conventional journaling while maintaining data protection reliability.
Solution Approach 2:
Instead of physically copying and moving large volumes of data to create access points in time, the patent creates lightweight metadata objects that reference the original data locations. These metadata objects serve as virtual copies that enable rapid access to historical data states without the overhead of actual data replication, significantly reducing system complexity.
2Adaptability or versatility
If traditional data replication methods are used to enable access to multiple points in time, then data recovery capability is provided, but extensive data movement and resource consumption are required
Solution Approach 1:
The patent introduces metadata objects as intermediary structures between the storage system and the virtual access points in time. These metadata objects contain location and offset information that enables direct access to historical data without requiring data movement. The metadata acts as a mediator that provides versatile point-in-time access while minimizing resource consumption by avoiding physical data replication.
Solution Approach 2:
The patent replaces the mechanical data movement operations inherent in traditional replication methods with computational metadata processing. Instead of physically moving data blocks to create access points, the system uses MapReduce operations to generate and merge metadata objects, substituting mechanical I/O operations with in-memory computational operations that consume significantly fewer resources.
3Manufacturing precision
If sequential processing is used to create virtual access points in time, then data accuracy is maintained, but processing speed is reduced
Solution Approach 1:
The patent divides the access point creation process into independent Map and Reduce stages that can execute in parallel. Multiple Map operations process different data objects simultaneously on different nodes, each maintaining data accuracy through proper metadata generation. The Reduce stage then merges these results to create the complete virtual access point. This segmentation enables high-speed parallel processing while preserving data object accuracy through structured metadata creation.
Data Source
AI summary
A storage system comprises a production site in operable communication with a replication site, and an object store, comprising a plurality of data objects associated with data stored in at least one data storage entity of the production site, disposed at the replication site. The replication site is configured to generate a virtual access point in time (PIT) based at least in part on the plurality of data objects, by providing a plurality of nodes in operable communication with the storage system, where each respective node is configured to, substantially concurrently, perform a MapReduce operation to generate a respective first set of disk objects based on the plurality of data objects, and to create a respective set of metadata objects. The first set of disk objects and the first set of metadata objects are merged together into a virtual access PIT from each of the plurality of parallel nodes.