Delta Map Data Recovery for Near-Constant Time Versioning

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current systems for processing and recovering synthesized large data sets (SLDS) face challenges due to their dynamic nature, rapid changes, and the need for efficient version management, leading to increased recovery time and storage requirements, with existing methods like reverse incrementals and RMAN being inefficient and requiring significant storage space.

Innovation Solution

A system and method that generate delta maps for changes in dataset segments across multiple versions, allowing for the recovery of any version of the SLDS in near-constant time by summing the initial dataset version, first delta map, and second delta map, while maintaining multiple incremental backups and avoiding costly hardware requirements.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If conventional snapshot techniques are used to capture and restore versions of SLDS, then version capture capability is provided, but recovery time increases as a function of the number of versions maintained

Engineering Contradiction:

Improveversion capture capabilityVSAvoidrecovery time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The patent segments the dataset into fixed-size blocks and creates separate bloom filters for each block. This segmentation allows the system to efficiently track changes at the block level rather than managing entire dataset versions, enabling constant-time recovery regardless of the number of versions maintained.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces bloom filters as an intermediary data structure between the dataset blocks and version information. These bloom filters efficiently encode change information across multiple versions without requiring storage of complete version snapshots, enabling fast recovery by combining selective blocks from different versions.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If multiple versions of SLDS are maintained for rapid changing datasets, then version flexibility is improved, but storage space consumption increases unreasonably

Engineering Contradiction:

Improveversion flexibilityVSAvoidstorage space

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The patent extracts only the essential change information from complete dataset versions by using bloom filters to track which blocks have changed. This extraction approach maintains version flexibility while storing only the minimal necessary data to reconstruct any version, dramatically reducing storage requirements.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent discards complete version snapshots and recovers versions on-demand by selectively combining blocks from the base version with incremental changes from subsequent versions. This allows the system to maintain unlimited version flexibility while using constant storage space.

Inventive Principle:

Principle #34Discarding and recovering

3Productivity

If reverse incrementals are used where the last backup is a full backup and previous incrementals are converted, then retention window is moved forward, but a full backup must always be maintained for recovery

Engineering Contradiction:

Improveretention window managementVSAvoidfull backup storage requirement

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The patent implements a dynamic version management system where the base version can be freely changed without requiring conversion of incremental backups. The system dynamically adapts to any base version selection by using bloom filters to track changes relative to the chosen base, eliminating the constraint of always needing the most recent full backup.

Inventive Principle:

Principle #15Dynamics

4Productivity

If Oracle's RMAN merges the last incremental backup with the full backup to create a new full backup, then incremental merging is achieved, but only the last full backup and one last incremental backup are available for recovery

Engineering Contradiction:

Improveincremental merging efficiencyVSAvoidrecovery version options

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The patent performs preliminary action by maintaining bloom filters that track changes from the base version through all incremental backups. This preliminary tracking structure enables the system to recover any version by selectively combining blocks, providing full version flexibility without the constraint of only supporting the latest full backup plus one incremental.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10877943B1System and method of near-constant time recovery of continuously changing synthesized large datasets

Publication Date: 2020.12.29 COHESITY INC

AI summary

A system and method for data recovery of large datasets is provided. The method may include generating a first version of a dataset having a plurality of segments. The first version may be generated by collecting a plurality of data inputs from a plurality of computing devices and storing these in a map. The method may further include generating a first delta map corresponding to changes in the plurality of segments for a second version of the dataset, wherein the second version is one of a plurality of versions of the dataset generated after the first version. In addition, the method may include generating a second delta map corresponding to changes in the plurality of segments for all versions prior to the second version. Further, the method may include generating a recovered version of the dataset by summing the first version, the first delta map, and the second delta map.