Data Warehouse Batch Isolation with Rollback

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional database management systems require offline periods during data transfer, leading to service disruptions and data integrity issues, as they struggle to maintain data availability and prevent corruption while updating data records.

Innovation Solution

A server-implemented method for incremental data transfer that creates a temporary data repository with the same configuration as the target repository, identifies dependency relationships, and merges new data records with existing ones, allowing continuous user access and minimizing downtime.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional data transfer methods are used to update the data repository, then data integrity is maintained, but the data repository must be taken offline causing service disruptions

Engineering Contradiction:

Improvedata integrityVSAvoidservice availability

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent divides the data repository into multiple partitions or segments that can be updated independently. Instead of taking the entire repository offline, only specific partitions are updated while others remain accessible to users, thus maintaining service availability while ensuring data integrity through controlled updates.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary mechanism (such as a temporary staging area or version control system) that holds new data records before they are committed to the main repository. This intermediary allows validation and verification of data integrity before making changes visible to users, preventing corruption while maintaining continuous access.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If the data repository is kept online during data transfer, then service availability is maintained, but data corruption or incompleteness may occur

Engineering Contradiction:

Improveservice availabilityVSAvoiddata integrity

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent performs preliminary actions by validating and preparing new data records in a staging area before they are merged into the live repository. This preliminary validation ensures data integrity is verified beforehand, allowing the repository to remain online without risk of corruption from incomplete or invalid data transfers.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent replaces the mechanical approach of completely stopping data access during updates with a software-based concurrency control mechanism. This allows multiple operations (read and write) to occur simultaneously through proper locking, versioning, or transaction management, maintaining availability while ensuring integrity.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Stability of the object's composition

If all interrelated data records are updated together, then data consistency is maintained, but the offline period is extended

Engineering Contradiction:

Improvedata consistencyVSAvoidoffline period

Core Design Contradiction:

Stability of the object's compositionVSLoss of time

Solution Approach 1:

The patent segments the data repository into independent partitions that can be updated separately. Related data records are grouped into logical units or micro-batches that can be updated in smaller, manageable transactions rather than requiring a complete offline period for all records simultaneously, reducing downtime while maintaining consistency within each segment.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements dynamic update strategies where the system adapts the update process based on data dependencies and criticality. Less critical data can be updated asynchronously or in the background, while critical data receives priority updates, allowing the system to maintain consistency for essential records without extending the overall offline period.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS11733905B1Data warehouse batch isolation with rollback and roll forward capability

Publication Date: 2023.08.22 MASSACHUSETTS MUTUAL LIFE INSURANCE CO

AI summary

Methods and systems disclosed herein allow data to be transferred from a data source to a target database with little to no offline period or data corruptions. The methods and systems describe a server that generates a temporary data repository having a similar configuration as the target data repository; transmits the set of new data records from the data source to the temporary data repository; identifies dependency relationship attributes among the data records stored within the target data repository; and when the server identifies that a predetermined number of data records and their respective dependent data records are stored within the temporary data records, the server merges the set of data records and the set of new data records. The server also stores a pre/post merger record of data such that the server can revert to a previous version of data or roll forward to another version.