Distributed Data Warehouse Publication for Atomic Bulk Writes

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional distributed data warehouse systems face challenges in maintaining atomicity and efficiency during bulk operations, as they either compromise on atomicity by loading data twice or on efficiency by exposing new records at different times, and struggle to handle composition of bulk mutations across multiple tables.

Innovation Solution

The solution involves moving the responsibilities for large fact tables into user space, allowing external computing devices to perform operations normally reserved for the data warehouse's master node, by generating publication identifiers and version values for source data, and using these to populate fact tables and update publication tables, ensuring atomicity and efficiency through multi-version concurrency control.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional distributed data warehouse systems use master node locking and serialization to ensure data integrity, then data integrity is maintained, but system parallelism and performance are reduced

Engineering Contradiction:

Improvedata integrityVSAvoidsystem parallelism

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent extracts the transaction coordination function from the centralized master node and relocates it to distributed worker nodes. Each worker node independently manages its own transaction state and publishes results to a coordination table, eliminating the need for master node locking and serialization while maintaining data integrity through distributed consensus.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent segments the monolithic transaction management process into independent micro-tasks that can be executed in parallel by multiple worker nodes. Each worker node handles specific bulk load operations independently, with results published to a coordination table that enables atomic composition without centralized control.

Inventive Principle:

Principle #1Segmentation

2Productivity

If bulk load operations are performed without transactional guarantees for higher performance, then system performance improves, but atomicity of composed transactions is lost

Engineering Contradiction:

Improvebulk load performanceVSAvoidtransaction atomicity

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent performs preliminary actions by having worker nodes publish their bulk load results to a coordination table before final commit. This preliminary publication allows the system to verify all operations succeeded before making the transaction atomic, enabling both high performance and transactional guarantees.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces a coordination table as an intermediary between worker nodes and the final commit operation. Worker nodes publish their results to this table, which acts as a mediator to verify all operations completed successfully before allowing atomic composition, thus maintaining transactional guarantees without sacrificing performance.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Speed

If data is exposed incrementally during bulk operations to maintain performance, then system responsiveness improves, but atomicity of the bulk operation is compromised

Engineering Contradiction:

Improvequery responsivenessVSAvoidbulk operation atomicity

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

The patent performs preliminary publication of bulk load results to a coordination table before exposing data to queries. This allows the system to verify atomicity of all bulk operations before making data visible, ensuring both responsiveness and atomicity by separating the publication phase from the exposure phase.

Inventive Principle:

Principle #10Preliminary action

4Reliability

If multiple copies of data are loaded to maintain atomicity, then transaction atomicity is preserved, but storage overhead and complexity increase

Engineering Contradiction:

Improvetransaction atomicityVSAvoidstorage overhead

Core Design Contradiction:

ReliabilityVSLoss of substance

Solution Approach 1:

The patent extracts the atomicity guarantee mechanism from traditional approaches that require multiple data copies and replaces it with a coordination table that tracks publication status. Worker nodes publish results once to the coordination table, eliminating the need for multiple data copies while preserving atomicity through centralized verification.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS11681698B2Systems and methods for atomic publication of distributed writes to a distributed data warehouse

Publication Date: 2023.06.20 VOLKSWAGEN GROUP OF AMERICA INVESTMENTS LLC

AI summary

Systems and methods for managing data. The methods comprise by a computing system: generating publication identifiers and version values for source data to be stored into a data warehouse; causing a plurality of fact tables in the data warehouse to be populated with the source data and the publication identifiers; causing a publication table in the data warehouse to be updated to include the publication identifiers and the version values so as to be respectively associated with resource names; receiving a query for information directed to the plurality of fact tables; retrieving the publication identifiers from the publication table, in response to the query; and obtaining source data from each said fact table of the plurality of fact tables that is associated with publication identifiers that are stored in both the fact table and the publication table.