Data Warehouse Supply with Time-Controlled Buffer Prioritization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Real-time data warehousing faces challenges in maintaining transactional consistency due to varying processing times across parallel branches, leading to resource-intensive synchronization and high computing requirements, especially in large-scale systems like banks handling thousands of transactions per second.

Innovation Solution

A system with an interface for receiving transaction data, a process control component for distributing data elements to processing branches, buffer storages for temporary storage, and a mode control device that prioritizes data processing based on time control information to ensure continuous and transaction-consistent data delivery to the data warehouse.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data elements are distributed to parallel processing branches for real-time processing, then processing speed and productivity are improved, but transactional consistency deteriorates due to varying processing times across branches

Engineering Contradiction:

Improveprocessing speedVSAvoidtransactional consistency

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

A buffer storage component is introduced as an intermediary between the interface and processing branches. Data elements are temporarily stored in the buffer with time control information before being processed, allowing parallel processing to proceed while maintaining the ability to reconstruct consistent transaction views by retrieving all data elements of a transaction together after processing.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

Time control information is assigned to data elements before they enter the processing branches. This preliminary action enables subsequent prioritized processing based on transaction completion times, ensuring that data elements can be reassembled in the correct temporal order to maintain transactional consistency in the data warehouse.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If synchronization mechanisms are implemented to maintain transactional consistency across parallel branches, then data consistency is improved, but resource consumption and system complexity increase

Engineering Contradiction:

Improvedata consistencyVSAvoidsynchronization complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The synchronization function is extracted from the processing branches themselves and relocated to the buffer storage and transfer component. Instead of requiring complex inter-branch synchronization, each branch operates independently while the buffer and transfer mechanism handle consistency by storing and retrieving data elements based on time control information.

Inventive Principle:

Principle #2Taking out (Extraction)

3Reliability

If JOIN operations are performed at database level to collect data elements before transfer, then transactional consistency is improved, but computing resource requirements increase

Engineering Contradiction:

Improvetransactional consistencyVSAvoidcomputing resource

Core Design Contradiction:

ReliabilityVSPower

Solution Approach 1:

Data elements are pre-tagged with time control information and transaction identifiers before entering processing branches. This preliminary action eliminates the need for complex JOIN operations at database level, as the transfer component can directly retrieve complete transactions using the pre-assigned identifiers, significantly reducing computing resource requirements.

Inventive Principle:

Principle #10Preliminary action

4Reliability

If periodic batch processing is used to supply the data warehouse, then transactional consistency is maintained, but data availability and responsiveness deteriorate

Engineering Contradiction:

Improvedata consistencyVSAvoiddata availability

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system dynamically switches between continuous processing mode and batch processing mode. In continuous mode, processed data elements are immediately transferred to the data warehouse for real-time availability. In batch mode, data is collected and transferred together to optimize for transactional consistency, allowing the system to adapt to different operational requirements.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS7599968B2Technique for supplying a data warehouse whilst ensuring a consistent data view

Publication Date: 2009.10.06 UBS BUSINESS SOLUTIONS AG

AI summary

A technique is described for supplying a data warehouse while ensuring a consistent data view. The system comprises an interface for receiving data elements which are assigned to individual transactions and require processing before delivery into the data warehouse. Also provided is a process control component for analyzing the data elements with regard to the respective assignment to particular processing operations and for controlling the distribution of the data elements to associated processing branches. At least one first buffer storage is disposed downstream from the process control component, for the temporary storage of the distributed data elements, time-control information being assigned to each stored data element. The processing of the data elements read-out from the at least one buffer storage is effected in two or more parallel processing branches, a processing of the data elements that is non-dependent on the time control information being effected in a first mode, and a prioritizing processing of particular data elements that is based on the time control information being provided in a second mode. A transfer component, which transfers the processed data elements to the data warehouse, is disposed down-stream from the processing branches.