ETL Data Fusion for Real-Time ACID Compliance

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

As data processes scale up in terms of dataset dimensionality, data volume, data types, data content, and data source speed, existing technologies face complexity and computational intensity in monitoring and updating numerical values, especially when dealing with conflicts in format, type, or content, and compliance with ACID principles, particularly in distributed networks.

Innovation Solution

The system employs a processor to send multidimensional datasets from a massively parallel processing database to an ETL application, where datasets from different formats and sources are cleansed and fused, allowing real-time updates and report modifications without requiring manual hardcoded changes, thus simplifying data management and compliance with ACID principles for non-programmers.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the data process is programmed to monitor the numerical value in the data file in real-time, then the result can be kept updated, but the monitoring becomes logically complex and computationally intensive as the data process is scaled up

Engineering Contradiction:

Improvedata update timelinessVSAvoidmonitoring complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary component that sits between the data file and the data process. This intermediary monitors changes in the data file and automatically triggers updates to the data process, eliminating the need for complex real-time monitoring within the data process itself. The intermediary handles the complexity of change detection and notification, allowing the data process to remain simple while still achieving real-time update capability.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If the numerical value is hardcoded into the data process to avoid usage of the data file, then data availability can be improved, but modifying the numerical value requires specialized computer programming knowledge

Engineering Contradiction:

Improvedata availabilityVSAvoiddata modification ease

Core Design Contradiction:

ReliabilityVSEase of operation

Solution Approach 1:

The patent extracts the numerical value from the data process and places it in an external data file. This extraction allows the value to be easily accessible and modifiable without requiring changes to the data process code. The data process simply references the external file, maintaining data availability while enabling non-programmers to modify values by simply editing the data file.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent introduces an intermediary mechanism that bridges the external data file and the data process. This intermediary automatically detects changes in the data file and triggers appropriate updates in the data process, eliminating the need for hardcoded values while ensuring data availability. Users can modify values in the external file without programming knowledge, and the intermediary handles the rest.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If the data process is distributed over a network, then scalability can be improved, but data conflict management and ACID compliance become further complicated

Engineering Contradiction:

Improvedata process scalabilityVSAvoidconflict management complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent introduces a centralized intermediary component that manages data file changes across the distributed network. This intermediary acts as a single point of truth that coordinates updates between multiple data processes, automatically detecting conflicts and resolving them according to ACID principles. Each distributed data process simply communicates with the intermediary, which handles the complexity of conflict management and ensures consistency across the network.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Adaptability or versatility

If multiple data sources with different formats and types are integrated, then data versatility is improved, but data cleansing and fusion become computationally intensive

Engineering Contradiction:

Improvedata source diversityVSAvoiddata processing energy

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The patent applies preliminary data cleansing and standardization transformations to data from various sources before it enters the main data process. By pre-processing the data to establish consistent formats and structures upfront, the system reduces the computational burden during subsequent processing stages. This preliminary action ensures that diverse data sources can be integrated without excessive energy consumption during runtime operations.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11669538B1Systems, devices, and methods for data analytics

Publication Date: 2023.06.06 MASSACHUSETTS MUTUAL LIFE INSURANCE CO

AI summary

Various systems and methods use a value in a data file for a data process, as the data process is scaled up in terms of dataset dimensionality, data volume, data types, data content, data source quantity, and data source speed, while remaining compliant with ACID principles. As such, these technologies provide for sourcing of data from various data sources, where the data includes the data file storing the value. The data is cleansed and fused, which enables a report to be generated. In response to the value in the data file being modified, the data, inclusive of the data file storing the value, is again cleansed and fused based on the value being modified. This processing in-turn enables the report to modified based on the value being modified.