Operator Graph Data Integration Path Optimization

Solution Overview

Problem

The data integration process is costly and inefficient when dealing with large volumes of disparate, 'dirty' data from various sources, as existing methods require multiple processing steps and often rely on costly manual or automated operators without optimizing the selection of operators for quality and cost.

Innovation Solution

A system represents data integration as an operator graph with nodes and edges, using operator metrics to determine an optimal processing path that maximizes quality within a given cost budget, selecting operators to perform logical operations efficiently, and orchestrating the data integration process to achieve optimized results.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If multiple processing steps are used to clean and normalize data, then data quality is improved, but processing cost and time increase

Engineering Contradiction:

Improvedata qualityVSAvoidprocessing time

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The data integration process is segmented into multiple operators (e.g., data collection, cleaning, normalization, verification), each performing a specific function. The system represents these as nodes in an operator graph, allowing independent optimization of each segment while maintaining overall quality goals.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system dynamically selects the optimal path through the operator graph by considering both quality metrics and cost metrics of each operator. This dynamic selection allows the process to adapt to different data characteristics and budget constraints, balancing quality improvement with processing efficiency.

Inventive Principle:

Principle #15Dynamics

2Manufacturing precision

If multiple processing steps are used to clean and normalize data, then data quality is improved, but processing cost increases

Engineering Contradiction:

Improvedata qualityVSAvoidprocessing cost

Core Design Contradiction:

Manufacturing precisionVSLoss of energy

Solution Approach 1:

The data integration process is segmented into multiple operators (e.g., data collection, cleaning, normalization, verification), each performing a specific function. The system represents these as nodes in an operator graph, allowing independent optimization of each segment while maintaining overall quality goals.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system changes the parameters of operator selection by considering both quality metrics and cost metrics. It determines optimal processing paths by evaluating multiple operators with different quality-cost profiles, allowing flexible adjustment of processing intensity based on available budget and quality requirements.

Inventive Principle:

Principle #35Parameter changes

3Quantity of substance

If existing data integration methods are used, then data can be processed, but efficiency is reduced when dealing with large volumes of data

Engineering Contradiction:

Improvedata volumeVSAvoidprocessing efficiency

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The data integration process is segmented into multiple operators (e.g., data collection, cleaning, normalization, verification), each performing a specific function. The system represents these as nodes in an operator graph, allowing independent optimization of each segment while maintaining overall quality goals.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system incorporates feedback mechanisms where quality metrics and cost metrics are evaluated after each operator execution. This feedback information is used to determine optimal paths for subsequent processing steps, continuously improving efficiency as the system learns from actual performance data.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS9235652B1Optimizing a data integration process

Publication Date: 2016.01.12 BYTEDANCE INC

AI summary

Embodiments of the present invention provide systems, methods and computer readable media for optimizing a data integration process. In embodiments, a system can be configured to represent the processing of a data record that includes attributes, and to use that representation to determine an optimal processing of that data record. In embodiments, the system represents the processing of a data record as an operator graph comprising nodes and edges, where each node is an operator node that represents an operator for implementing at least one logical operation on at least one of the attributes and each edge between a pair of nodes represents the movement of data between the nodes. In embodiments, each operator node includes one or more operator metrics (e.g. operator cost metrics and operator quality metrics). In embodiments, the system determines optimal processing of the data record by determining a best path within the operator graph.