Information Integration Flow Graph Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Information integration from multiple heterogeneous sources is costly in terms of computing resources and time, requiring efficient optimization of information integration flow plans to reduce costs and improve performance.
Innovation Solution
An information integration optimization system that uses heuristics to modify existing flow plans by applying transitions such as swap, distribution, partitioning, replication, and add shedding to create a state space of modified flow graphs, optimizing the flow graph structure and costs through a GUI engine, cost estimator, and state space manager.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If information integration is performed from multiple heterogeneous sources, then data completeness and analysis capability are improved, but computing resource consumption and time cost increase
Solution Approach 1:
The patent segments the information integration process into distinct flow operations (extract, transform, load, filter, join, etc.) that can be independently analyzed and optimized. Each operation is represented as a node in a flow graph, allowing selective optimization of specific segments without reprocessing the entire integration pipeline, thus reducing computing resource consumption while maintaining data completeness.
Solution Approach 2:
The patent performs preliminary analysis of the flow graph to identify redundant operations, optimization opportunities, and critical paths before executing the full information integration. By pre-processing the flow plan to eliminate unnecessary operations and optimize data paths, the system reduces computing resource consumption and time cost while preserving the necessary data integration functionality.
2Reliability
If information integration from multiple sources is performed, then decision-making capability is improved, but time consumption increases
Solution Approach 1:
The patent identifies and skips redundant or low-value flow operations in the information integration process. By analyzing the flow graph to detect unnecessary transformations, duplicate data retrievals, or non-critical processing steps, the system can bypass these operations and proceed directly to essential integration tasks, thereby reducing time consumption while maintaining the reliability of decision-making capabilities.
Solution Approach 2:
The patent optimizes execution parameters of flow operations based on the specific characteristics of the data sources and integration requirements. By dynamically adjusting parameters such as batch sizes, parallelism levels, and transformation complexity, the system accelerates the information integration process without compromising the quality and reliability of the integrated data for decision-making.
3Productivity
If flow plans are optimized to reduce costs, then computing efficiency is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal optimization framework that handles multiple types of flow operations (extract, transform, load, filter, join, aggregate) through a common set of optimization techniques. The flow graph representation and analysis methods are applicable across diverse information integration scenarios, allowing the system to achieve computing efficiency improvements without requiring separate complex optimization mechanisms for each operation type, thus controlling system complexity.
Data Source
AI summary
A computer implemented method and apparatus calculate a freshness cost for each of a plurality of information integration flow graphs and select one of the plurality of information integration flow graphs based upon the calculated freshness cost.