Analytic Flow Optimization via Runtime Statistics

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Optimizing analytic data flows in business intelligence is challenging due to the labor-intensive and time-consuming nature of creating correct analytic data flows, which often rely on the expertise of flow designers and are prone to inaccuracies from erroneous or outdated statistics, leading to suboptimal cost models.

Innovation Solution

The system automatically collects runtime statistics and combines them with historical statistics to generate accurate cost models, using a processor and statistical analysis module to sample source data, execute flows, and optimize analytic flows without requiring manual effort or changes to existing tools.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If manual optimization of analytic flows is performed by flow designers, then expertise-based optimization can be achieved, but the process becomes labor-intensive and time-consuming

Engineering Contradiction:

Improveaccuracy of cost predictionsVSAvoidtime required for flow optimization

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system enables self-service optimization by automatically collecting runtime statistics and generating optimized flow designs without requiring manual intervention from flow designers. The statistical analysis module autonomously processes execution data and produces optimization recommendations, eliminating the labor-intensive manual optimization process while maintaining high accuracy through data-driven insights

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces the mechanical manual process of flow optimization with an automated statistical analysis system. Instead of relying on human designers to manually analyze and optimize flows, the system uses computational methods to automatically collect runtime statistics, analyze performance data, and generate optimized flow designs, thereby reducing time consumption while maintaining or improving optimization quality

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Measurement precision

If manual optimization of analytic flows is performed by flow designers, then expertise-based optimization can be achieved, but the process becomes labor-intensive

Engineering Contradiction:

Improveaccuracy of cost predictionsVSAvoideffort required for flow optimization

Core Design Contradiction:

Measurement precisionVSEase of operation

Solution Approach 1:

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The optimization system is designed to be universally applicable to various flow design and execution tools. The statistical analysis module can integrate with different analytics platforms and execution engines, providing automated optimization capabilities across multiple systems without requiring tool-specific manual processes, thereby reducing overall effort while maintaining accuracy

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Measurement precision

If ad-hoc optimization process is used, then flexibility can be maintained, but the result is largely dependent on the abilities and experience of the flow designer

Engineering Contradiction:

Improveaccuracy of cost predictionsVSAvoidconsistency of optimization results

Core Design Contradiction:

Measurement precisionVSAdaptability or versatility

Solution Approach 1:

The system implements feedback mechanisms by continuously collecting runtime statistics from flow executions and using this data to iteratively improve optimization accuracy. The statistical analysis module processes actual execution performance data and feeds it back into the optimization process, ensuring consistent and reliable results that are not dependent on individual designer expertise. This data-driven feedback loop maintains adaptability while standardizing optimization quality

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system performs preliminary statistical analysis and cost model generation before flow execution, allowing optimization decisions to be made based on pre-computed statistics rather than ad-hoc manual analysis. This preliminary action ensures consistent optimization results by establishing data-driven baselines and patterns before actual flows run, reducing dependency on individual designer abilities while maintaining flexibility through automated adaptability

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS9600786B2Optimizing analytic flows

Publication Date: 2017.03.21 HEWLETT PACKARD ENTERPRISE DEV LP

AI summary

A technique of optimizing analytic flows includes sampling source data using a sampling method, executing a flow over the sampled data, obtaining runtime statistics from the executed flow, and combining runtime statistics with historical statistics.