Analytics Workflow Optimization via Automated Subset Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual optimization of analytics workflows for varying data sets and differing requirements is time-consuming and costly, especially in large analytic systems, due to the need for manual reconfiguration to achieve desired precision, recall, and resource efficiency.
Innovation Solution
A method and system that generate and apply multiple subsets of analytics to data sets, record performance values, and calculate an optimal subset based on defined optimization goals, reducing the need for manual intervention by determining the most resource-efficient and precise analytic workflow configuration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual optimization of analytics workflows is performed to achieve desired precision and recall, then measurement precision is improved, but loss of time increases
Solution Approach 1:
The system performs self-optimization by automatically generating multiple analytics workflow configurations, evaluating their performance metrics (precision, recall, resource efficiency), and selecting optimal configurations without requiring manual intervention. The analytics system evaluates its own workflows and iteratively improves performance through automated feedback loops.
Solution Approach 2:
The system pre-generates multiple analytics workflow configurations and pre-evaluates their performance characteristics before actual deployment. By preparing and assessing multiple candidate configurations in advance, the system identifies optimal workflows beforehand, avoiding time-consuming manual optimization during production use.
2Measurement precision
If manual optimization of analytics workflows is performed to achieve desired precision and recall, then measurement precision is improved, but productivity decreases
Solution Approach 1:
The system automatically optimizes its own analytics workflows by generating configurations, evaluating performance metrics including resource efficiency, and selecting optimal setups without manual intervention, thereby maintaining high precision while improving productivity through automation.
Solution Approach 2:
The system varies multiple parameters across different analytics workflow configurations (including data sampling rates, analysis depth, resource allocation) and systematically evaluates their impact on both precision and resource efficiency. By changing and optimizing multiple parameters simultaneously, the system achieves high precision with improved resource utilization.
3Productivity
If multiple subsets of analytics are applied to data sets with defined optimization goals, then productivity is improved, but device complexity increases
Solution Approach 1:
The system divides the analytics workflow into multiple independent subsets or modules that can be generated, evaluated, and optimized separately. Each subset represents a discrete configuration that can be independently assessed against optimization goals, allowing parallel processing and reducing overall system complexity while improving configuration speed.
Solution Approach 2:
The system systematically varies parameters across multiple analytics subsets (such as data sampling rates, analysis depth, resource allocation) and evaluates their performance. By managing parameter variations in a structured manner across multiple subsets, the system achieves rapid workflow configuration while maintaining manageable complexity through organized parameter control.
Data Source
AI summary
Embodiments of the present invention disclose a method, computer program product, and system for optimizing data analysis. A set of analytics are received and a plurality of subsets of the set of analytics is generated. An optimization goal(s) is defined. The plurality of subsets of analytics are applied to a set of data. The output of applied analytics are recorded on a data storage device by the computer. Performance values of the computer applying the plurality of subsets of analytics are recorded and displayed. An optimal subset of the plurality of subsets of analytics is calculated, using recorded performance values and outputs of the applied plurality of subsets of analytics, being based on the optimization goal. A set of applied analytics are displayed, where the set of applied analytics includes a number of subsets of analytics of the plurality of analytics that meet the criteria of the optimal subset.