Automated Data Analytics Lifecycle for Dynamic Resource Provisioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data analytics solutions face limitations in handling increasing data set sizes and varying structures, leading to cost uncertainties and inflexibility, resulting in costly computing systems that inadequately manage the data they are intended to handle.
Innovation Solution
The method involves defining a data analytic plan, obtaining test and training data sets, fitting and evaluating models, and automating the data analytics lifecycle to provision a computing system, allowing for flexible adjustments and retraining of models within a controlled environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional data analytics solutions are used to handle increasing data set sizes and varying structures, then the computing system can process the data, but the cost increases and the system becomes inflexible
Solution Approach 1:
The patent implements dynamic provisioning of computing resources that can automatically scale and adapt based on the actual data set size and structure. The system adjusts computational resources in real-time to match workload demands, transforming the static conventional approach into a dynamic one that maintains flexibility while handling varying data quantities.
Solution Approach 2:
The system changes key parameters such as computing resource allocation, data sampling rates, and model complexity dynamically based on data characteristics. By adjusting these parameters according to the actual data set size and structure, the system maintains cost-effectiveness and flexibility without requiring over-provisioning for maximum possible data volumes.
2Reliability
If conventional data analytics solutions are provisioned to handle large data sets, then the system has sufficient capacity, but the cost becomes uncertain and excessive
Solution Approach 1:
The patent applies partial action by using data sampling techniques to analyze representative subsets of the full data set rather than processing every single record. This allows the system to maintain reliable analytical results while consuming significantly fewer computing resources, thereby reducing cost uncertainty and excessive energy consumption associated with processing complete large-scale data sets.
Solution Approach 2:
The system performs preliminary analysis and data exploration before committing to full-scale processing. By evaluating data characteristics, estimating required resources, and selecting appropriate analysis methods in advance, the system avoids costly over-provisioning and ensures that sufficient capacity is allocated only when and where needed, reducing overall computing costs while maintaining reliability.
3Ease of manufacture
If the data analytics solution is defined with fixed parameters, then the implementation is straightforward, but the system cannot adapt to changing data characteristics or requirements
Solution Approach 1:
The patent implements self-service through automated resource provisioning and configuration systems that automatically adjust computing resources and analysis parameters based on detected data characteristics. The system monitors data inputs, evaluates performance metrics, and autonomously reconfigures itself without requiring manual intervention, thereby maintaining ease of implementation while achieving high adaptability to changing requirements.
Solution Approach 2:
The system incorporates continuous feedback loops that monitor data characteristics, model performance, and resource utilization. This feedback information is used to dynamically adjust solution parameters and resource allocation, enabling the system to adapt to changing data characteristics while maintaining straightforward automated operation through established feedback control mechanisms.
Data Source
AI summary
A data analytic plan is defined for analyzing a given data set associated with a given data problem. A test data set and a training data set are obtained from the given data set associated with the given data problem. At least one model is executed to confirm an adequacy of the at least one model for the data analytic plan by fitting the at least one model on the training data set and evaluating the at least one model fitted on the training data set against the test data set. The defining, obtaining and executing steps are performed on one or more processing elements associated with a computing system and automate at least part of a data analytics lifecycle.