Data Attributes Recommendation Engine for Analytics Platform Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current collaborative analytics platforms face inefficiencies due to high data attribute requirements from analytics functions, leading to unnecessary resource consumption and slowed performance, as well as discouragement from data sharing due to excessive data demands.
Innovation Solution
A machine learning-based Data Attributes Recommendation Engine (DRE) optimizes resource utilization by identifying significant data sources and attributes for specific analysis results through training a model on data utilization patterns, using Restricted Boltzmann Machines to determine attribute weightings and provide recommendations for data ingestion and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If analytics functions require a comprehensive set of data attributes to ensure complete analysis capability, then analysis accuracy is improved, but system resource consumption increases and performance slows down
Solution Approach 1:
The patent applies partial action by training a machine learning model to identify and select only the most relevant subset of data attributes needed for each specific analytics function, rather than providing all available attributes. This allows the system to achieve sufficient analysis accuracy with fewer attributes, thereby improving system performance and reducing resource consumption.
Solution Approach 2:
The patent changes the parameter of data attribute selection from a static comprehensive set to a dynamic optimized subset. The machine learning model learns from historical data usage patterns and adjusts which attributes are selected based on the specific analytics function and context, optimizing the balance between analysis accuracy and system performance.
2Reliability
If analytics functions specify high requirements for data attributes to ensure thorough analysis, then analysis completeness is improved, but data sharing participation decreases
Solution Approach 1:
The patent enables data owners to share only the essential subset of attributes identified by the machine learning model as necessary for specific analytics functions, rather than requiring sharing of all attributes. This partial sharing approach maintains analysis completeness for targeted functions while reducing the burden on data owners, thereby increasing data sharing participation.
Solution Approach 2:
The system performs preliminary analysis using the trained machine learning model to determine which data attributes are actually needed before requesting data sharing. This preliminary identification of necessary attributes allows data owners to make informed decisions about sharing, increasing their willingness to participate while ensuring analysis completeness.
3Loss of information
If all available data attributes are processed to ensure no useful information is lost, then information completeness is improved, but resource utilization efficiency decreases
Solution Approach 1:
The patent extracts only the most relevant data attributes from the complete set available in the data lake, based on machine learning model predictions of which attributes will be useful for each analytics function. This extraction process prevents information loss for critical attributes while eliminating processing of redundant attributes, thereby improving resource utilization efficiency.
Solution Approach 2:
The system processes only a partial set of attributes that the machine learning model identifies as most valuable for each specific analytics task, rather than processing all attributes. This partial processing approach maintains information completeness for decision-critical attributes while significantly reducing computational resource consumption.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer implemented method of selecting a subset of data attributes from an incoming set of data that is to be processed by a set of data analytic functions, the method being performed by a computer system configured to provide the incoming set of data, having a plurality of data attributes, to a set of data analytic functions that produce one or more respective analysis results, the method comprising: defining a list of available data inputs for producing the plurality of respective analysis results, wherein the data inputs includes at least one data type and a set of data attributes associated with the respective data type; determining a utilisation of each data attribute of the set of data attributes that produces each of the respective analysis results; assigning a value to each data attribute based on the determined utilisations; constructing a set of training samples based on the set of data attributes associated with the at least one data type and the assigned values for producing the respective analysis results; training a model, based on the set of training samples, to determine a plurality of weightings of each of the set of data attributes for producing the respective analysis results; using the trained model, based on the determined plurality of weightings for producing the respective analysis results, to determine a plurality of data attributes of the incoming set of data that leads to the respective analysis results.