Regression-Based Data Schema Generation for Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating data schemas for big data inputs are either resource-intensive or inaccurate, as they often rely on analyzing all input data or using random sample analysis, which can be time-consuming and may not accurately represent the data.
Innovation Solution
A computer-implemented method that uses a regression model to select the best methods for generating data schemas based on historic data, and then merges candidate schemas using predetermined rules to create a master data schema, thereby improving accuracy and reducing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all input data is analyzed to generate data schemas, then schema accuracy is improved, but resource consumption and processing time increase
Solution Approach 1:
The patent applies partial action by selecting a subset of methods from the available schema generation methods based on regression model predictions. Instead of applying all possible methods to all data, the system selectively applies only the most promising methods identified through historical performance analysis, thereby reducing computational resources while maintaining schema accuracy.
Solution Approach 2:
The system changes parameters dynamically by using a regression model to predict which schema generation methods will perform best for specific data categories. This allows the system to adapt its approach based on data characteristics rather than using a fixed, resource-intensive methodology for all cases.
2Use of energy by moving object
If random sample analysis is used to generate data schemas, then resource consumption is reduced, but schema accuracy deteriorates
Solution Approach 1:
The patent introduces an intermediary layer - the regression model - that predicts the effectiveness of different schema generation methods for specific data categories. This intermediary enables the system to make informed decisions about which methods to apply, avoiding both the resource waste of analyzing all data and the inaccuracy of random sampling.
Solution Approach 2:
The system performs preliminary action by using the regression model to pre-assess which methods are most likely to succeed for a given data category before actually generating schemas. This preliminary assessment guides the selection of methods, ensuring that resources are focused on the most promising approaches rather than relying on random sampling.
3Measurement precision
If multiple schema generation methods are applied and merged, then schema accuracy is improved, but device complexity increases
Solution Approach 1:
The patent segments the schema generation process by dividing it into distinct phases: data categorization using association rules, regression model prediction of method effectiveness, selective method application, and schema merging. This segmentation manages complexity by organizing the multi-method approach into structured, manageable steps rather than a monolithic complex process.
Solution Approach 2:
The system introduces dynamics by making the selection of schema generation methods adaptive rather than static. The regression model dynamically determines which methods to apply based on the specific data category and historical performance, allowing the system to optimize its complexity level for each case rather than maintaining high complexity for all cases.
Data Source
AI summary
Systems and computer-implemented methods select a subset of methods to generate data schemas for input data from a list of methods for generating data schemas, based on output of a regression model; generate a candidate schema for each method in the subset of methods to generate data schemas; and generate a master data schema for the input data by merging the candidate schema for each method in the subset of methods to generate data schemas, utilizing predetermined rules.