Ensemble Model Evaluation for Unknown Data Relevance Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges in predicting the performance on unknown data due to insufficient learning data and unique lawsuit-specific circumstances, leading to decreased precision and difficulty in repurposing labeled data across different lawsuits.
Innovation Solution
A method involving multiple machine learning models with different settings and datasets, combined with a neural network, to enhance the reliability of predicting document relevance by employing ensemble learning and cross validation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single machine learning model is used with limited learning data, then the model can be trained quickly, but the prediction precision on unknown data decreases
Solution Approach 1:
The patent combines multiple machine learning models with different settings into a single ensemble model. Each model is trained on different subsets of learning data, and their predictions are aggregated to produce the final output. This merging approach improves prediction precision on unknown data while managing complexity through systematic integration.
Solution Approach 2:
The patent divides the learning data into multiple subsets and trains separate machine learning models on each subset. By segmenting the data and creating specialized models for different portions, the system achieves better generalization performance when predicting on unknown data compared to using a single model trained on all data.
2Reliability
If machine learning models are trained on lawsuit-specific data, then the models can achieve good performance on that specific task, but the models cannot be repurposed for other lawsuits
Solution Approach 1:
The patent creates machine learning models that are designed to be applicable across multiple lawsuits rather than being specialized for a single lawsuit. By training models on diverse data subsets and using ensemble methods, the system produces models that maintain reliability across different legal cases and can be repurposed for various lawsuits.
Solution Approach 2:
The patent employs dynamic model selection and configuration where the ensemble of models can adapt to different lawsuit contexts. The system maintains multiple trained models that can be selectively applied or reconfigured based on the specific characteristics of different lawsuits, enabling both reliability and versatility.
3Measurement precision
If cross validation is performed with insufficient learning data, then the evaluation process can be completed, but the reliability of performance prediction decreases
Solution Approach 1:
The patent segments the limited learning data into multiple distinct subsets for training different models in the ensemble. By carefully dividing the available data and creating multiple models that each learn from different portions, the system achieves more reliable cross-validation results and better performance predictions despite having insufficient overall learning data.
Data Source
AI summary
An information processing method includes: obtaining a first evaluation result representing performance of a first machine learning model having learned using first learning data, the first evaluation result being calculated using first validation data; obtaining a second evaluation result representing performance of a second machine learning model having learned using second learning data, the second evaluation result being calculated using second validation data; and calculating, based on the first evaluation result and the second evaluation result, a comprehensive evaluation result representing performance of a single machine learning model including the first machine learning model and the second machine learning model, the performance of the single machine learning model being predicted when the single machine learning model is applied to unevaluated, unknown data relevant to a prescribed event.