Entropy-Based Reasoning Models With Surprisal-Guided Data Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer-based reasoning systems face challenges in balancing the need for broad training data coverage with the requirement of reducing model size, leading to inefficiencies due to large memory usage and processing costs.
Innovation Solution
The use of entropy-based techniques to assess the informational value of data elements, allowing for the reduction of data sets by retaining only those with high surprisal or information gain, while directing training towards areas with greater informational value.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If broad training data coverage is obtained, then model coverage and reliability are improved, but model size and computing resource usage increase
Solution Approach 1:
The patent extracts only the most informative data elements from the training set by calculating surprisal values for each element. Data elements with high surprisal (high informational value) are retained, while redundant elements with low surprisal are removed. This extraction process reduces model size while preserving the essential information needed for reliable decision-making.
Solution Approach 2:
The patent changes the parameter of data element selection from uniform inclusion to selective inclusion based on surprisal calculation. By introducing the surprisal parameter (informational value metric), the system transforms the training data selection process to prioritize quality over quantity, thereby reducing model size while maintaining coverage of critical scenarios.
2Reliability
If all training data elements are retained, then comprehensive coverage is achieved, but processing time and computational cost increase
Solution Approach 1:
The patent removes redundant data elements from the training set by calculating and comparing surprisal values. Only data elements that provide significant informational value (high surprisal) are retained for training, while the majority of redundant elements are extracted and discarded. This dramatically reduces processing time and computational cost while maintaining training effectiveness.
Solution Approach 2:
The patent applies partial action by selectively processing only the most informative portion of the training data rather than all data elements. By focusing computational resources on high-surprisal data elements, the system achieves effective training with reduced computational effort, avoiding the excessive processing of redundant information.
3Quantity of substance
If data elements with low informational value are included, then data volume increases, but model learning efficiency decreases
Solution Approach 1:
The patent introduces the surprisal parameter to evaluate and filter data elements based on their informational value. By changing the selection criterion from quantity-based to quality-based (surprisal-based), the system excludes data elements with low informational value, thereby improving learning efficiency without sacrificing essential training coverage.
Data Source
AI summary
Techniques are provided herein for creating well-balanced computer-based reasoning systems and using those to control systems. The techniques include receiving a request to determine whether to include one or more particular data elements in a computer-based reasoning model and determining two probability density or mass functions (“PDMFs”), one for the data set including the one or more particular data elements, once for the data set excluding it. Surprisal is determined based on those two PDMFs, and inclusion in the computer-based reasoning model is determined based on surprisal. A system is later controlled using the computer-based reasoning model.