AI Reasoning Model Pruning for Coverage and Training Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer-based reasoning systems face challenges in achieving sufficient breadth of training data while reducing model size, leading to inefficiencies in computational and memory resources due to large model sizes and unnecessary features.
Innovation Solution
Utilizing entropy-based techniques to balance model size and breadth by selecting and training on data elements that provide high information gain or surprisal, while removing elements with low surprisal or anomalies, and using conviction measures to direct training towards informative contexts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the model includes more training data elements to achieve broader coverage, then the model's coverage and reliability improve, but the model size and computational resource requirements increase
Solution Approach 1:
The patent extracts and removes data elements with low surprisal values from the training model. By calculating surprisal for each data element and selectively removing those below a threshold, the system reduces model size while preserving elements that provide the most information gain, thus resolving the contradiction between coverage and model size.
Solution Approach 2:
The patent changes the parameter of data element selection from traditional importance metrics to surprisal-based metrics. By using surprisal calculation to evaluate and select data elements, the system optimizes which elements to retain, achieving better coverage with fewer elements and reducing the overall model size.
2Reliability
If the model includes more features and data elements, then the model's coverage improves, but the computational efficiency and training speed deteriorate
Solution Approach 1:
The patent extracts and removes features and data elements that contribute minimally to information gain. By calculating surprisal values and removing elements with low values, the system reduces the number of features and data elements in the model, directly improving training efficiency and computational speed while maintaining essential coverage.
Solution Approach 2:
The patent applies partial action by selectively processing only the most informative data elements rather than all elements equally. By focusing computational resources on high-surprisal elements and excluding low-surprisal ones, the system achieves efficient training with improved productivity.
3Reliability
If the model retains all collected training data elements, then the breadth of coverage is maximized, but the memory resources and computational overhead increase
Solution Approach 1:
The patent extracts and removes redundant data elements from the training set by calculating surprisal values and eliminating those with low information content. This reduction in the number of stored data elements directly decreases memory resource requirements while preserving the breadth of coverage through retention of high-value elements.
Solution Approach 2:
The patent changes the parameter for evaluating data element value from traditional metrics to surprisal-based metrics. This parameter change enables more efficient selection of which data elements to store in memory, achieving broad coverage with reduced memory consumption by storing only the most informative elements.
Data Source
AI summary
Techniques are provided herein for creating well-balanced computer-based reasoning systems and using those to control systems. The techniques include receiving a request to determine whether to use one or more particular data elements, features, cases, etc. in a computer-based reasoning model (e.g., as data elements, cases or features are being added, or as part of pruning existing features or cases). Conviction measures are determined and inclusivity conditions are tested. The result of comparing the conviction measure can be used to determine whether to include or exclude the feature, case, etc. in the model and/or whether there are anomalies in the model. A controllable system may then be controlled using the computer-based reasoning model. Examples controllable systems include self-driving cars, image labeling systems, manufacturing and assembly controls, federated systems, smart voice controls, automated control of experiments, energy transfer systems, health care systems, cybersecurity systems, and the like.