Feature Selection Using Conditional Mutual Information
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional feature selection methods using mutual information are inefficient when considering interactions of more than three features and do not account for features that are relevant together but not individually, nor do they consider redundancy between features effectively.
Innovation Solution
A feature selection method using conditional mutual information to assess synergy and redundancy among multiple features, generating a subset of features that best represent a training document for classification, and utilizing binary representatives of features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional mutual information methods are used for feature selection, then the process is simple and fast, but it cannot effectively handle interactions of more than three features and does not account for features that are relevant together but not individually
Solution Approach 1:
The patent changes the parameter from standard mutual information to conditional mutual information, which allows the system to measure the dependency between features and target variable while conditioning on other features. This enables the system to detect interactions between more than three features and identify features that are relevant together but not individually, directly resolving the technical contradiction between selection efficiency and interaction detection accuracy.
2Reliability
If feature selection considers more feature interactions, then classification accuracy improves, but computational complexity and processing time increase
Solution Approach 1:
The patent performs preliminary feature ranking using individual mutual information scores before applying conditional mutual information for interaction detection. This preliminary action filters out obviously irrelevant features, reducing the search space for subsequent interaction analysis and thereby lowering computational complexity while maintaining high classification accuracy through comprehensive interaction consideration.
Solution Approach 2:
The patent uses a hybrid approach that combines partial consideration of individual feature relevance with excessive consideration of feature interactions through conditional mutual information. By focusing computational resources on interaction detection only for features that pass a relevance threshold, the system achieves high classification accuracy without proportionally increasing overall computational complexity.
3Productivity
If feature selection removes redundant features, then model simplification and training time reduction are achieved, but information loss may occur
Solution Approach 1:
The patent implements a feedback mechanism where conditional mutual information scores are continuously updated as features are selected or removed from the feature set. This feedback allows the system to dynamically adjust feature selection decisions, ensuring that redundant features are removed only when their conditional information contribution is negligible, thereby minimizing information loss while maximizing training efficiency.
Data Source
AI summary
Methods and systems for selecting a selected-sub-set of features from a plurality of features for training a machine learning module, the training of the machine learning module to enable classification of an electronic document to a target label, the plurality of features associated with the electronic document. In one embodiment, the method comprises analyzing a given training document to extract the plurality of features, and for a given not-yet-selected feature of the plurality of features: generating a set of relevance parameters iteratively, generating a set of redundancy parameters iteratively and determining a feature significance score based on the set of relevance parameters and the set of redundancy parameters. The method further comprises selecting a feature associated with a highest value of the feature significance score and adding the selected feature to the selected-sub-set of features.