Generalized Feature Clustering for Predictive Model Data Minimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge lies in minimizing data collection while maintaining the performance of predictive models, particularly under regulations like GDPR, and reducing data storage and computational resources without compromising decision-making quality.
Innovation Solution
A method involving a 2-phase clustering process to determine alternative generalized features, where instances are grouped into buckets based on similarity thresholds and then clustered, allowing for data minimization without affecting model performance by using features indicative of cluster membership.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all data is collected and stored indefinitely, then data availability for analysis is improved, but data storage requirements and computational resources increase overwhelming amounts
Solution Approach 1:
The patent extracts only the most relevant and necessary features from the complete dataset for model training. Instead of using all available data, the system identifies and extracts a minimal subset of features that maintain predictive performance while significantly reducing storage requirements and computational overhead.
Solution Approach 2:
The patent transforms the original high-dimensional feature space into a lower-dimensional representation by changing the parameters from individual feature values to cluster assignments. This parameter transformation reduces data volume while preserving the essential patterns needed for prediction.
2Quantity of substance
If data minimization policy is implemented keeping only relevant data, then data storage is reduced, but model performance may deteriorate
Solution Approach 1:
The patent merges multiple similar instances into clusters based on feature similarity. By combining instances with comparable characteristics into unified clusters, the system reduces data volume while maintaining the predictive information contained in individual instances, thus preserving model performance.
Solution Approach 2:
The patent performs preliminary clustering and feature selection before model training. This preliminary action identifies and retains only the most informative features and instances, ensuring that the reduced dataset maintains sufficient information for accurate predictions while minimizing storage requirements.
3Quantity of substance
If clustering is performed to group similar instances, then data storage and computational resources are reduced, but processing complexity during clustering increases
Solution Approach 1:
The patent segments the feature space into distinct clusters based on similarity thresholds. By dividing the continuous feature space into discrete segments or clusters, the system simplifies the clustering process and reduces computational complexity while still achieving effective data reduction and organization.
Data Source
AI summary
A method, apparatus and a product for data generalization for predictive models. The method comprising: based on a labeled dataset, determining a plurality of buckets, each of which has an associated label; determining a plurality of clusters, grouping similar instances in the same bucket; based on the plurality of clusters, determining an alternative set of features comprising a set of generalized features, wherein each generalized feature corresponds to a cluster of the plurality of clusters, wherein a generalized feature that corresponds to a cluster is indicative of the instance being mapped to the corresponding cluster; obtaining a second instance; determining a generalized second instance that comprises a valuation of the alternative set of features for the second instance; and based on the generalized second instance, determining a label for the second instance.