Feature Vector Transformation for Representative Data Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for distilling data sets for machine learning models are computationally intensive and have poor interpretability, consuming excessive resources and labor in training processes.
Innovation Solution
A method involving transformations to generate feature vectors with reduced distribution skewness and adjusted vector distances, allowing for the selection of representative target samples from a data set, thereby reducing the amount of data needed for training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large data sets are used for training machine learning models, then training accuracy is improved, but computing resource consumption increases excessively
Solution Approach 1:
The patent extracts representative samples from the original large data set through a multi-stage filtering process. First, samples are selected based on feature vector distribution characteristics (skewness and distance metrics), then representative samples are identified through iterative optimization. This extraction process creates a reduced data set that preserves the essential information needed for accurate training while significantly reducing the volume of data requiring processing.
Solution Approach 2:
The patent transforms the original data set by changing key parameters of the feature vectors. Specifically, it modifies the distribution characteristics by adjusting skewness parameters and distance metrics between vectors. These parameter changes enable the system to identify and select a subset of samples that maintains the statistical properties necessary for training accuracy while reducing overall data volume.
2Reliability
If large data sets are used for training machine learning models, then training accuracy is improved, but labeling labor consumption increases excessively
Solution Approach 1:
The patent extracts and selects representative samples that can serve as proxies for the entire data set. By identifying samples with specific distribution characteristics (low skewness, optimal distances), the system reduces the number of samples that require manual labeling while preserving the diversity and representativeness needed for accurate model training.
Solution Approach 2:
The patent performs preliminary automated filtering and selection of representative samples before the labeling process. By pre-organizing and pre-selecting the most informative samples based on feature vector properties, the system reduces the total number of samples that require human labeling effort, thereby reducing time consumption.
3Quantity of substance
If conventional data distillation methods are used, then data set size is reduced, but computational intensity and lack of interpretability increase
Solution Approach 1:
The patent segments the data reduction process into distinct, interpretable stages: (1) computing feature vectors for all samples, (2) calculating distribution skewness metrics, (3) computing distance metrics between vectors, (4) selecting representative samples based on these metrics, and (5) forming the reduced data set. This segmentation makes the complex distillation process more manageable and interpretable while effectively reducing data set size.
Data Source
AI summary
Embodiments disclosed herein include a method, an electronic device, and a computer program product for data processing. The method includes determining a first set of feature vectors representing samples in a data set. The method also includes generating a second set of feature vectors by performing a first transformation on the first set of feature vectors, wherein distribution skewness of the second set of feature vectors in a feature space is smaller than that of the first set of feature vectors. The method also includes generating a third set of feature vectors by performing a second transformation on the second set of feature vectors, wherein the third set of feature vectors and the second set of feature vectors have different distances between vectors. The method also includes selecting target samples as representatives from the samples based on a distribution of the third set of feature vectors in the feature space.