Stratifying Heterogeneous Data Samples via Hyper Information Frames
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for stratifying datasets in machine learning often result in biased data subsets, leading to overfitting and compromised model performance, especially when dealing with non-classified heterogeneous object labels and multi-label classifications.
Innovation Solution
The method involves extracting meta attributes from data samples, generating hyper information frames, converting non-numeric attributes to numeric values, reducing dimensionality, clustering, and stratifying data samples to create fair and unbiased training, validation, and test datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional methods are used to split datasets into training, validation, and test sets, then the process is simple and quick, but the resulting data subsets are biased and lead to overfitting
Solution Approach 1:
The patent segments the data stratification process into multiple distinct steps: extracting meta-attributes from data samples, generating hyper-information frames, converting non-numeric attributes to numeric values, reducing dimensionality, clustering samples into strata, and finally splitting into training/validation/test sets. This segmentation transforms a complex unreliable process into manageable steps that ensure statistical similarity across datasets.
Solution Approach 2:
The patent introduces hyper-information frames that add a new dimension to data representation by combining multiple attributes (spatial, temporal, spectral, etc.) into unified numeric vectors. This dimensionality transformation enables clustering algorithms to identify statistical similarities that would be invisible in traditional single-attribute representations, thereby reducing bias in data splitting.
2Reliability
If random sampling is used to create data subsets, then the process is straightforward, but certain data is systematically under-represented or over-represented leading to sampling bias
Solution Approach 1:
The patent performs preliminary clustering of data samples into strata based on statistical similarity before the actual data splitting process. By pre-organizing samples into homogeneous groups using hyper-information frames and clustering algorithms, the method ensures that subsequent random sampling from each stratum produces balanced, unbiased datasets with fair representation of all data characteristics.
Solution Approach 2:
The patent transforms non-numeric attributes (which are difficult to compare and cluster) into numeric values through the hyper-information frame conversion process. This parameter transformation enables the use of quantitative clustering methods to identify statistical similarities, ensuring that data samples with similar characteristics are grouped together regardless of their original attribute types, thereby eliminating systematic under-representation.
3Reliability
If detailed meta attributes are extracted and processed through multiple steps, then statistically similar datasets are achieved, but the processing time and computational resources increase
Solution Approach 1:
The patent extracts only the most relevant meta-attributes from data samples to construct hyper-information frames, rather than processing all possible attributes. This selective extraction focuses computational resources on the attributes that most significantly contribute to statistical similarity, reducing processing time while maintaining the reliability of dataset stratification.
Solution Approach 2:
The patent applies dimensionality reduction to discard redundant or less important attributes in the hyper-information frames while retaining the essential information needed for clustering. This discarding of unnecessary data reduces computational complexity and processing time, while the clustering algorithm recovers the statistical similarity information needed for reliable data stratification.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques for stratifying data samples for use in machine learning and/or data analytics. A method generally includes extracting one or more meta attributes from each respective data sample of a plurality of data samples in a dataset; generating a plurality of hyper information frames, wherein each respective hyper information frame is associated with a respective data sample of the plurality of data samples and comprises the data sample and at least a subset of the one or more meta attributes extracted from the respective data sample; converting any non-numeric attribute value in each hyper information frame of the plurality of hyper information frames into a numeric attribute value; generating reduced dimensionality hyper information frames; clustering the reduced dimensionality hyper information frames into a plurality of clusters; and stratifying the data samples by sampling from the plurality of clusters.