Multidimensional Data Indexing for Even Sample Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Selecting an evenly distributed set of training data from an unevenly distributed data corpus is challenging, particularly when the corpus contains tightly grouped clusters or frequently represented scenarios, as random selection tends to reflect these groupings or frequencies.
Innovation Solution
The system indexes a data corpus to a set of multidimensional points using an indexing module that maps samples to points in a multidimensional space, generating indexes that describe these mappings, allowing for the selection of an evenly distributed set of samples using algorithms like Sobol sequences to ensure uniform distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If random selection is used to select training data from a data corpus, then the selection process is simple and unbiased, but the selected samples will reflect the uneven distribution and clusters present in the original corpus
Solution Approach 1:
The patent applies preliminary action by pre-computing an index structure that maps data samples to multidimensional points before the actual selection process. This index is built once and then used to guide subsequent sample selections, ensuring even distribution without re-processing the entire corpus each time. The Sobol sequence points are generated in advance to define the target distribution.
Solution Approach 2:
The patent transforms the selection problem from the original data space to a multidimensional point space using Sobol sequences. By mapping samples to points in a higher-dimensional space and selecting based on these points, the method achieves uniform distribution that cannot be obtained through simple random selection in the original space.
2Reliability
If the data corpus contains tightly grouped clusters or frequently represented scenarios, then the data reflects real-world distributions, but random selection cannot produce an evenly distributed training set
Solution Approach 1:
The patent applies local quality by associating different regions of the multidimensional space with different data characteristics. Each Sobol point represents a specific region, and samples are selected based on their proximity to these points. This allows the selection process to locally adjust for over-represented or under-represented regions while maintaining global uniformity.
Solution Approach 2:
The patent changes the selection parameter from random probability to distance-based proximity to Sobol points. By using the multidimensional distance metric instead of uniform random sampling, the method transforms the selection criteria to achieve even distribution while preserving the underlying data structure and relationships.
Data Source
AI summary
Indexing a data corpus to a set of multidimensional points, including: generating a set of points in a multidimensional space; identifying, for each sample in a plurality of samples in a data corpus, a nearest point in the set of points; and generating an index mapping each sample with the nearest point in the set of points.