Dataset Sampling via Pattern Profiling for Data Diversity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods fail to efficiently generate a smaller dataset that is both representative of and significantly smaller than a larger dataset, often resulting in a smaller dataset that does not adequately represent the diversity of record formats in the larger dataset, leading to inefficient computational processing and inaccurate trend identification.
Innovation Solution
The solution involves profiling the larger dataset to identify patterns, then sampling records to create a smaller dataset that represents as many unique patterns as possible, with a focus on increasing representation of unique pattern combinations across multiple data columns, ensuring the smaller dataset is filled with records that are representative of the diversity of record formats in the larger dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If random selection is used to create a smaller dataset, then the dataset size is reduced, but the representativeness of record diversity is compromised
Solution Approach 1:
The patent applies preliminary action by profiling the larger dataset before sampling to identify all unique record formats and patterns. This pre-analysis enables the subsequent sampling process to deliberately select records that cover all identified patterns, ensuring representativeness is maintained while achieving dataset reduction. The profiling step creates a roadmap for selective sampling that prevents loss of diversity.
Solution Approach 2:
The patent changes the sampling parameter from random selection to pattern-based selection. By transitioning from uniform random sampling to stratified sampling based on identified record formats, the system maintains representativeness of all pattern types while reducing overall dataset size. Each pattern is represented proportionally or equally in the smaller dataset, preserving the diversity structure.
2Reliability
If a larger sample size is used to improve representativeness, then the computational efficiency is reduced
Solution Approach 1:
The patent applies partial action by selecting only the minimum necessary number of records needed to represent all unique patterns in the larger dataset. Rather than sampling a large proportion of records, the system identifies the essential subset that covers all pattern types. This selective partial sampling achieves representativeness with a smaller dataset size, thereby maintaining computational efficiency.
3Ease of operation
If existing sampling methods are used, then the process is simple, but the resulting dataset does not adequately represent unique pattern combinations
Solution Approach 1:
The patent applies preliminary action by performing a comprehensive profiling step before sampling to identify all unique record formats and their combinations across multiple data columns. This pre-analysis creates a detailed map of pattern distribution that guides the subsequent selective sampling process, ensuring that all unique pattern combinations are captured in the smaller dataset with high accuracy.
Solution Approach 2:
The patent changes the sampling approach from simple random selection to multi-dimensional pattern-based selection. By considering combinations of patterns across multiple columns simultaneously, the system accurately represents the joint distribution of record formats. This enhanced sampling methodology improves manufacturing precision of pattern representation while maintaining reasonable operational complexity.
Data Source
AI summary
A method for generating a smaller dataset from a larger dataset, each dataset holding a plurality of records, includes profiling the larger dataset to identify a plurality of patterns, each of which is descriptive of one or more records held in the larger dataset. A plurality of slots of the smaller dataset is filled with records held in the larger dataset. Multiple records held in the larger dataset are individually retrieved, and for each retrieved record it is determined whether to place the retrieved record into a slot of the smaller dataset and evict a record already occupying that slot, or not place the retrieved record into the smaller dataset. This determination is based on a pattern of the retrieved record and a representation status of the pattern in the smaller dataset.