Uniform Sampling With Bounded Footprint Using Partition Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing sampling methods fail to create uniform, compact, and flexibly mergeable samples with a bounded footprint, making them unsuitable for scalable and flexible data warehousing scenarios.
Innovation Solution
A sampling system that partitions data sets into disjoint subsets, samples them independently and in parallel to produce uniform random samples, and merges them pairwise to create compact and uniform samples, using hybrid Bernoulli and reservoir sampling schemes to manage memory and maintain statistical uniformity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If full-scale data warehouse is sampled as a whole by a single sampler, then sampling can be implemented, but the system lacks scalability and flexibility for partial data sampling
Solution Approach 1:
The patent divides the data warehouse into multiple partitions, with each partition independently sampleable. This segmentation allows flexible sampling of specific partitions without requiring a complete warehouse sample, directly resolving the contradiction between sampling flexibility and system complexity.
Solution Approach 2:
The sampling system is designed to handle both full-warehouse sampling and partial partition sampling through a unified architecture. The same sampling mechanisms work at different scales, providing adaptability without proportionally increasing complexity.
2Ease of operation
If Bernoulli sampling is used, then sampling is simple and computationally inexpensive, but the sample size is random and cannot be controlled
Solution Approach 1:
The system dynamically adjusts the sampling rate for each partition based on the desired total sample size and partition data volume. This dynamic control maintains Bernoulli sampling simplicity while achieving precise sample size control through adaptive parameter adjustment.
Solution Approach 2:
The sampling system uses feedback from partition sizes and desired sample sizes to calculate appropriate sampling rates for each partition. This feedback mechanism ensures the total sample size matches targets while maintaining the simplicity of Bernoulli sampling methodology.
3Quantity of substance
If reservoir sampling is used, then the sample footprint is bounded a priori, but the algorithm cannot merge reservoir samples
Solution Approach 1:
The patent implements a merging mechanism that combines samples from multiple partitions into a unified sample. The merging process handles both reservoir samples and Bernoulli samples, maintaining footprint bounds while enabling versatile sample combination across partitions.
Solution Approach 2:
The system uses an intermediary merging process that translates between different sampling formats (reservoir and Bernoulli) to enable combination. This intermediary mechanism resolves the incompatibility between reservoir sampling's footprint control and merging capability.
4Volume of stationary object
If concise sampling is used, then the sample has compact representation, but the samples produced are not uniform
Solution Approach 1:
The system applies different sampling strategies to different partitions based on local characteristics. Some partitions use concise sampling where appropriate, while others use uniform sampling methods, with the merging process ensuring overall sample uniformity while maintaining compact representation where possible.
Data Source
AI summary
A sampling infrastructure/scheme that supports flexible, efficient, scalable and uniform sampling is disclosed. A sample is maintained in a compact histogram form while the sample footprint stays below a specified upper bound. If, at any point, the sample footprint exceeds the upper bound, then the compact representation is abandoned, the sample purged to obtain a subsample. The histogram of the purged subsample is expanded to a bag of values while sampling remaining data values of the partitioned subset. The expanded purged subsample is converted to a histogram and uniform random samples are yielded. The sampling scheme retains the bounded footprint property and to a partial degree the compact representation of the Concise Sampling scheme, while ensuring statistical uniformity. Samples from at least two partitioned subsets are merged on demand to yield uniform merged samples of combined partitions wherein the merged samples also maintain the histogram representation and bounded footprint property.