Parallel Bootstrap Sampling via Assignment Map in Data Warehouses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for creating bootstrap samples in data warehousing require multiple scans and resource-intensive distribution, making them inefficient for large datasets.
Innovation Solution
A method that generates a series of random integers across process nodes to create an assignment map, allowing for a single scan of the dataset and reducing the need for sample distribution, enabling parallel processing of bootstrap samples.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple scans and resource-intensive distribution methods are used to create bootstrap samples, then sample distribution completeness is improved, but processing time and resource consumption increase
Solution Approach 1:
The patent generates a complete assignment map before the actual bootstrap sampling process. This preliminary action creates a deterministic mapping between rows and bootstrap samples that eliminates the need for multiple scans and complex distribution operations during execution, thereby reducing processing time while maintaining distribution completeness
Solution Approach 2:
The patent creates a virtual copy of the dataset through the assignment map structure, where each row is assigned to multiple bootstrap samples via the map. This allows bootstrap samples to be generated through simple lookups and joins rather than physical data copying and distribution, significantly reducing resource consumption
2Measurement precision
If multiple scans and distribution operations are performed, then bootstrap sample accuracy is improved, but computational resources consumed increase
Solution Approach 1:
The patent replaces the mechanical process of physically distributing data across multiple scans with a logical substitution using the assignment map. The map stores pre-computed assignments that can be retrieved through efficient database joins, substituting heavy I/O and distribution operations with lighter computational lookups
Solution Approach 2:
The patent changes the approach from iterative sampling with replacement during multiple scans to a single-pass assignment using pre-generated random integers. By generating all random assignments upfront and storing them in the map, the system transforms a resource-intensive iterative process into an efficient single-scan operation
3Measurement precision
If traditional bootstrap sampling methods are used, then statistical accuracy is maintained, but processing efficiency decreases
Solution Approach 1:
The patent performs preliminary generation of random integers and creation of the assignment map before the actual bootstrap sampling. This preliminary action ensures that the statistical properties of bootstrap sampling are maintained while enabling efficient parallel processing during the actual sampling phase
Solution Approach 2:
The patent segments the bootstrap sampling process into two independent phases: (1) pre-computation of the assignment map containing all random assignments, and (2) execution phase where multiple bootstrap samples can be generated in parallel by different process nodes using the pre-computed map. This segmentation enables parallelization without compromising statistical accuracy
Data Source
AI summary
A method of bootstrap sampling a dataset is described. With a process node, a series of random integers is generated. An assignment map is created. The assignment map includes a row identifier for each row of data of the dataset. A plurality of bootstrap sample identifiers defined by the series are assigned to at least one row identifier. An output table created from the assignment map. Rows of the output table include each instance of the bootstrap sample identifiers, the row identifier assigned with the bootstrap sample identifier, and data of the row.