Data Partition Resizing for Ensemble Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data partitioning methods in predictive modeling frameworks, such as MapReduce, often result in uneven partition sizes and non-random data distribution, leading to suboptimal model building and predictive accuracy issues.
Innovation Solution
A method to determine and adjust the base model partition size, using a predictive modeling framework to randomly assign input data records into optimized partitions, ensuring uniform and random distribution for improved model building and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing data partitioning methods are used in predictive modeling frameworks, then data can be processed in distributed systems, but partition sizes become uneven and data distribution becomes non-random
Solution Approach 1:
The patent changes the partitioning parameter from fixed-size blocks to random sampling-based partitions. By using random sampling to assign data records to partitions, the system achieves both uniform partition sizes and random data distribution, resolving the contradiction between processing capability and partition uniformity.
Solution Approach 2:
The patent replaces the mechanical deterministic partitioning mechanism (fixed block sizes) with a probabilistic random sampling mechanism. This substitution allows partitions to be uniformly sized while containing randomly distributed data, eliminating the non-random distribution problem inherent in traditional methods.
2Productivity
If existing data partitioning methods are used, then distributed processing can be implemented, but predictive accuracy deteriorates due to non-random data distribution
Solution Approach 1:
The patent changes the data assignment parameter from deterministic block allocation to random sampling probabilities. Each data record is assigned to partitions based on random sampling, ensuring that partitions contain representative random samples of the overall data distribution, which improves predictive accuracy while maintaining distributed processing efficiency.
Solution Approach 2:
The patent implements a feedback mechanism where the random sampling process ensures that each partition receives data representative of the overall data distribution. This feedback loop maintains data quality and predictive accuracy across distributed processing by continuously ensuring random and uniform data assignment.
3Ease of manufacture
If fixed block partitioning is used for simplicity, then implementation is easier, but model building quality deteriorates due to uneven partition sizes
Solution Approach 1:
The patent replaces the simple but flawed fixed block partitioning mechanism with a random sampling-based partitioning system. Although slightly more complex in concept, the implementation remains straightforward using standard random sampling techniques, while dramatically improving model building quality through uniform partition sizes and random data distribution.
Solution Approach 2:
The patent changes the partitioning approach from fixed deterministic blocks to flexible random sampling partitions. This parameter change maintains implementation simplicity through the use of standard random sampling algorithms while significantly improving model building quality by ensuring uniform and representative data distribution across partitions.
Data Source
AI summary
Techniques are described for revising data partition size for use in generating predictive models. In one example, a method includes determining an initial number of base model partitions of data from a plurality of data sources; determining an initial base model partition size based at least in part on the initial number of base model partitions; and evaluating the initial base model partition size at least in part with reference to at least one base model partition size reference. The method further includes determining a finalized number of base model partitions based at least in part on the initial base model partition size; determining a revised base model partition size; and generating revised base models based at least in part on the revised base model partition size, including using a predictive modeling framework to randomly assign input data records from the plurality of data sources into the base model partitions.