Parallel Regression Model Computation via Map-Reduce
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing all possible regression models becomes impractical with a large number of predictors due to exponential growth in computing time, as existing methods either rely on exhaustive searches or sequential strategies, which are inefficient.
Innovation Solution
The method employs a map-reduce framework to decompose the global search sequence into local search sequences, using multiple nodes (mappers) to compute local regression models in parallel and a single node (reducer) to combine results, applying sweep operations to efficiently compute all possible regression models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If exhaustive search is used to compute all possible regression models, then completeness of model selection is improved, but computing time grows exponentially with the number of predictors
Solution Approach 1:
The patent divides the exhaustive search of all 2^k regression models into multiple independent segments that can be processed in parallel. Each mapper processes a subset of regression models independently, and the reducer combines results. This segmentation allows the computing task to be distributed across multiple nodes, reducing the time each node needs to spend while maintaining complete model selection across the distributed system.
Solution Approach 2:
The patent transitions from sequential processing (one-dimensional time) to parallel distributed processing (adding spatial dimension across multiple nodes). By distributing the regression model computation across mappers and reducers in a map-reduce framework, the system adds a spatial dimension to the computation, allowing simultaneous execution of multiple regression model calculations that would otherwise be sequential.
2Device complexity
If sequential strategies are used to move from one regression model to another, then computing resources are reduced, but productivity and speed of model evaluation decrease
Solution Approach 1:
The patent merges multiple independent regression model computations into a single distributed parallel execution framework. Instead of running sequential strategies on separate systems or sequentially on one system, the patent combines multiple computing nodes into a coordinated map-reduce system where mappers and reducers work together to evaluate multiple regression models simultaneously, achieving both resource efficiency and high productivity.
3Productivity
If parallel computing strategies are used to distribute intensive computation, then productivity is improved, but device complexity and system architecture become more complex
Solution Approach 1:
The patent employs a universal map-reduce framework that can handle multiple regression model computations through a single standardized architecture. The mappers and reducers are designed with multi-functionality to process different subsets of regression models using the same computational patterns, reducing the need for specialized code for each parallel task and simplifying the overall system architecture despite the parallel computing complexity.
Data Source
AI summary
Provided are techniques for computing a task result. A processing data set of records is created, wherein each of the records contains data specific to a sub-task from a set of actual sub-tasks and contains a reference to data shared by the set of actual sub-tasks, and wherein a number of the records is equivalent to a number of the actual sub-tasks in the set of actual sub-tasks. With each mapper in a set of mappers, one of the records of the processing data set is received and an assigned sub-task is executed using the received one of the records to generate output. With a single reducer, the output from each mapper in the set of mappers is reduced to determine a task result.