Distributed Model Approximation via Expectation Propagation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current libraries for processing large datasets, such as MLib for Apache Spark or H2O ML, are limited in the range of model types they support, leading to false results when used outside their calibrated range, and custom models require extensive and expensive re-training, which is not easily parallelizable in distributed computing.
Innovation Solution
A distributed computing framework that receives a data model and dataset, determines if it lacks a hierarchical structure, performs expectation propagation to approximate it with a hierarchical structure, divides the dataset into channels and microbatches, processes through parallel iterators and SIMD layers, and asynchronously merges results to update model parameters, allowing for flexible processing of arbitrary models at scale.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If custom models are developed for particular use cases, then model accuracy and adaptability are improved, but processing time and processor power requirements increase significantly
Solution Approach 1:
The patent segments the dataset into multiple partitions that can be processed independently and in parallel across distributed computing nodes. Each partition is handled separately through the iterative fitting process, allowing simultaneous processing of multiple data subsets without requiring sequential re-training, thus reducing total processing time while maintaining model accuracy.
Solution Approach 2:
The patent performs preliminary actions by pre-processing the dataset into ready-to-use partitions and establishing the hierarchical model structure before actual model fitting begins. This preparation work is done once and enables subsequent parallel processing operations to proceed efficiently without repeated setup overhead.
2Adaptability or versatility
If custom models are developed for particular use cases, then model adaptability is improved, but processor power requirements increase significantly
Solution Approach 1:
The patent divides the computational workload into segments that can be distributed across multiple processing nodes. Each node processes a specific data partition independently, allowing the system to scale processor power horizontally by adding more nodes rather than requiring a single powerful processor, thus reducing the processor power burden on any single unit while maintaining overall model adaptability.
Solution Approach 2:
The patent creates a universal framework that can handle various custom models through a common iterative fitting process. The hierarchical model structure and parallel processing architecture provide multi-functionality, allowing the same system to adapt to different model types and use cases without requiring specialized hardware for each model, thereby improving processor efficiency and versatility.
3Measurement precision
If extensive re-training is performed for custom models, then model precision is improved, but the process becomes expensive and not easily parallelizable
Solution Approach 1:
The patent segments both the data and the processing operations to enable parallelization. The dataset is divided into partitions, and the model fitting process is structured so that each partition can be processed independently and concurrently. This segmentation transforms the previously sequential re-training process into a parallel operation, reducing both time and complexity while maintaining model precision.
Solution Approach 2:
The patent introduces dynamics by implementing an iterative fitting process that adapts and refines model parameters across multiple passes through the data. This dynamic approach allows the model to progressively improve precision through repeated exposure to the data in parallel batches, rather than requiring a single complex sequential training run, making the process more adaptable and easier to parallelize.
Data Source
AI summary
A system including: at least one processor; and at least one memory having stored thereon computer program code that, when executed by the at least one processor, controls the system to: receive a data model identification and a dataset; in response to determining that the data model does not contain a hierarchical structure, perform expectation propagation on the dataset to approximate the data model with a hierarchical structure; divide the dataset into a plurality of channels; for each of the plurality of channels: divide the data into a plurality of microbatches; process each microbatch of the plurality of microbatches through parallel iterators; and process the output of the parallel iterators through single-instruction multiple-data (SIMD) layers; and asynchronously merge results of the SIMD layers.