Accelerated ML Learning via Spatio-Temporal Data Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As data size grows, training machine learning models with highly granular data becomes computationally intensive and challenging, particularly in scaling and filtering out noisy portions of the data.
Innovation Solution
The method involves partitioning the independent variable of input data into partitions based on space/time measurements, creating data samples in each partition with reduced granularity, training machine learning models independently on each partition, comparing results, and merging data samples where necessary to achieve accelerated learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If highly granular data is used for training machine learning models, then model accuracy is improved, but computational time and computing resources increase
Solution Approach 1:
The input data is partitioned into multiple partitions based on the independent variable (spatio-temporal dimensions). Each partition is then subsampled independently to create multiple data samples with reduced granularity. This segmentation allows the system to process smaller data subsets in parallel, reducing overall computational time while maintaining model accuracy through statistical comparison of results across partitions.
2Measurement precision
If highly granular data is used for training machine learning models, then model accuracy is improved, but computing resources increase
Solution Approach 1:
The data is divided into multiple partitions that can be processed independently and in parallel. Each partition undergoes subsampling to create multiple data samples, reducing the amount of data that needs to be processed at any one time. This segmentation strategy reduces memory requirements and enables distributed computing, thereby reducing total computing resources needed.
Solution Approach 2:
Instead of processing all highly granular data, the system applies subsampling to create multiple data samples from each partition. By processing multiple reduced versions of the data and comparing results statistically, the system achieves acceptable model accuracy with significantly less computing resources than processing the complete granular dataset.
3Measurement precision
If highly granular data is used for training machine learning models, then model accuracy is improved, but data filtering complexity increases
Solution Approach 1:
The system partitions data based on the independent variable into manageable segments. Each partition is then subsampled independently, creating multiple data samples that can be processed separately. This segmentation simplifies the filtering process by breaking down the complex task of handling entire granular datasets into simpler, independent operations on smaller partitions.
Solution Approach 2:
The system performs subsampling and creates multiple data samples from each partition before training machine learning models. This preliminary action of reducing data granularity in advance simplifies subsequent processing steps and makes the overall system more manageable, while statistical comparison of results ensures accuracy is maintained.
Data Source
AI summary
A computer-implemented method, a computer program product, and a computer system for accelerated learning. A computer partitions an independent variable of input data into partitions. A computer creates data samples in each of the partitions, where each of the data samples has less granularity. For each of the partitions, a computer trains a machine learning model independently on each of the data samples and compares results of training on the data samples. For each of one or more partitions in which results of training on a predetermined number of the data samples are statistically identical at a predetermined confidence level, a computer outputs a result of training on one of the data samples. For each of one or more rest partitions in which no result of training has been outputted, a computer merges each pair of the data samples and uses merged data samples to train the machine learning model.