Column-Distributed Tree-Based Data Mining in Cloud Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud computing systems face challenges in achieving time- and resource-efficient distributed data mining due to the difficulty in managing large volumes of column-distributed data for training and inference using machine learning models, particularly with tree-based models, which require excessive communication and lack support for parallel processing.
Innovation Solution
A distributed, tree-based data mining system that transmits training tasks from a resource manager node to worker nodes with local data stores, merges local parameters to determine global splits, and updates the ML model, enabling parallel training and inference by processing data closer to its sources and minimizing data movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed data mining is performed on column-distributed data using traditional tree-based models, then data mining can be performed on large datasets, but excessive communication overhead and lack of parallel processing efficiency occur
Solution Approach 1:
The patent segments the centralized tree-based data mining process into distributed operations across multiple worker nodes. Each worker node independently processes local data partitions and computes local optimal splits, which are then merged at the master node to determine global optimal splits. This segmentation enables parallel processing of data across the distributed cluster, significantly improving productivity while reducing communication overhead compared to centralized approaches.
2Reliability
If data is centralized for processing, then tree-based models can be trained effectively, but data movement and resource consumption increase
Solution Approach 1:
The patent implements a self-service distributed data mining approach where each worker node autonomously processes its local data partition using the column-distributed data already stored in its local memory. Worker nodes independently compute local optimal splits without requiring data to be moved or centralized. This self-service mechanism maintains model training effectiveness while eliminating the energy consumption associated with data movement across the network.
3Productivity
If traditional distributed learning methods are used, then data can be processed in parallel, but support for column-distributed data structures is lacking
Solution Approach 1:
The patent applies local quality by designing the distributed data mining algorithm to work specifically with column-distributed data structures. Each worker node receives and processes data in columnar format from its local data store, computing local optimal splits that respect the columnar organization. The master node then merges these local splits to determine global optimal splits, maintaining compatibility with the column-distributed structure throughout the entire training process while enabling parallel processing.
Data Source
AI summary
Methods, systems, and computer-readable storage media for training of machine learning (ML) models and inference using the ML models based on distributed data mining in cloud systems, and more particularly for a distributed, tree-based data mining system that uses column-distributed tree-based data mining in a cloud system to support training of and inference using ML models.