In-Warehouse ML Training With Heuristic Memory Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users lack expertise in machine learning techniques and face inefficiencies in training models within virtual warehouses due to memory management issues and network resource burdens, leading to failed or inefficient training processes.
Innovation Solution
A method for automated model selection, tuning, and training within virtual warehouses using a single SQL query, which estimates peak memory usage via a heuristic based on data set factors, allowing training within the warehouse without data transfer, thus reducing network resource usage and ensuring accurate memory allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data is transferred from virtual warehouse to local storage for ML training, then ML model training can be performed, but network resource usage increases and training efficiency decreases
Solution Approach 1:
The patent merges the ML training capability directly into the virtual warehouse environment by integrating ML training operators with SQL query execution. This allows data to remain in the virtual warehouse during training operations, eliminating the need for data transfer to local storage and thereby reducing network resource usage while maintaining full ML training functionality.
2Productivity
If memory is allocated for ML training in virtual warehouse, then training can proceed, but inaccurate memory estimation leads to either memory exhaustion or wasted resources
Solution Approach 1:
The patent performs preliminary memory usage estimation through heuristic analysis of the training data set characteristics (size, schema, data types) before actual ML training begins. This preliminary action allows the system to pre-allocate appropriate memory resources, preventing both memory exhaustion during training and wasteful over-allocation, thereby ensuring efficient training execution.
3Ease of manufacture
If users implement ML training manually with data transfer, then training can be performed, but process complexity and time increase
Solution Approach 1:
The patent creates a universal interface that allows users to perform ML training using standard SQL queries, which is the native language of the virtual warehouse. This multi-functional approach combines data querying and ML training into a single unified process, eliminating the need for separate data extraction, local processing, and model deployment steps, thereby significantly reducing setup time and process complexity.
Data Source
AI summary
Embodiments of the present disclosure provide techniques for classification with automated model selection, tuning, and training. A processing device receives, from a client, a data query referencing an input data set of a database associated with a virtual warehouse. The processing device allocates an amount of memory of the virtual warehouse to be used to train a machine learning (ML) model based on the input data set and a peak memory estimate, where the peak memory estimate is based on a heuristic. The processing device trains, based on the input data set and the data query, the ML model in the virtual warehouse using the amount of memory.