Database-Integrated Model Training for Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional back-end applications for machine learning require either frequent fetching of large data records, which is time-consuming, or a substantial memory capacity to store the data, leading to insufficient accuracy due to limited sample sizes that can fit in RAM.
Innovation Solution
Allocating memory space within the database for maintaining the data model during training, where data records are read directly from the database for each iteration, and the model is trained and updated inside the database, with convergence checks performed to save the model once it has stabilized.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data records are frequently fetched from database for model training iterations, then model training can be performed, but training time increases significantly
Solution Approach 1:
The patent applies preliminary action by pre-fetching data records from the database into memory space allocated within the database before model training begins. This preliminary data loading eliminates the need for repeated fetching during training iterations, significantly reducing training time while maintaining model accuracy.
2Measurement precision
If substantial memory capacity is allocated to store data records, then model training accuracy improves, but memory resource consumption increases
Solution Approach 1:
The patent merges the database storage function with the memory function by allocating memory space directly within the database system. This integration allows the database to serve dual purposes: storing data records and providing memory space for loading training data, thereby improving model accuracy without requiring separate substantial memory resources.
Solution Approach 2:
The database system is designed to perform multiple functions: it acts as both the persistent storage for data records and the working memory space for model training. This multi-functionality eliminates the need for separate memory allocation, allowing the system to handle large datasets for accurate training while optimizing resource utilization.
3Speed
If data records are loaded into external memory for training, then training speed improves, but system complexity increases
Solution Approach 1:
The patent combines the database management system and memory management into a single integrated architecture. By allocating memory space within the database, the system eliminates the need for separate external memory management layers, data transfer interfaces, and coordination mechanisms, thereby reducing system complexity while maintaining fast training speeds.
Data Source
AI summary
A method and arrangement for processing data when training a data model involving multiple iterations of data records in a dataset (400c, 606) stored in a database (400, 600). Memory space (400d, 608) is allocated in the database for maintaining the data model during the training operation. The data records in the dataset are read (4:3) directly from the database for each iteration, and the data model is trained (4:4) inside the database by repeatedly applying the read data records in the training operation and updating (4:5) the data model. It is also checked (4:6) whether the updated data model has converged according to a predefined convergence condition. The data model is eventually saved (4:7) once the data model has converged.