Batch Feature Data Reading to Prevent Memory Overflow
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The degradation of data reading performance in storage systems due to data overflow when large volumes of training data are loaded into memory for merging, leading to inefficient data management in machine learning applications.
Innovation Solution
A data processing method that involves acquiring a data reading task, determining target feature data from a storage system based on metadata, reading the data based on operation semantics, and merging it in batches until completion, avoiding the need to load all data into memory at once, thus preventing overflow and performance degradation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all training data is loaded into memory for merging, then data reading performance is improved, but data overflow into disk occurs leading to performance degradation
Solution Approach 1:
The patent segments the training data into multiple batches that are processed sequentially. Instead of loading all data at once, the system divides the large dataset into smaller manageable batches, processes each batch through the merging operation, and releases memory. This segmentation approach maintains high data reading performance while preventing memory overflow and disk spilling.
2Measurement precision
If large volumes of training data are processed, then machine learning model accuracy is improved, but data management efficiency deteriorates
Solution Approach 1:
The patent implements preliminary actions by pre-processing and organizing training data into structured formats before the main machine learning training process. Data is pre-loaded, validated, and organized into batches in advance, which streamlines the subsequent training process. This preliminary organization maintains high model accuracy while significantly improving data management efficiency during training.
3Reliability
If data is merged in memory, then data processing completeness is improved, but memory resource consumption increases
Solution Approach 1:
The patent implements periodic action by processing data in repeated cycles of batch loading, merging, and releasing. Each cycle processes a subset of data completely through the merging operation, ensures data processing completeness for that batch, then releases memory resources before loading the next batch. This periodic approach maintains processing completeness while controlling memory resource consumption within acceptable limits.
Data Source
AI summary
A data processing method, apparatus, device, and storage medium are provided. The method includes acquiring a data reading task; determining the target feature data from a data file stored in a target storage system based on the data reading task and metadata corresponding to the target storage system; reading the target feature data from the data file based on operation semantic information corresponding to the target feature data, and determining the read target feature data as the first target feature data; and merging the first target feature data to acquire target merged feature data in the case that the size of data volume of the read first target feature data meets a preset condition, and repeating the process of reading and merging the first target feature data until a completion of reading of the target feature data in the data file.