Neural Network Mini-Batch Pipeline Using Mixed Encoded and Raw Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face performance bottlenecks due to increased time spent on data preparation stages such as image loading and decoding, despite advancements in gradient calculation times.
Innovation Solution
A neural network system that includes a profile circuit to determine an optimal format ratio of encoded and raw data, a data control circuit to manage data buffers, and a learning control circuit to generate mini-batches, optimizing data distribution and reducing the impact of data preparation stages on training throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If hardware accelerators are used to reduce gradient calculation time, then gradient calculation speed is improved, but data preparation time increases and becomes a bottleneck
Solution Approach 1:
The patent applies preliminary action by pre-decoding images and storing them in raw format in the buffer memory before training operations begin. This decoupling of decoding from the training pipeline allows gradient calculation to proceed without waiting for real-time decoding, effectively performing the decoding action in advance to eliminate it as a bottleneck.
Solution Approach 2:
The patent segments the training pipeline into distinct stages: image loading, decoding, and gradient calculation. By separating these stages and allowing parallel execution where encoded images are loaded while decoded raw images are already available in buffer, the system eliminates sequential dependencies and balances the time distribution across pipeline stages.
2Quantity of substance
If all images are stored in encoded format, then storage efficiency is improved, but decoding time increases and bottlenecks training throughput
Solution Approach 1:
The patent applies local quality by maintaining different image formats in different locations: encoded images are stored in the storage device for space efficiency, while decoded raw images are maintained in the buffer memory for immediate processing. This spatial differentiation of data formats allows the system to optimize for both storage efficiency and processing speed simultaneously.
Solution Approach 2:
The buffer memory acts as an intermediary between the storage device containing encoded images and the processing unit performing gradient calculation. The buffer stores pre-decoded raw images, mediating the data flow and allowing the processing unit to access ready-to-use data without waiting for decoding operations during training.
3Loss of energy
If images are decoded in real-time during training, then memory usage is reduced, but processing time increases and reduces training efficiency
Solution Approach 1:
The system performs decoding as a preliminary action before training operations, storing decoded images in the buffer memory. This allows multiple images to be pre-decoded and staged in the buffer, so that during training the processing unit can continuously access decoded data without incurring real-time decoding delays, effectively trading upfront decoding time for sustained processing efficiency.
Data Source
AI summary
A neural network system includes a storage configured to store a data set including a plurality of encoded data and a plurality of raw data; a profile circuit configured to determine a format ratio of the plurality of encoded data to the plurality of raw data; a data control circuit configured to generate a mini batch used for a neural network learning operation based on the data set stored in the storage; and a learning control circuit configured to provide a request for generating the mini batch to the data control circuit while controlling the neural network learning operation.