GPU Shared Memory Weight Data Access for Neural Network Calculation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data calculation methods in neural network models require frequent reading of weight data, leading to low calculation efficiency due to the need for each thread to read weight data for each input data point, resulting in time-consuming processes and reduced performance.
Innovation Solution
The method involves reading weight data of a data processing layer into a GroupShared variable and dividing input data based on the number of threads in the thread group, allowing threads to share weight data and reduce the number of times weight data needs to be read, thereby improving calculation efficiency by performing data calculations on grouped input and weight data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If each thread reads weight data for each input data point individually, then data access is simple and straightforward, but the number of reading operations increases significantly, reducing calculation efficiency
Solution Approach 1:
The patent merges weight data reading operations by having threads within the same thread group share common weight data. Instead of each thread reading weight data independently for each input data point, the weight data is read once into shared memory for the entire thread group, significantly reducing the number of reading operations and improving calculation efficiency
Solution Approach 2:
The patent performs preliminary action by pre-loading weight data into shared memory before the actual calculation begins. The weight data is read and stored in shared memory in advance, so that during the calculation phase, threads can access the pre-loaded weight data without performing additional reading operations, thereby reducing time loss
2Productivity
If weight data is read frequently for each input data point, then each thread has access to needed data, but the overall data processing speed decreases due to repeated reading operations
Solution Approach 1:
The patent combines the weight data reading operations into a single operation per thread group by using shared memory. All threads within a thread group access the same shared weight data, eliminating the need for repeated reading operations and simplifying the data access pattern while maintaining data availability for all threads
3Productivity
If input data is processed thread-by-thread independently, then implementation is straightforward, but the calculation time increases due to sequential processing
Solution Approach 1:
The patent segments the input data into groups that correspond to thread groups, where each thread group processes a specific portion of the input data simultaneously. This segmentation enables parallel processing while maintaining data organization, allowing multiple threads to work on different input data points concurrently rather than sequentially
Solution Approach 2:
The patent merges the processing capabilities of multiple threads within a thread group by enabling them to simultaneously access shared weight data and process their respective input data points in parallel. This combination of threads working together on shared resources significantly increases calculation throughput compared to independent sequential processing
Data Source
AI summary
A method and an apparatus for data calculation in a neural network model, and an image processing method and apparatus. The method for data calculation includes: reading weight data shared by a group of data processing of a data processing layer in a neural network model, into a GroupShared variable of a thread group of a graphics processing unit (GPU), dividing input data of the data processing layer based on the number of threads in the thread group, reading, for each group of input data, weight data corresponding to the group of input data for a data processing in the group of data processing from the GroupShared variable, and performing, by each thread in the thread group, the data processing by using a group of read input data and weight data corresponding to the group of input data, to obtain a calculation result corresponding to the group of input data.