AI Accelerator Input Path Switching for Batch-Sequence Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial intelligence accelerators face challenges in optimizing calculations for batch and sequence sizes, leading to inefficient resource utilization and slowed processing speeds in applications like LSTM operations.
Innovation Solution
An accelerating device that determines calculation methods based on batch and sequence sizes, controlling input paths for data to a processing element array, allowing for parallel processing and improved efficiency through first and second calculation methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a fixed calculation method is used in existing AI accelerators, then the hardware structure can be simplified, but the resource utilization efficiency deteriorates when batch and sequence sizes vary
Solution Approach 1:
The patent implements dynamic selection between first and second calculation methods based on batch size and sequence size parameters. The accelerator dynamically switches between different calculation approaches (e.g., batch-major vs sequence-major processing) to adapt to varying workloads, thereby maintaining high resource utilization efficiency without requiring multiple dedicated hardware structures for each calculation method.
2Speed
If the accelerator processes data with fixed batch and sequence sizes, then processing speed can be optimized for specific cases, but adaptability to different data sizes deteriorates
Solution Approach 1:
The patent changes processing parameters (calculation method selection) based on input data characteristics (batch size and sequence size). By comparing these parameters against reference values, the system selects the appropriate calculation method to maintain optimal processing speed across different data sizes, thus achieving both speed optimization and adaptability.
3Device complexity
If existing accelerators use a single calculation method, then the control logic can be simplified, but calculation speed for varying batch and sequence sizes deteriorates
Solution Approach 1:
The control logic dynamically selects between first and second calculation methods based on real-time batch size and sequence size parameters. This dynamic adaptation allows the system to optimize calculation speed for varying data dimensions without requiring complex dedicated control logic for each possible data size configuration.
4Productivity
If the accelerator optimizes for large batch sizes, then parallel processing efficiency is improved, but performance on small batch sizes with long sequences deteriorates
Solution Approach 1:
The system changes the calculation method parameter based on the relationship between batch size and sequence size. When batch size is large, it uses the first calculation method optimized for parallel processing. When batch size is small and sequence size is long, it switches to the second calculation method, ensuring optimal performance across different data configurations without sacrificing parallel processing efficiency for large batches.
Data Source
AI summary
Accelerating devices, data storing devices, data processing systems and operating methods of accelerating devices are disclosed. In one aspect, an accelerating device includes an accelerator. The accelerator processes a calculation by using a calculation method selected based on at least one of a batch size and a sequence size and by controlling an input path of first input data and second input data to the processing element array according to the selected calculation method. The calculation method includes a first calculation method and a second calculation method, and the first input data and the second input data are input to the processing element array based on the batch size according to the first calculation method and the first input data being input to the processing element array based on the sequence size according to the second calculation method.