Adaptive Buffer Management for Dynamic Tensor Shapes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning frameworks face challenges in handling dynamic tensor shapes, leading to inefficiencies and precision loss when dealing with complex data in deep neural networks, particularly in object detection and natural language processing tasks, as existing solutions like padding, cropping, or just-in-time compilation either alter data or increase compilation workload.
Innovation Solution
An adaptive buffer management system using a shape buffer pool that caches compilation results for common tensor shapes and updates them using a least recently used algorithm, allowing for dynamic tensor shape representation and reducing the need for recompilation by using a buffer dialect and management pass to determine dynamic buffer needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If padding or cropping is used to handle dynamic tensor shapes, then the system can process variable-shaped data, but data precision is lost and information is altered
Solution Approach 1:
The patent introduces a dynamic buffer management system that adapts buffer allocation at runtime based on actual tensor shapes. The shape buffer pool dynamically creates and manages buffers for different tensor configurations without altering the input data, allowing the system to handle variable-shaped data while preserving original information integrity.
2Adaptability or versatility
If just-in-time compilation is used to support dynamic tensor shapes, then the system can compile code at runtime, but the compilation workload increases significantly
Solution Approach 1:
The patent pre-compiles and stores compilation results for common tensor shapes in a shape buffer pool before runtime execution. When a tensor shape matches a pre-computed shape in the pool, the system reuses the stored compilation result instead of performing new compilation, significantly reducing runtime compilation workload while maintaining support for dynamic tensor shapes.
Solution Approach 2:
The system creates a copy of compilation results for common tensor shapes and stores them in the shape buffer pool. This allows the system to reuse previously compiled code for identical tensor shapes without recompiling, reducing the compilation workload while maintaining adaptability to dynamic shapes.
3Device complexity
If a fixed buffer allocation strategy is used, then memory management is simplified, but the system cannot efficiently handle variable tensor shapes
Solution Approach 1:
The patent implements a dynamic buffer management system with a shape buffer pool that adapts to different tensor shapes at runtime. The system dynamically allocates and manages buffers based on actual tensor requirements, enabling efficient handling of variable shapes while maintaining manageable complexity through structured buffer pool organization.
Data Source
AI summary
The disclosure relates to adaptive buffer management to support a dynamic tensor shape in a DNN. An apparatus for the DNN may include processor circuitry configured to: determine whether a tensor shape of an input tensor of an object in the DNN is dynamic and exists in a shape buffer pool; run the object by use of a compilation result for the object stored in the shape buffer pool when the tensor shape of the input tensor is dynamic and exists in the shape buffer pool; and invoke the compilation procedure to perform JIT compilation for the object so as to get the compilation result for the object when the tensor shape of the input tensor is dynamic and does not exist in the shape buffer pool.