Mixed-Precision Tensor Processing with Hardware Precision Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware accelerators for machine learning models face inefficiencies in processing mixed-precision tensors due to software-based approaches that incur significant overhead costs and slowdowns from misalignment between tensor regions and shard boundaries, leading to decreased processing speed and accuracy.
Innovation Solution
A hardware accelerator with hardware-level support for mixed-precision tensor operations, utilizing a precision map to process tensor regions with different precisions in parallel, avoiding subdivision into serially processed sub-shards, and storing regions in non-interleaved memory to enhance processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If software-based approaches are used to process mixed-precision tensors, then flexibility in handling different precisions is improved, but processing speed and efficiency deteriorate due to significant overhead costs and slowdowns
Solution Approach 1:
The patent replaces software-based processing with hardware-level support for mixed-precision tensor operations. The hardware accelerator is designed with dedicated circuits that can natively execute operations on tensors with different precisions (e.g., FP16, BF16, INT8) without requiring software intervention, thereby eliminating the overhead and speed penalties associated with software-based approaches while maintaining precision flexibility.
Solution Approach 2:
The patent implements different precision formats for different regions of the tensor based on their specific requirements. The hardware accelerator is configured to process different tensor regions with their appropriate precisions simultaneously, allowing critical regions to use higher precision while less critical regions use lower precision, optimizing both accuracy and performance without uniform precision constraints.
2Ease of manufacture
If tensor regions are subdivided into serially processed sub-shards, then processing of mixed-precision tensors becomes manageable, but processing time increases due to sequential handling
Solution Approach 1:
The patent divides the mixed-precision tensor into multiple shards, where each shard contains a specific precision format. The hardware accelerator is designed to process multiple shards in parallel using different functional units optimized for specific precision types, eliminating the need for serial processing while maintaining manageable data handling through the shard structure.
Solution Approach 2:
The patent introduces a parallel processing dimension by implementing multiple processing pipelines within the hardware accelerator, each capable of handling different precision formats simultaneously. This transforms the processing from a single-threaded sequential approach to a multi-threaded parallel approach, dramatically reducing processing time while maintaining the shard-based organization for manageability.
3Speed
If conventional hardware accelerators process mixed-precision tensors, then hardware acceleration is utilized, but accuracy deteriorates due to misalignment between tensor regions and shard boundaries
Solution Approach 1:
The patent assigns different precision formats to different tensor regions based on their specific accuracy requirements. The hardware accelerator is configured with precision maps that track which regions require which precision levels, ensuring that each region is processed with the appropriate precision without being forced into uniform precision handling that causes accuracy loss at boundaries.
Solution Approach 2:
The patent introduces precision metadata and precision maps as intermediary structures that bridge the tensor data and the hardware processing units. These intermediaries carry precision information alongside the tensor data, allowing the hardware accelerator to correctly interpret and process each region with its intended precision without misalignment issues, maintaining both speed and accuracy.
Data Source
AI summary
A computing device including memory storing a mixed-precision tensor. The mixed-precision tensor includes one or more first tensor regions within which first tensor elements have a first precision and one or more second tensor regions within which second tensor elements have a second precision. The memory further stores a precision map indicating the first and second tensor regions. The computing device further includes a hardware accelerator configured to receive the precision map and the one or more first tensor regions, as indicated by the precision map, and perform a tensor processing operation on the one or more first tensor regions in the first precision. The hardware accelerator receives the one or more second tensor regions, as indicated by the precision map, and performs the tensor processing operation on the one or more second tensor regions in the second precision. The hardware accelerator stores a combined tensor processing output.