Content-Adaptive Arrays With Dynamic Datatype Upcasting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing ML applications face challenges in efficiently processing arrays with varying data types due to the fixed datatype approach, leading to potential loss of information through underflow and increased hardware complexity, especially when transitioning between different precision levels.
Innovation Solution
Implementing a content adaptive array with metadata (type selector bits) to dynamically adjust datatypes based on data values, coupled with conversion circuitry to upcast or downcast data to compatible formats, reducing hardware complexity and preserving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a fixed datatype is used for the entire array, then hardware processing is simplified, but information loss occurs through underflow and accuracy decreases
Solution Approach 1:
The patent applies dynamics by making the datatype selection dynamic rather than fixed. Type selector bits are included in the array metadata to indicate the datatype of different portions of the array, allowing the system to adaptively choose appropriate precision levels (e.g., FP32, FP16, INT8) for different data ranges. This dynamic adaptation prevents underflow in low-precision regions while maintaining hardware efficiency through structured datatype management.
2Loss of information
If different precision levels are used for different data ranges, then information loss is minimized, but hardware complexity increases
Solution Approach 1:
The patent segments the array into different portions, each with its own datatype indication through type selector bits. This segmentation allows different precision levels to be applied to different data ranges without requiring a single complex conversion system. The hardware can process each segment according to its indicated datatype, reducing overall complexity compared to universal conversion circuitry.
Solution Approach 2:
The type selector bits act as an intermediary between the stored data and the processing hardware. These metadata bits carry information about the datatype without requiring physical conversion circuitry for every possible datatype combination. The intermediary provides a simple mechanism for the hardware to adapt to different precision requirements.
3Measurement precision
If high precision datatype is used throughout, then accuracy is maintained, but memory bandwidth and storage efficiency decrease
Solution Approach 1:
The patent applies local quality by allowing different precision levels (qualities) at different locations (portions) of the array. Type selector bits indicate which datatype applies to each portion, enabling high precision (e.g., FP32) where needed and low precision (e.g., INT8) where sufficient. This local adaptation maintains accuracy where required while optimizing memory bandwidth and storage efficiency in other regions.
4Quantity of substance
If low precision datatype is used throughout, then memory efficiency is improved, but underflow occurs and accuracy decreases
Solution Approach 1:
The patent applies local quality by allowing different precision levels (qualities) at different locations (portions) of the array. Type selector bits indicate which datatype applies to each portion, enabling high precision (e.g., FP32) where needed and low precision (e.g., INT8) where sufficient. This local adaptation maintains accuracy where required while optimizing memory bandwidth and storage efficiency in other regions.
Data Source
AI summary
Embodiments herein describe a content adaptive array that can include different types of data. A compute unit can include conversion circuitry (e.g., upcast circuitry) that can identify the datatype(s) in the content adaptive array and convert the data so it has a desired datatype. For example, if the content adaptive array has both FP and INT, the upcast circuitry converts the data into the same datatype (e.g., FP8). If the array includes FP4 and FP8 (or INT4 and INT8), the upcast circuitry converts the data into FP8. This means the circuitry in the compute unit that performs the data operation (e.g., matrix multiplication) does not have to support many different types of datatypes.