DNN Floating-Point Split Processing for Energy-Efficient CIM

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing DNN processors face challenges in achieving high performance and energy efficiency for floating-point operations due to limitations in supporting wide dynamic range and throughput, with existing CIM technologies either losing accuracy or having low throughput.

Innovation Solution

The apparatus classifies floating-point data into outlier and inlier data, using a CIM operator for fixed-point operations on inlier data and an NPU for floating-point operations on outlier data, with a separate transmission path for weights, and caches weights for outlier data to reduce read cycles.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Use of energy by moving object

If pre-alignment is performed to align mantissas according to exponent difference, then energy efficiency is improved, but accuracy is lost near the least significant bit

Engineering Contradiction:

Improveenergy efficiencyVSAvoidaccuracy

Core Design Contradiction:

Use of energy by moving objectVSMeasurement precision

Solution Approach 1:

The patent segments floating-point data into two categories: inlier data (where pre-alignment can be applied) and outlier data (where pre-alignment is skipped to preserve accuracy). This segmentation allows the system to apply energy-efficient pre-alignment only where safe, while maintaining accuracy for data where exponent differences are large.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies different processing quality to different data segments: full pre-alignment for inliers and no pre-alignment for outliers. This local differentiation optimizes energy efficiency for the majority of data while preserving accuracy for the minority of cases where it matters most.

Inventive Principle:

Principle #3Local quality

2Adaptability or versatility

If only single-cycle exponent operation is supported, then floating-point operation capability is achieved, but throughput is reduced

Engineering Contradiction:

Improvefloating-point operation capabilityVSAvoidthroughput

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent segments the processing pipeline into separate paths: one for inlier data handling and another for outlier data handling. This allows parallel processing of different data types, improving overall throughput while maintaining floating-point operation capability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary classification of data into inliers and outliers before processing. This preliminary action enables the system to prepare appropriate processing paths in advance, avoiding bottlenecks and improving throughput.

Inventive Principle:

Principle #10Preliminary action

3Use of energy by moving object

If fixed-point operation is used in CIM, then energy efficiency is improved, but floating-point representation capability is lost

Engineering Contradiction:

Improveenergy efficiencyVSAvoidfloating-point representation capability

Core Design Contradiction:

Use of energy by moving objectVSAdaptability or versatility

Solution Approach 1:

The patent segments floating-point operations into fixed-point operations (for inlier data) and floating-point operations (for outlier data). This allows the energy-efficient fixed-point CIM operations to handle the majority of data while maintaining full floating-point capability when needed.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent uses an intermediary classification mechanism that identifies which data can be processed as fixed-point and which requires floating-point processing. This intermediary layer enables the system to leverage both fixed-point efficiency and floating-point capability.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20250370710A1Apparatus for operating deep neural network for energy-efficient floating-point operation and method for floating-point operation using the same

Publication Date: 2025.12.04 KOREA ADVANCED INST OF SCI & TECH

AI summary

An apparatus for a DNN operation includes a preprocessor configured to classify outlier data and inlier data from a predetermined number of pieces of grouped and input floating-point data and to perform presorting on the inlier data, a CIM operator configured to perform a fixed-point operation on the inlier data, an NPU operator configured to receive the outlier data and corresponding input channel information from the preprocessor and to perform a floating-point operation on the outlier data, and an aggregation core configured to sum and output an operation result of each of the CIM operator and the NPU operator, wherein the NPU operator reads a weight for each input channel for the floating-point operation on the outlier data through a separate transmission line implemented in the CIM operator, and causes the outlier data to be processed in parallel with an operation cycle of the inlier data.