Compiler Tensor Placement to Reduce Tree-Memory Communication Costs

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing accelerator chips with a tree structure topology face challenges in efficiently determining the placement of tensor elements in multiple memories, leading to high communication costs due to non-uniform memory access.

Innovation Solution

A compiler is developed to generate machine code for accelerator chips with distributed memories, associating each tensor element with addresses based on stride and hierarchy divisions, optimizing memory allocation to reduce communication costs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If tensor elements are allocated to distributed memories in accelerator chips with tree structure topology, then memory capacity and parallel processing capability are improved, but communication cost increases due to non-uniform memory access

Engineering Contradiction:

Improvememory capacityVSAvoidcommunication cost

Core Design Contradiction:

Quantity of substanceVSLoss of energy

Solution Approach 1:

The patent segments the tensor into multiple sub-tensors and allocates them to different memories in the tree structure. Each memory stores a portion of the tensor elements, enabling parallel processing while the segmentation strategy optimizes access patterns to reduce communication overhead between memory nodes.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies local quality by assigning different allocation strategies to different regions of the tensor based on their access patterns. Elements that are frequently accessed together are placed in the same memory or nearby memories in the tree structure, optimizing local access efficiency and reducing long-distance communication costs.

Inventive Principle:

Principle #3Local quality

2Productivity

If tensor elements are distributed across multiple memories, then processing parallelism is improved, but access time increases due to non-uniform memory topology

Engineering Contradiction:

Improveprocessing parallelismVSAvoidaccess time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent performs preliminary action by pre-calculating and storing the allocation pattern of tensor elements to memories before actual processing begins. The allocation unit determines the optimal placement of each tensor element in advance, considering the tree structure topology, so that during execution, data can be accessed directly without dynamic computation overhead.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces a hierarchical dimension to the memory access model by utilizing the tree structure topology. Instead of treating all memories as a flat pool, the patent organizes access paths through hierarchical levels, allowing parallel access to multiple memories simultaneously while minimizing the depth of traversal required for data retrieval.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS12367025B2Compiler, generation method, chip, and execution method

Publication Date: 2025.07.22 PREFERRED NETWORKS INC

AI summary

A compiler, for generating machine code to be executed in a chip including a plurality of distributed memories connected by a tree structure topology, includes at least one memory and at least one processor. The at least one processor is configured to associate each element of a tensor to be processed with an address in the plurality of memories included in the chip, based on a stride and a number of divisions in a predetermined hierarchy of the tree structure with respect to the tensor to be processed.