AI Accelerator Cache Banking for Parallel Deep-Learning Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep-learning accelerators face challenges in maximizing parallel performance with limited memory capacity and efficient access to different memory areas for multiple processor cores.
Innovation Solution
An AI accelerator with a cache memory structure that includes L0 instruction and L1 caches, configured with multiple cache banks and flexible memory mapping, connected via instruction, metadata, private, and global buses, to support efficient parallel operations for multiple processors, including GEMM operators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If multiple processor cores are integrated in a chip to improve parallel operation extent, then the extent of parallel operation is improved, but memory capacity becomes limited and operation efficiency deteriorates
Solution Approach 1:
The cache memory is divided into multiple cache banks (e.g., first cache bank, second cache bank, third cache bank) that can be independently accessed by different processor cores. This segmentation allows parallel access to different cache regions, enabling multiple processor cores to operate simultaneously without contending for the same memory resources, thus maintaining high parallel operation extent while efficiently utilizing limited memory capacity.
Solution Approach 2:
The patent introduces a multi-dimensional cache addressing scheme where cache addresses are divided into bank select signals and offset signals. This dimensional separation allows the system to address different cache banks along one dimension and different data elements within banks along another dimension, effectively expanding the usable memory space and improving operation efficiency without requiring additional physical memory capacity.
2Productivity
If cache memory size is increased to improve operation efficiency, then operation efficiency is improved, but device complexity and area increase
Solution Approach 1:
The cache memory structure is designed to serve multiple functions: it provides L1 cache functionality for fast data access, L2 cache functionality through shared cache banks, and supports both private and shared memory access modes. The same cache banks can be dynamically allocated to different processor cores based on access patterns, making the cache memory structure universally applicable to various access scenarios without requiring separate dedicated caches for each function.
Solution Approach 2:
The patent merges multiple cache levels (L1 and L2) and multiple memory types (private and shared memory) into a unified cache memory structure. By combining these functions into a single integrated system with shared cache banks, the patent reduces overall device complexity and area compared to having separate independent caches for each function, while maintaining high operation efficiency.
3Speed
If separate memory areas are created for different processor cores to improve access efficiency, then access efficiency is improved, but device complexity increases
Solution Approach 1:
The memory mapping structure is designed to be dynamic rather than static. The cache controller can dynamically determine which cache bank to access based on the current operation type (private memory access, local memory access, or global memory access) and the address translation results. This dynamic adaptation allows efficient access to different memory areas without requiring complex permanent separate pathways for each access type.
Solution Approach 2:
The patent introduces an address translation mechanism that acts as an intermediary between processor core addresses and cache memory addresses. The address translation unit translates virtual addresses to physical addresses and determines the appropriate cache bank, simplifying the memory mapping structure by providing a systematic intermediate step rather than requiring complex direct mapping logic for each access scenario.
Data Source
AI summary
Disclosed herein is an AI accelerator. The AI accelerator includes processors, each performing a deep-learning operation using multiple threads; and a cache memory including an L0 instruction cache for providing instructions to the processors and an L1 cache mapped to the multiple areas of mapped memory.