AI Processor Scheduling via On-Chip Memory Pre-Fetching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Specialized AI processors for performing linear algebra matrix operations in artificial neural networks are often underutilized due to memory limitations, data dependencies, and other tasks that reduce throughput.
Innovation Solution
The development of new scheduling methods to optimize the computational efficiency of specialized AI processors, including the use of 'layer-first' and 'sample-first' scheduling systems, and dynamic scheduling techniques to manage memory and data dependencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If specialized AI processors are used for linear algebra matrix operations, then computational speed is improved, but processor utilization is reduced due to memory limitations and data dependencies
Solution Approach 1:
The patent applies preliminary action by pre-fetching data from external memory into on-chip memory before the computational core needs it. The memory management unit anticipates upcoming data requirements and loads data in advance, eliminating idle time for the computational core and improving processor utilization while maintaining high computational speed.
Solution Approach 2:
The patent introduces an intermediary on-chip memory structure that mediates between the slow external memory and the fast computational core. This intermediate buffer resolves the speed mismatch by holding frequently accessed data close to the computational units, ensuring continuous operation without idle cycles due to memory access delays.
2Productivity
If on-chip memory is increased to reduce memory access delays, then computational efficiency is improved, but chip area is increased
Solution Approach 1:
The patent applies local quality by providing different memory capacities to different computational cores based on their specific requirements. Rather than uniformly increasing memory for all cores, each core receives only the memory capacity it needs for its workload, optimizing computational efficiency while minimizing overall chip area usage.
Solution Approach 2:
The patent segments the memory system into multiple smaller on-chip memory units distributed across different computational cores. This segmentation allows memory to be allocated locally where needed, reducing the need for large centralized memory and minimizing total chip area while maintaining high computational efficiency through localized data access.
3Speed
If data is kept in on-chip memory for faster access, then computational speed is improved, but memory capacity is consumed
Solution Approach 1:
The patent implements dynamic memory management where data in on-chip memory is continuously allocated and deallocated based on real-time computational needs. The memory management unit dynamically adjusts which data resides in on-chip memory versus external memory, ensuring that high-speed access is provided only to currently active data while freeing capacity for future needs, thus balancing speed and capacity requirements.
Data Source
AI summary
Artificial intelligence is an increasingly important sector of the computer industry. However, artificial intelligence is an extremely computationally intensive field such that performing artificial intelligence calculations can be expensive, time consuming, and energy consuming. Fortunately, many of the calculations required for artificial intelligence applications can be performed in parallel such that specialized linear algebra matrix processors can greatly increase computational performance. But even with linear algebra matrix processors; performance can be limited due to complex data dependencies. Without proper coordination, linear algebra matrix processors may end up idle or spending large amounts of time moving data around. Thus, this document discloses methods for efficiently scheduling linear algebra matrix processors.