Tile Scheduler Cost Matrix for Neural Network Resource Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI hardware systems face inefficiencies in resource allocation and data management during neural network operations, leading to suboptimal performance and utilization of processing cores.
Innovation Solution
A computing device with a tile scheduler that updates a cost matrix for each processing core based on meta information of allocated tiles, optimizing the allocation of tiles across multiple processing cores to minimize data transfer and maximize core utilization through dynamic scheduling and concurrent model execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If static resource allocation is used for processing cores, then device complexity is reduced, but system utilization and performance deteriorate due to idle cores and suboptimal resource distribution
Solution Approach 1:
The patent implements dynamic resource allocation by continuously updating the cost matrix based on current system state and reassigning tiles to processing cores accordingly. The tile scheduler dynamically adjusts allocations in response to changing workloads, transforming the static allocation system into a dynamic one that adapts to real-time conditions, thereby improving system utilization without requiring fundamental architectural changes
Solution Approach 2:
The system employs feedback mechanisms by monitoring the state of processing cores and tile queues, then using this information to update the cost matrix and adjust resource allocations. The tile scheduler receives feedback about core availability and workload status, processes this information through cost matrix updates, and generates optimized allocation decisions, creating a closed-loop control system that continuously improves resource distribution
2Loss of energy
If dynamic tile allocation is implemented, then data transfer efficiency is improved, but computational overhead and processing time increase due to continuous cost matrix updates
Solution Approach 1:
The patent applies partial action by updating only the necessary portions of the cost matrix based on current system state changes rather than performing complete recalculations. The tile scheduler selectively processes allocation decisions based on actual needs, updating cost values only for affected processing cores and tiles, thereby reducing computational overhead while maintaining data transfer efficiency
Solution Approach 2:
The system performs preliminary actions by pre-calculating and maintaining the cost matrix in readiness for allocation decisions. The cost matrix is continuously updated with anticipated cost values based on current system state, so that when allocation decisions are needed, the scheduler can make rapid decisions using pre-prepared cost information, reducing the time penalty of dynamic allocation
3Productivity
If multiple neural network models are executed concurrently, then processing throughput is improved, but resource contention and allocation complexity increase
Solution Approach 1:
The patent implements a universal tile scheduler that manages multiple neural network models concurrently using a single unified cost matrix framework. The scheduler handles diverse tile types (first tiles from one model, second tiles from another model) through the same allocation mechanism, allowing multiple models to share processing cores efficiently without requiring separate management systems, thereby managing complexity while maintaining high throughput
Solution Approach 2:
The system segments the resource allocation problem by separating first tiles from one neural network model and second tiles from another model, then managing their allocation independently through the unified scheduler. This segmentation allows the system to handle multiple models concurrently by treating their tile allocations as distinct but manageable units within the overall cost matrix framework, reducing contention while maintaining throughput
Data Source
AI summary
A computing device and method is disclosed. The computing device includes a plurality of processing cores, and a tile scheduler configured to update a cost matrix of each of the plurality of processing cores based on meta information of each of first tiles previously allocated to the plurality of processing cores and meta information of each of second tiles, and allocate the second tiles with respect to the plurality of processing cores using the updated cost matrix of each of the plurality of processing cores.