Neural Network Operating System for Distributed Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in executing large-scale neural networks on a single Neural Processing Unit (NPU) due to size constraints, leading to difficulties in handling diverse and complex hardware/software environments for AI application services, where current methods lack dynamic mapping of neural network segments to multiple NPUs, limiting performance improvement.
Innovation Solution
A neural network operating system that segments large-scale neural networks into sub-networks, abstracts these into tasks, and dynamically maps them across multiple NPUs using a broker and task processors, enabling efficient inference and result distribution across multiple neural processing units.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a large-scale neural network is executed on a single NPU, then the system structure is simple, but the NPU cannot handle the network due to size constraints
Solution Approach 1:
The large-scale neural network is segmented into multiple sub-networks that can be distributed across multiple NPUs. The system divides the computational workload by partitioning the neural network model into segments, each executable on a separate NPU, thereby enabling execution of networks that exceed single-NPU capacity constraints
Solution Approach 2:
The system implements a nested software architecture where the neural network operating system contains multiple task processors, each capable of handling sub-networks. This nested structure allows the system to manage complex distributed execution while maintaining a unified interface for applications, effectively nesting multiple levels of abstraction to resolve the complexity contradiction
2Productivity
If neural network segments are executed on multiple NPUs in a distributed manner, then performance is improved, but the hardware/software environment becomes complex and diverse
Solution Approach 1:
The neural network operating system implements universal task processors that can handle various types of neural network sub-networks through a standardized interface. This multi-functionality allows different NPUs with varying capabilities to execute the same task types, reducing the impact of hardware diversity while maintaining distributed performance benefits
Solution Approach 2:
The broker component acts as an intermediary between the neural network operating system and task processors, managing task distribution and coordination. This intermediary layer abstracts the complexity of distributed execution, handling task allocation, result aggregation, and error management, thereby isolating application complexity from the underlying distributed hardware/software environment
3Adaptability or versatility
If system software is run only in a specific NPU, then the software environment is simplified, but large-scale neural network inference becomes difficult in various hardware environments
Solution Approach 1:
The system dynamically allocates task processors to NPUs based on the specific hardware configuration and workload requirements. Rather than being statically bound to specific NPUs, the software architecture adapts its execution plan according to the available hardware resources, enabling compatibility across diverse NPU configurations while managing the complexity through runtime decision-making
Data Source
AI summary
Disclosed herein are an apparatus and method for distributed processing of a neural network. The apparatus may include a neural network model compiler for segmenting a neural network into a predetermined number of sub-neural networks, two or more neural processing units, and a neural network operating system for abstracting the sub-neural networks into a predetermined number of tasks, performing inference using the multiple neural processing units in a distributed manner in response to a neural network inference request from at least one application, and returning an inference result to the application.