Split AI Inference Across User Devices and Servers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional AI inference-driven user-facing applications face scalability and responsiveness issues due to computational and network bandwidth overload, leading to a suboptimal user experience.
Innovation Solution
A distributed AI system that dynamically balances computation load between a server and user devices by splitting tasks and models based on device type, available resources, and network status, using a split system to execute AI inferencing across multiple nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If all computation is performed at a single computing system (central server), then the system structure is simple and centralized, but the system has limited scalability and becomes slow due to computational overload and network bandwidth overload
Solution Approach 1:
The patent divides the centralized computing system into multiple distributed computing nodes (user devices and edge servers). The AI model is split into multiple sub-models that can be executed independently across different nodes. This segmentation allows the system to handle more requests simultaneously, improving scalability and responsiveness while distributing the computational load across multiple devices rather than relying on a single overloaded server.
2Quantity of substance
If computation load is increased to handle more user requests, then the system can serve more users, but the system becomes slower due to computational overload and network bandwidth overload
Solution Approach 1:
The computing load is segmented and distributed across multiple user devices and edge servers. Each node executes a portion of the AI sub-models locally, reducing network bandwidth overhead and avoiding central server overload. This enables the system to handle more requests simultaneously without sacrificing response speed.
Solution Approach 2:
The patent introduces a new dimension of computation by enabling AI inferencing to run not just on the central server but also on user devices and edge servers. This multi-dimensional computing architecture allows parallel processing across different locations, increasing throughput while maintaining response speed.
3Device complexity
If AI inferencing is performed entirely on the server, then resource utilization is simplified, but the system lacks scalability and becomes slow during high volume requests
Solution Approach 1:
The AI model is segmented into multiple sub-models that can be distributed and executed across different computing nodes. This segmentation enables scalable resource utilization where each node manages its own sub-model execution independently, allowing the system to grow by adding more nodes without proportionally increasing central server complexity.
Solution Approach 2:
The system enables multiple types of devices (user devices with different capabilities and edge servers) to all participate in AI inferencing. Each device type can execute appropriate sub-models based on its resources, creating a universal computing platform that improves scalability without requiring complex device-specific resource management.
4Productivity
If more computing resources are allocated to handle high volume requests, then the system can process more requests, but the cost and resource consumption increase
Solution Approach 1:
User devices execute AI sub-models locally using their own computing resources, eliminating the need to transmit all data and computation to remote servers. This self-service approach reduces network bandwidth consumption and distributes computational energy usage across many devices, improving request processing capacity without proportionally increasing centralized resource consumption.
Data Source
AI summary
A system to dynamically balance load between a server and a user device is disclosed. The system may include a system transceiver and a system processor. The system transceiver may be configured to obtain a request to execute a task from a user device. The system processor may obtain the request from the system transceiver and determine a machine learning (ML) model required to be implemented to execute the task. The system processor may determine a user device type, and determine a first ML sub-model, associated with the ML model, to be executed on the user device, and a second ML sub-model, associated with the ML model, to be executed on a server, based on the user device type. The system processor may cause the user device to execute the first ML sub-model and the server to execute the second ML sub-model to execute the task.