Deterministic Cloud Computing for Batch-One DNN Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Network (DNN) inference workloads in cloud computing environments face challenges in meeting both quality of service (QoS) and quality of experience (QoE) requirements due to bursty workloads and high computational overhead, especially on resource-limited devices like smartphones and wearables, where batching strategies often fail to meet sub-second latency commitments and user expectations.
Innovation Solution
A deterministic streaming system with Tensor Streaming Processors (TSPs) that maintain a batch size of one, allowing precise scheduling and execution of tasks to meet stringent QoS requirements, and a scheduler that adjusts accuracy and quality metrics to align resources with workload demands, ensuring timely completion without queuing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batching strategy is used to improve throughput, then resource utilization improves, but latency increases and QoS commitments are violated
Solution Approach 1:
The system segments the batching process by maintaining a batch size of one while using functional slices to process different aspects of the same inference task in parallel. This allows the system to avoid the latency penalty of traditional batching while still achieving high throughput through fine-grained parallelism across multiple functional units within each processor.
Solution Approach 2:
The patent introduces a new dimension of parallelism by organizing processors into functional slices that operate simultaneously on different parts of the computation. This dimensional expansion allows the system to achieve both low latency (by avoiding queues) and high throughput (by utilizing multiple functional slices in parallel).
2Loss of energy
If batch size is increased to improve resource efficiency, then computational overhead decreases, but queue waiting time increases and QoS is violated
Solution Approach 1:
The system segments the computational workload across multiple functional slices, each handling a portion of the inference task. This segmentation allows the system to process work efficiently without accumulating large batches in queues, thereby reducing both computational overhead and waiting time simultaneously.
Solution Approach 2:
By maintaining a batch size of one and using functional slices to keep all functional units continuously engaged, the system eliminates idle time and queue waiting while maintaining high resource utilization. The continuous processing across functional slices ensures no loss of useful action.
3Reliability
If cloud environment is overprovisioned to meet QoE expectations during workload bursts, then QoE is maintained, but resource usage and energy consumption increase
Solution Approach 1:
The system dynamically adjusts resource allocation by using functional slices that can be activated or deactivated based on workload demands. During bursts, additional functional slices are engaged to maintain QoE, while during low-utilization periods, fewer slices are active, reducing energy consumption while maintaining reliability.
Solution Approach 2:
The patent changes the operational parameters of the processing system by allowing dynamic configuration of functional slice activation. This enables the system to adapt energy consumption levels to actual workload requirements while maintaining QoE fulfillment through deterministic performance guarantees.
Data Source
AI summary
A computing system can include a plurality of deterministic processors. Each deterministic processor can be configured to execute one or more assembled programs in a program order of the one or more assembled programs. The computing system can include a plurality of communication links providing communication between pairs of processors of the plurality of deterministic processors.