Client Device Partial Response Generation for Cloud Service Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing cloud-based services for processing user input on client devices introduce significant latency, leading to a less natural user experience due to the delay between user input and response, as the client device must wait for a complete response from a cloud service before presenting it to the user.
Innovation Solution
Implementing streamlined functionality on the client device to generate and provide an initial response while the server application generates the primary response, using a local Deep Neural Network (DNN) to determine a directional marker or topic, allowing for immediate playback of a first audio clip that is later appended with the primary response, thereby minimizing perceived latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the client device waits for a complete response from the cloud service before presenting it to the user, then the response accuracy is improved, but the latency increases
Solution Approach 1:
The system performs preliminary actions by generating and presenting an initial response to the user before the cloud service completes processing. The client device presents this initial response immediately, and then updates it with the primary response from the cloud service, thereby reducing perceived latency while maintaining response accuracy through the subsequent update.
2Adaptability or versatility
If the client device uses cloud-based services to process user input, then the processing capability is improved, but the latency between user input and response increases
Solution Approach 1:
The response generation process is segmented into two parts: an initial response generated and presented by the client device, and a primary response generated by the cloud service. This segmentation allows the client device to provide immediate feedback to the user while the cloud service processes the complete request, thereby reducing perceived latency while maintaining processing capability.
3Loss of time
If the client device immediately plays back a response without waiting for cloud service completion, then the latency is reduced, but the response completeness deteriorates
Solution Approach 1:
The system maintains continuity of useful action by continuously updating the initial response with the primary response from the cloud service. The client device presents the initial response immediately and then updates it with additional information from the cloud service, ensuring that the response remains complete and accurate throughout the interaction.
Data Source
AI summary
Systems and methods for distributing cloud-based language processing services to partially execute in a local device to reduce latency perceived by the user. For example, a local device may receive a request via audio input, that requires a cloud-based service to process the request and generate a response. A partial response may be generated locally and played back while a more complete response is generated remotely.