Personalized Speech Endpointing via Historical Interaction Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing automated assistants rely on audio-based endpointing measures to determine when a user has finished speaking, which can lead to increased latency or premature cutoffs of spoken utterances, affecting the accuracy and efficiency of human-to-computer interactions.

Innovation Solution

A personalized endpointing measure is generated based on a portion of the text representation of a spoken utterance immediately preceding a candidate endpoint and a user-specific measure derived from historical interactions, allowing for more accurate determination of when a user has finished speaking.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If a threshold amount of time is used to determine endpointing, then the system can identify when the user has stopped speaking, but the latency increases and the user may have to repeat utterances

Engineering Contradiction:

Improveendpointing accuracyVSAvoidlatency

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system changes the parameter from a fixed time threshold to a dynamic, personalized endpointing measure that is adjusted based on user-specific historical interaction data. This allows the endpointing decision to be made more quickly and accurately without relying on arbitrary time delays.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The system implements feedback by using historical interaction data to continuously refine and personalize the endpointing measure for each user. The endpointing model learns from past interactions to improve its accuracy over time, reducing both latency and incorrect endpointing.

Inventive Principle:

Principle #23Feedback

2Loss of time

If a shorter threshold is used for endpointing, then the latency is reduced, but the system risks cutting off part of the spoken utterance

Engineering Contradiction:

ImprovelatencyVSAvoidutterance completeness

Core Design Contradiction:

Loss of timeVSReliability

Solution Approach 1:

The system transitions from using a fixed time threshold parameter to a personalized endpointing measure that adapts to individual user speaking patterns. This personalized approach allows for more reliable determination of utterance completion without requiring excessive delay.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The system performs preliminary analysis by examining the text representation of the spoken utterance and comparing it against historical interaction patterns before finalizing the endpointing decision. This preliminary action helps ensure the utterance is complete before triggering the endpoint.

Inventive Principle:

Principle #10Preliminary action

3Productivity

If audio-based endpointing measures are used, then the system can detect when the user stops speaking, but the accuracy decreases due to pauses within utterances

Engineering Contradiction:

Improveresponse speedVSAvoidendpointing accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The system introduces an intermediary element - the text representation of the spoken utterance - to bridge the gap between audio detection and accurate endpointing determination. By analyzing the text content and comparing it with historical patterns, the system can more accurately identify true endpoints.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system changes from relying solely on audio-based temporal measures to incorporating semantic content analysis through text representation. This parameter change allows the system to distinguish between meaningful pauses and incomplete utterances more effectively.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS12322376B2Personalized speech query endpointing based on prior interaction(s) involving a specific phrase preceding an endpoint

Publication Date: 2025.06.03 GOOGLE LLC

AI summary

A personalized endpointing measure can be used to determine whether a user has finished speaking a spoken utterance. Various implementations include using the personalized endpointing measure to determine whether a candidate endpoint indicates a user has finished speaking the spoken utterance or whether the user has paused and has not finished speaking the spoken utterance. Various implementations include determining the personalized endpointing measure based on a portion of a text representation of the spoken utterance immediately preceding the candidate endpoint and a user-specific measure. Additionally or alternatively, the user-specific measure can be based on the text representation immediately preceding the candidate endpoint and one or more historical interactions with the user. In various implementations, each of the historical interactions are specific to the text representation and the user, and indicate whether a previous instance of the text representation was a previous endpoint for the user.