Automated Assistant Gesture Invocation in Noisy Environments

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Automated assistants often fail to accurately interpret user requests in noisy environments, leading to inefficient use of computational and network resources due to the need for repeated invocation phrases and processing of unintended audio inputs.

Innovation Solution

A portable computing device transitions between operating modes, allowing invocation via spoken utterance in quiet conditions and via non-audio physical gestures in noisy conditions, using proximity sensors and contextual data to determine the appropriate mode.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the automated assistant processes audio input in noisy environments using spoken trigger phrases, then it can invoke the assistant, but it fails to accurately interpret user requests and processes unintended audio inputs

Engineering Contradiction:

Improveaccuracy of user request interpretationVSAvoidcomputational and network resources

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The system changes the invocation parameter from audio-based trigger phrases to gesture-based inputs when noise levels exceed a threshold. The processor monitors audio characteristics and dynamically switches between audio invocation and gesture invocation modes, transforming the interaction modality parameter based on environmental conditions.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent replaces the acoustic detection mechanism with a gesture detection mechanism when audio processing becomes unreliable. Instead of using the microphone and speech recognition system in noisy environments, the system uses motion sensors or camera-based gesture recognition to detect user intent, substituting one detection modality for another.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Reliability

If the automated assistant requires repeated invocation phrases in noisy environments, then it can attempt to capture user input, but it wastes computational and network resources

Engineering Contradiction:

Improvesuccessful invocation of automated assistantVSAvoidefficient use of computational resources

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system performs preliminary gesture detection before activating full speech processing. When a gesture indicating invocation intent is detected, the system activates the automated assistant and switches to listening mode, preventing unnecessary processing attempts and resource waste from repeated failed trigger phrase detection.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The gesture detection system serves as an intermediary mechanism that bridges the user and the automated assistant in noisy environments. Instead of directly processing potentially unreliable audio trigger phrases, the system uses gestures as an intermediate signal to confirm user intent before activating full processing modes.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Ease of operation

If the automated assistant uses audio processing in noisy environments, then it can detect trigger phrases, but it captures audio from multiple sources including background noise

Engineering Contradiction:

Improveability to invoke automated assistantVSAvoiddiscernibility of trigger phrase from background noise

Core Design Contradiction:

Ease of operationVSLoss of information

Solution Approach 1:

The patent segments the invocation process into two distinct phases: gesture-based activation phase and audio listening phase. The gesture detection segment handles the initial invocation trigger, separating this function from audio processing. Once activated, the system then listens for user requests, knowing that the gesture has already confirmed legitimate user intent.

Inventive Principle:

Principle #1Segmentation

Applied Scientific Principles

This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.

Function Achieved in This Case

Enables reliable and resource-efficient interaction with automated assistants in noisy environments by allowing invocation through physical gestures, reducing unnecessary processing and power consumption.

Implementation Method 1

the automated assistant is invoked by the portable computing device in response to the proximity sensor detecting that a physical gesture was performed by a user

Methodology Applied

Scientific EffectProximity sensing:

Data Source

PatentEP3853712B1Operating modes that designate an interface modality for interacting with an automated assistant

Publication Date: 2024.12.25 GOOGLE LLC

Figure 1A~1B
Figure 2A
Figure 2B

AI summary

Implementations described herein relate to transitioning a computing device between operating modes according to whether the computing device is suitably oriented for received non-audio related gestures. For instance, the user can attach a portable computing device to a docking station of a vehicle and, while in transit, wave their hand near the portable computing device in order to invoke the automated assistant. Such action by the user can be detected by a proximity sensor and/or any other device capable of determining a context of the portable computing device and/or an interest of the user in invoking the automated assistant. In some implementations location, orientation, and/or motion of the portable computing device can be detected and used in combination with an output of the proximity sensor to determine whether to invoke the automated assistant in response to an input gesture from the user.