Gesture-Based Content Selection for Visual Search and Snippet Actions

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Users face difficulties in efficiently obtaining and processing relevant information from displayed content due to challenges in constructing effective search queries, especially with visual data, leading to irrelevant search results and cumbersome screenshot capture and cropping processes.

Innovation Solution

A computing system processes gesture inputs on a touchscreen to generate a gesture mask, classify the gesture, and perform data processing actions such as search, save, or share based on the gesture classification, using machine-learned models to determine the relevant content portion and action.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If users take screenshots and crop them to search for visual information, then they can obtain additional information about the visual content, but the process becomes cumbersome and time-consuming

Engineering Contradiction:

Improveinformation accessibilityVSAvoidtime for screenshot capture and cropping

Core Design Contradiction:

Loss of informationVSLoss of time

Solution Approach 1:

The system automatically captures and processes the relevant portion of the display based on the gesture input without requiring the user to manually take screenshots or crop images. The computing system performs the information extraction and processing autonomously, eliminating the need for user intervention in the capture and cropping steps.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces the mechanical manual process of screenshot capture and cropping with automated computer vision and gesture recognition systems. The system uses display data, gesture inputs, and machine-learned models to automatically identify and process the relevant content portions, substituting manual mechanical operations with automated digital processing.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Loss of information

If users construct search queries using text alone, then they can search for information, but it becomes difficult to accurately describe and search for visual content

Engineering Contradiction:

Improvesearch query accuracyVSAvoiddifficulty in constructing search queries

Core Design Contradiction:

Loss of informationVSEase of operation

Solution Approach 1:

The system introduces gesture inputs as an intermediary between the user and the search query construction process. Instead of requiring users to manually type or select text to describe visual content, the gesture serves as a mediator that directly captures the user's intent and translates it into an automated search query based on the gesture classification and content analysis.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system changes the parameter of information input from text-based to gesture-based. By accepting gesture inputs and processing them through gesture recognition models, the system transforms the search query generation process to better capture visual content characteristics, improving accuracy without requiring users to construct complex text queries.

Inventive Principle:

Principle #35Parameter changes

3Loss of information

If the system processes the entire displayed content for search, then comprehensive information can be obtained, but computational efficiency decreases

Engineering Contradiction:

Improvecompleteness of search resultsVSAvoidcomputational efficiency

Core Design Contradiction:

Loss of informationVSProductivity

Solution Approach 1:

The system segments the displayed content into relevant portions based on the gesture input and gesture mask. Instead of processing the entire display, the system identifies the specific region of interest defined by the gesture and processes only that segment, maintaining search completeness for the relevant area while significantly improving overall computational efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system applies local quality processing by focusing computational resources on the specific portion of content identified by the gesture input. The gesture mask and content selection process ensure that only the locally relevant content is processed in detail, while the rest of the display is ignored, optimizing computational efficiency without sacrificing the completeness of results for the target area.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS20260056653A1Content Selection and Action Determination Based on a Gesture Input

Publication Date: 2026.02.26 GOOGLE LLC

AI summary

Systems and methods for content processing can include obtaining a gesture input and display data, determining content selected by the gesture input, classifying the gesture, and performing a particular data processing action based on the content selection and the gesture classification. The particular data processing action can vary based on gesture classification. The content selection determination can include determining a gesture mask and then determining the features of the displayed content item that are within the gesture mask.