Wearable Gesture Recognition via Depth Sensor Data Fusion

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current mobile devices require manual user input through mechanical keyboards or touch screens for interacting with augmented reality and other applications, which can be inconvenient and disruptive, especially in social interactions.

Innovation Solution

A wearable multimedia device with a camera and depth sensor that detects real-time data streams to infer user intent through gesture recognition, allowing for touchless interaction by projecting 3D data points, identifying hand clusters, and determining pointing directions to select services or content without the need for manual input.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If manual user input through mechanical keyboard or touch screen is used, then user interaction with applications is enabled, but user convenience deteriorates and social interaction is interrupted

Engineering Contradiction:

Improveuser interaction convenienceVSAvoidsocial interaction interruption

Core Design Contradiction:

Ease of operationVSLoss of time

Solution Approach 1:

The patent replaces mechanical input devices (keyboards, touch screens) with a gesture recognition system using depth sensors and cameras to detect hand movements and gestures in 3D space, enabling touchless interaction that maintains convenience while avoiding social interaction interruption

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system introduces an intermediary gesture recognition layer between the user and the application interface, where hand gestures captured by depth sensors and cameras are translated into command inputs, allowing users to interact without direct physical contact with the device

Inventive Principle:

Principle #24Intermediary (Mediator)

2Ease of operation

If gesture recognition using depth sensor and camera is implemented, then touchless interaction is enabled, but device complexity increases

Engineering Contradiction:

Improvetouchless interaction capabilityVSAvoidsensor and processing system complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent integrates multiple sensors (depth sensor, camera, accelerometer, gyroscope) into a unified gesture recognition system that handles various interaction scenarios (hand detection, gesture classification, pointing recognition) through a single multi-functional processing pipeline, reducing overall system complexity despite the presence of multiple sensors

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11816269B1Gesture recognition for wearable multimedia device using real-time data streams

Publication Date: 2023.11.14 HEWLETT PACKARD DEVELOPMENT COMPANY LP

AI summary

Systems, methods, devices and non-transitory, computer-readable storage mediums are disclosed for gesture recognition for a wearable multimedia device using real-time data streams. In an embodiment, a method comprises: detecting a trigger event from one or more real-time data streams running on a wearable multimedia device; taking one or more data snapshots of the one or more real-time data streams; inferring user intent from the one or more data snapshots; and selecting a service or preparing content for the user based on the inferred user intent. In an embodiment, a hand and finger pointing direction is determined from a depth image, a 2D bounding box for the hand/finger is projected into a 2D image space and compared to bounding boxes for identified/labeled objects in the 2D image space to identify an object that the hand is holding or the finger is pointing toward.