Robot Engagement Detection from Visible and Hidden Body Keypoints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing robotic systems lack the ability to efficiently determine the extent of human engagement, which is crucial for effective human-robot interaction and coordination.
Innovation Solution
A control system configured to capture 2D images of humans and identify visible and hidden keypoints, using a machine learning model to determine the extent of engagement based on the coordinates and visibility of these keypoints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional robotic systems are used without engagement detection, then the system complexity is low, but the ability to determine human engagement and coordinate effectively is poor
Solution Approach 1:
The engagement detection system segments the human body into multiple keypoints (e.g., head, shoulders, hands) and tracks their positions and visibility states independently. This segmentation allows the system to analyze engagement by evaluating specific body part configurations rather than processing entire body images, improving detection reliability while managing computational complexity through divided analysis
Solution Approach 2:
The system transitions from 2D image data to 3D spatial reasoning by inferring the visibility states of keypoints that are not directly visible in the 2D camera view. By considering the third dimension (depth) and occlusion relationships, the system can determine whether keypoints are hidden behind the body or simply out of frame, enhancing engagement assessment accuracy without requiring additional 3D sensors
2Measurement precision
If 2D images are used for engagement detection, then the measurement approach is simple, but the precision of determining hidden keypoint positions is limited
Solution Approach 1:
The system introduces an intermediary inference mechanism that uses visible keypoints and occlusion analysis as mediators to estimate the positions and visibility states of hidden keypoints. Rather than directly observing all keypoints, the system uses visible body parts as reference frames to logically deduce the states of obscured parts, improving measurement precision through indirect observation and reasoning
Solution Approach 2:
The patent replaces direct mechanical/optical observation of all body parts with a computational inference system. Instead of using multiple cameras or depth sensors to directly capture hidden keypoints, the system substitutes these physical measurement devices with algorithmic reasoning that processes 2D image data and infers 3D keypoint states through machine learning models and geometric analysis
Data Source
AI summary
A method includes receiving, from a camera disposed on a robotic device, a two-dimensional (2D) image of a body of an actor and determining, for each respective keypoint of a first subset of a plurality of keypoints, 2D coordinates of the respective keypoint within the 2D image. The plurality of keypoints represent body locations. Each respective keypoint of the first subset is visible in the 2D image. The method also includes determining a second subset of the plurality of keypoints. Each respective keypoint of the second subset is not visible in the 2D image. The method further includes determining, by way of a machine learning model, an extent of engagement of the actor with the robotic device based on (i) the 2D coordinates of keypoints of the first subset and (ii) for each respective keypoint of the second subset, an indicator that the respective keypoint is not visible.