Human-Aware Visual SLAM for Metric 3D Scene Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional scene reconstruction techniques from digital videos with changing camera angles or moving humans result in inaccurate object scaling and fail to generate accurate camera trajectories due to the lack of a consistent coordinate plane, leading to misproportioned objects and incomplete trajectory information.
Innovation Solution
A scene reconstruction system that uses a pretrained monocular depth model to generate depth maps and human meshes, combined with a simultaneous localization and mapping (SLAM) model to estimate object sizes relative to human meshes, ensuring accurate scaling and camera trajectory determination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional scene reconstruction techniques are used, then the system can process digital videos, but the object scaling becomes inaccurate and camera trajectories cannot be generated accurately
Solution Approach 1:
The patent introduces a human mesh as an intermediary reference object with known metric dimensions. By establishing a coordinate plane through the human mesh and using it as a mediator for scale calibration, the system can accurately determine object sizes and camera trajectories without direct measurement errors
Solution Approach 2:
The system changes the parameter reference from pixel-based relative measurements to metric-based absolute measurements by incorporating human body dimensions (known parameters) into the scene reconstruction, enabling accurate scaling and trajectory calculation
2Shape
If conventional scene reconstruction techniques are used, then the system can generate scene reconstructions, but the objects become misproportioned due to lack of consistent coordinate plane
Solution Approach 1:
The patent segments the scene reconstruction into two parts: a human mesh component with known metric dimensions and a scene object component. This segmentation allows independent optimization of each component while maintaining consistent coordinate plane through their relationship
Solution Approach 2:
The human mesh serves as a mediator that bridges the gap between camera coordinates and world coordinates, providing a consistent reference frame that ensures proper object proportions in the final reconstruction
3Measurement precision
If the system uses human mesh dimensions to determine object sizes, then accurate scaling is achieved, but the computational process becomes more complex
Solution Approach 1:
The system uses the human mesh, which is already computed during the scene reconstruction process, to provide scale information. This self-service approach leverages existing computational results rather than requiring entirely new complex measurements
Solution Approach 2:
The system transforms the measurement problem from direct object measurement (complex) to human reference measurement (simpler, as human dimensions are known), changing the parameter being measured to achieve accurate object size determination
Data Source
AI summary
In implementation of techniques for scene reconstruction from digital video of moving humans, a computing device implements a scene reconstruction system to receive a digital video depicting a scene including a human and an object. The scene reconstruction system then determines a depth of the human and a depth of the object in the digital video and generates a human mesh modeled from the human in the digital video. Using a machine learning model, the scene reconstruction system determines a size of the object by comparing the depth of the human, the depth of the object, and an estimated dimension of the human mesh. The scene reconstruction system then generates a scene reconstruction including the human mesh and a three-dimensional representation of the object based on the size of the object.