Augmenting Image Data via 3D Point Cloud Planar Segments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mobile computing devices lack the ability to effectively utilize their computational resources to provide users with additional relevant information during live image capture, typically only displaying basic data such as distance or business information in the viewfinder.
Innovation Solution
The system processes image data and 3D point cloud data to extract planar segments from the environment, which are then used to generate geometrically consistent augmented content like text, images, or videos, displayed on the device, enhancing the live view with relevant information about objects in the scene.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If basic information display is used in the viewfinder, then device complexity is reduced, but information completeness deteriorates
Solution Approach 1:
The patent segments information processing by separating basic information display from augmented content display. The system divides the viewfinder into different regions: the original camera view and augmented reality overlays. This allows comprehensive information to be presented without overwhelming the device complexity, as each segment handles specific processing tasks independently.
Solution Approach 2:
The patent introduces an intermediary processing layer that sits between the camera capture and the display output. This intermediary system processes image data, identifies objects, retrieves related information, and presents it in an augmented format. This mediator handles the complexity of information processing while keeping the user interface relatively simple through geometrically consistent overlays.
2Loss of information
If computational resources are utilized to provide additional information, then information completeness is improved, but processing time increases
Solution Approach 1:
The patent implements preliminary action by pre-processing and pre-fetching information. The system continuously analyzes the scene and pre-loads related information about identified objects before they become the focal point. This allows the augmented information to be displayed with minimal delay when the user focuses on a particular object, reducing perceived processing time while maintaining comprehensive information availability.
Solution Approach 2:
The patent maintains continuity of useful action by continuously processing image data in real-time rather than processing only when requested. The system continuously identifies objects, tracks their positions, and maintains information about them in readiness. This continuous processing ensures that when the user focuses on an object, the information is already prepared and can be displayed immediately, reducing processing delays.
3Manufacturing precision
If augmented content is displayed geometrically consistent with the environment, then visual accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent applies local quality by making augmented content geometrically consistent only in the specific regions where objects are detected and displayed. Rather than transforming the entire viewfinder, the system applies geometric transformations locally to the augmented overlays matching the 3D position and orientation of identified objects. This localized approach maintains visual accuracy where needed while minimizing overall computational complexity.
Solution Approach 2:
The patent transitions from 2D display to 3D spatial positioning by using depth information from the point cloud data. The augmented content is positioned in three-dimensional space relative to the detected objects, creating geometric consistency in multiple dimensions. This dimensional enhancement improves visual accuracy by placing information in the correct spatial context while using efficient algorithms to handle the increased complexity.
Data Source
AI summary
Embodiments of the invention describe processing a first image data and 3D point cloud data to extract a first planar segment from the 3D point cloud data. This first planar segment is associated with an object included in the first image data. A second image data is received, the second image data including the object captured in the first image data. A second planar segment related to the object is generated, where the second planar segment is geometrically consistent with the object as captured in the second image data. This planar segment is generated based, at least in part, on the second image data, the first image data and the first planar segment. Embodiments of the invention may further augment the second image data with content associated with the object. This augmented image may be displayed such that the content is displayed geometrically consistent with the second planar segment.