3D Tool Information Overlays for Stereo Robotic Displays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In robotic systems, providing tool information as graphic overlays on displayed images can obstruct views of tools and objects, especially when end effectors are in close proximity, and when information is not positioned proximate to the tools, it becomes distracting for operators to associate the information with the correct tool, particularly in stereo displays where depth perception is involved.
Innovation Solution
A method is implemented where the processor determines the tool information and pose of each tool, translating this information into a three-dimensional overlay that conforms to the tool's shape and is rendered directly on the tool's image in the display, allowing for clear association without obstructing the view and minimizing eye movement for the operator.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If tool information is provided as text or graphic in an area proximate to the corresponding tool, then the information can be readily associated with the tool, but it may objectionably obstruct images of the tools and objects
Solution Approach 1:
The patent transitions from two-dimensional display positioning to three-dimensional spatial positioning by rendering tool information as a 3D overlay that conforms to the tool's shape and position in the stereo display, allowing information to be associated with tools without obstructing views
Solution Approach 2:
The patent applies different visual properties to different regions by making the overlay semi-transparent and conforming it to the specific geometry of the tool, allowing information to be displayed locally on the tool while maintaining visibility of the tool itself
2Object-affected harmful factors
If tool information is provided in a boundary area or off to one side of the viewing area, then images of end effectors are not obscured, but it becomes distracting for operators to visually find and associate the information with the corresponding tool
Solution Approach 1:
The patent moves tool information from the 2D display plane to 3D space by rendering overlays at the depth and position of the actual tools in the stereo display, eliminating the need for eye movement between information and tool locations
Solution Approach 2:
The patent merges the tool information overlay with the tool image itself by positioning the overlay directly on the tool's location in the stereo display, creating a unified visual element that combines both information and tool visualization
3Loss of information
If the overlay is rendered in a stereo display at a different depth than the three-dimensional images, then tool information can be provided, but the operator's eyes must focus at different depths causing additional distraction
Solution Approach 1:
The patent creates a uniform depth plane by rendering the tool information overlay at the same depth as the tool images in the stereo display, eliminating depth differences that would require eye refocusing and reducing visual strain
Data Source
AI summary
A robotic system may comprise a first robotic arm operatively coupleable to a first tool. The first tool has a first working end. The system may also comprise an image capture device, a display, and a processor. The processor may be configured to cause an image of a work site, which was captured by the image capture device from a perspective of an image reference frame, to be displayed on the display. The image of the work site includes an image of the first working end of the first tool. The processor may also determine a position of the first working end of the first tool in the image of the work site and render a tool information overlay at the position of the first working end of the first tool in the image of the work site. The tool information overlay visually indicates state information for the first tool. The processor may also change the tool information overlay while the first tool is in a first operational state by changing a brightness of the tool information overlay.