3D Tool Information Overlays for Clear Stereo Image Guidance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In robotic systems, providing tool information as graphic overlays on displayed images of tools can obstruct views of the tools and objects being operated on, and can be distracting for operators to visually associate the information with the correct tool, especially in stereo displays.
Innovation Solution
A method implemented in the robotic system to render tool information as graphic overlays directly on the images of tools in the display, using a three-dimensional overlay that conforms to the tool's shape, allowing for clear association of the information with the correct tool without obstructing the view.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If tool information is provided as text or graphic in an area proximate to its corresponding tool, then the information can be readily associated with the tool, but it may objectionably obstruct images of the tools and objects upon which the tools are performing a procedure
Solution Approach 1:
The patent transitions from two-dimensional display placement to three-dimensional spatial positioning. Tool information is rendered as a three-dimensional graphic overlay positioned in three-dimensional space near the tool, allowing the information to be associated with the tool through spatial proximity without obstructing the two-dimensional image view on the display.
Solution Approach 2:
The patent applies different properties to different parts of the information display. The tool information is rendered with semi-transparent or selectively transparent properties, allowing certain areas to be opaque for readability while other areas remain transparent to maintain visibility of the underlying tool images.
2Object-affected harmful factors
If tool information is provided in an area not proximate to the tool, then the view of tools and objects is not obstructed, but it becomes distracting for the operator to visually find and associate the tool information with its corresponding tool
Solution Approach 1:
The patent uses three-dimensional positioning to place tool information in virtual space near the corresponding tool, maintaining spatial association without requiring the operator to shift gaze across the two-dimensional display. The three-dimensional overlay system allows the information to appear adjacent to the tool in the rendered view while not physically blocking the underlying image.
Solution Approach 2:
The patent introduces a three-dimensional graphic overlay as an intermediary between the tool information and the display image. This overlay acts as a mediator that carries the information while allowing the underlying tool image to remain visible through transparency or selective opacity, facilitating association without obstruction.
3Stability of the object's composition
If the tool information is stationary and the end effectors are moving, then the information remains stable on display, but it becomes not readily apparent which tool the tool information pertains to when end effectors are in close proximity
Solution Approach 1:
The patent implements dynamic tool information rendering where the position, opacity, or visibility of the information overlay adjusts based on the real-time positions and movements of the end effectors. When end effectors are in close proximity, the system dynamically modifies the display properties to maintain clear association between the information and the correct tool.
Solution Approach 2:
The system continuously monitors the positions of end effectors and uses this feedback to adjust the rendering of tool information. The display properties of the information overlay are dynamically modified based on the relative positions of tools, ensuring that the information remains clearly associated with its corresponding tool even during complex movements.
Data Source
AI summary
A system comprises: a robotic arm operatively coupleable to a tool comprising a working end; and an input device communicatively coupled to the robotic arm. The input device is manipulatable by an operator. The system further comprises a processor configured to cause an image of a work site, captured by an image capture device from a perspective of an image reference frame, to be displayed on a display. The image of the work site includes an image of the working end of the tool. The processor is further configured to determine a position of the working end of the tool in the image of the work site and render a tool information overlay at the position of the working end of the tool in the image of the work site. The tool information overlay visually indicates an identity of the input device.