Remote Visual Instruction Overlay for Mobile Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing remote specialist assistance systems are limited by unnatural interfaces, require specialized devices, and are not suited for use with handheld mobile phones, leading to increased overhead time and reduced mobility for specialists, and are not intuitive for users.
Innovation Solution
A method and system that uses commercially available smartphones to enable a specialist to provide visually intuitive instructions by capturing video streams of both the object of interest and the specialist's device, performing foreground/background segmentation, and overlaying the specialist's device onto the object's background, allowing natural gestures and demonstrations without additional equipment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If remote specialist assistance systems use traditional interfaces (cursor, pointing devices), then the specialist can provide instructions, but the interface becomes unnatural and less intuitive
Solution Approach 1:
The system captures video of the specialist's hand gestures and pointing actions, then overlays this video feed onto the user's camera view. This creates a natural visual copy of the specialist's actions that appears directly in the context of the object being discussed, eliminating the need for cursors or pointing devices and making the interface intuitive and natural
Solution Approach 2:
The system uses video overlay as an intermediary between the specialist and user. Instead of direct communication through complex interfaces, the specialist's hand gestures are captured and transmitted as visual information that the user sees through their device, creating a natural bridge that preserves the intuitiveness of in-person interaction
2Reliability
If remote assistance requires specialized devices (head mounted cameras, markers), then the specialist can be assisted, but the overhead time increases and mobility is reduced
Solution Approach 1:
The system uses standard mobile devices (smartphones, tablets) that specialists already possess for multiple purposes. The device's camera serves both for capturing the object of interest and for capturing the specialist's instructional gestures. This universal use of common devices eliminates the need for specialized equipment while maintaining assistance quality
Solution Approach 2:
The specialist's own mobile device serves the dual purpose of both the instruction delivery tool and the communication medium. The device's existing camera and display are utilized without requiring additional specialized equipment, allowing specialists to provide assistance using their personal devices without increased overhead time
3Adaptability or versatility
If remote assistance systems use specialized equipment (head mounted displays, external cameras), then visual instruction can be provided, but the system becomes less adaptable and requires additional equipment
Solution Approach 1:
The system is designed to work with standard mobile devices that are universally owned and carried by specialists. The device's camera, processor, and display are used for multiple functions: capturing the object, capturing gestures, processing video overlays, and displaying instructions. This eliminates the need for specialized equipment and maximizes adaptability
Solution Approach 2:
The system merges multiple functions into the specialist's existing mobile device. The device simultaneously serves as the camera for capturing the object, the camera for capturing gestures, the processor for creating video overlays, and the display for showing instructions. This consolidation eliminates the need for separate specialized equipment while maintaining full functionality
Data Source
AI summary
The present invention provides a method of providing a digitally represented visual instruction from a specialist (2) operating a visual instruction device (4) to a user (6) in need of the visual instruction in relation to an object of interest (8) and a system therefore. In the method a first video stream comprising images of the object of interest is recorded by a first mobile device (12) associated with the user. The first video stream is sent (22) to a second mobile device (18) associated with the specialist. A second video stream comprising the visual instruction device is recorded by the second mobile device and a foreground/background segmentation operation and a background replacement operation is performed on the second and first video stream to provide a third video stream comprising images of the visual instruction device with the first video stream as background. The third video stream is sent (28) to the first mobile device and is displayed essentially simultaneously on first and second displays on both mobile devices allowing the user, by viewing the first display, to be instructed by the specialist and the specialist, by viewing the second display, to operate the visual instruction device properly in relation to the object of interest. An alternative method using a projection device in the first mobile device and systems corresponding to the methods are also provided.