AR/VR Streaming Object Recognition with Remote Viewport Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
AR/VR devices face challenges in efficiently processing and rendering large numbers of objects in real-time due to limited computational resources, particularly when the user moves rapidly, leading to difficulties in locating target objects within the viewport.
Innovation Solution
A method and system for streaming-based object recognition that offloads object detection and recognition tasks to a remote server, allowing devices to transmit video data and coordinates, receive processed data in a sequence based on object distances, and render objects accordingly, optimizing computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If object detection and recognition tasks are performed locally on the device, then processing speed and real-time performance improve, but computational resource consumption increases beyond device capabilities
Solution Approach 1:
The patent extracts the computationally intensive object detection and recognition tasks from the local device and relocates them to a remote server. The device transmits only video data and viewport coordinates to the server, which performs the heavy computational work and returns recognition results. This extraction principle resolves the contradiction by maintaining fast local response while offloading energy-intensive processing to external infrastructure.
Solution Approach 2:
The patent introduces a remote server as an intermediary between the device and the object recognition process. The server acts as a mediator that receives video data from the device, performs comprehensive object detection and recognition across multiple scales and viewpoints, and returns structured recognition results. This intermediary approach allows the device to achieve advanced recognition capabilities without directly consuming the computational resources required for such processing.
2Measurement precision
If comprehensive object detection across multiple viewpoints and distances is performed, then recognition accuracy improves, but processing time and computational load increase
Solution Approach 1:
The patent applies preliminary action by transmitting viewport coordinates and establishing processing parameters before the actual object detection begins. The server receives advance information about the device's viewpoint, orientation, and distance, allowing it to preconfigure the detection process. This preliminary setup enables the server to optimize its processing strategy, achieving comprehensive multi-viewpoint and multi-distance detection without incurring excessive processing delays during runtime.
Solution Approach 2:
The patent implements dynamics by continuously updating viewport coordinates and processing parameters based on the device's real-time movement and orientation changes. The system dynamically adjusts detection parameters, transmission intervals, and processing strategies according to current operational conditions. This dynamic adaptation allows the system to maintain high recognition accuracy across varying viewpoints and distances while optimizing processing time based on actual scene complexity and device motion patterns.
3Productivity
If video data is transmitted continuously at high frequency, then real-time recognition performance improves, but network bandwidth consumption and data transmission overhead increase
Solution Approach 1:
The patent extracts only the essential data elements required for remote processing: video data frames and viewport coordinates. Instead of transmitting complete scene information or redundant metadata, the system extracts and transmits only the minimum necessary data for the server to perform accurate object detection and recognition. This extraction principle reduces data transmission volume while maintaining real-time recognition performance by sending precisely the information needed for effective processing.
Data Source
AI summary
Systems and methods for streaming-based object recognition may include a device which transmits, to a server, video data and first coordinates of a first viewport for the device at a first time instance. The video data may include one or more objects in the first viewport. The device may transmit second coordinates of a second viewport for the device at a second time instance. The device may receive, from the server, data corresponding to the one or more objects within the second viewpoint. The data may be received in a sequence according to coordinates of the one or more objects relative to the second viewport. The device may render the data relative to the one or more objects within a third viewport at a third time instance.