Real-time ML Object Detection Parallel Threading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for real-time object inference in video data face processing inefficiencies, leading to significant lag and frame jitter, with display frame rates often falling below 50 frames per second, which is inadequate for applications like medical procedure monitoring.
Innovation Solution
A system that utilizes parallel processing to maintain a real-time frame rate of at least 50 frames per second for video input data while overlaying graphical indications of detected objects, using a machine learning network to perform inference on video frames and render both video and graphical information simultaneously.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If real-time object inferencing is performed using a machine learning network, then object detection accuracy is improved, but processing speed deteriorates due to GPU and CPU intensive operations
Solution Approach 1:
The patent divides the processing into separate threads: one thread dedicated to rendering video frames and another thread dedicated to performing machine learning inference. This segmentation allows independent optimization of each process, maintaining high video rendering speed while performing accurate object detection without one process blocking the other.
Solution Approach 2:
The patent introduces a temporal dimension by using parallel threading, allowing video rendering and object inference to occur simultaneously in different time streams rather than sequentially. This enables the system to maintain real-time video frame rates while performing computationally intensive object detection.
2Adaptability or versatility
If video data is processed through machine learning inference, then object identification capability is improved, but display frame rate deteriorates falling below 50 frames per second
Solution Approach 1:
The system segments the display output into two independent components rendered in parallel: the base video frames and the overlay graphical indications. The video rendering thread maintains high frame rates by processing only video data, while the inference thread processes object detection independently. This prevents the slow inference process from bottlenecking the overall display frame rate.
Solution Approach 2:
The patent performs preliminary rendering of video frames in one thread before overlaying the results of object inference in another thread. This preliminary action ensures that the video portion is already prepared and ready for display at the required frame rate, while object detection can proceed independently without delaying the video output.
3Device complexity
If sequential processing is used for video rendering and object inference, then processing simplicity is maintained, but processing efficiency deteriorates causing significant lag
Solution Approach 1:
The patent implements segmentation by creating separate processing threads for video rendering and object inference. This maintains relative simplicity within each thread while achieving high overall efficiency through parallel execution. The modular thread structure is easier to manage than complex sequential processing with multiple dependencies.
Solution Approach 2:
The system uses feedback mechanisms where the inference thread receives video frame data and returns object detection results that are then overlaid on the rendered video. This feedback loop operates independently in parallel, allowing both processes to maintain their own optimized workflows while coordinating through well-defined interfaces.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for real-time deployment of machine learning networks. One of the operations is performed by the system receiving video data from a video image capturing device. The received video data is converted into multiple video frames. These video frames are encoded into a particular color space format. The system renders a first display output depicting imagery from the multiple encoded video frames. The system performs an inference on the video frames using a machine learning network in order to determine the occurrence of one or more objects in the video frames. The system renders a second display output depicting graphical information corresponding to the determined one or more objects from the multiple encoded video frames. The system then generates a composite display output including the imagery of the first display output overlaid with the graphical information of the second display output.