Unified Video Network for Multi-Object Detection, Segmentation, and Pose
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-stage models for object detection in live video streaming face significant performance degradation and increased inference time when detecting multiple objects, particularly when the number of instances exceeds twenty, leading to unreliable detection.
Innovation Solution
A single deep learning network architecture, known as the Human Trinity model, is employed to simultaneously perform object detection, instance segmentation, and pose estimation, utilizing a backbone network and instance/pixel heads to convert input images into feature maps, which are then processed to generate object detection, instance segmentation, and pose estimation results without the need for additional association operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a multi-stage model is used to detect multiple objects and their pixels, then detection capability is improved, but inference time significantly increases when the number of instances exceeds twenty
Solution Approach 1:
The patent merges object detection, instance segmentation, and pose estimation into a single-stage model architecture. The detection head, segmentation head, and pose estimation head all process images simultaneously through a unified backbone network, eliminating the sequential processing steps of multi-stage models and achieving constant inference time regardless of the number of objects detected
Solution Approach 2:
The single-stage model performs multiple functions (object detection, instance segmentation, and pose estimation) within one unified architecture. The backbone network extracts features that are simultaneously used by all three task heads, making the system universally applicable to multiple object analysis tasks without requiring separate processing stages
2Reliability
If a multi-stage model is used to detect multiple objects, then detection accuracy is improved, but power consumption increases significantly when the number of instances increases
Solution Approach 1:
By combining multiple detection functions into a single stage, the model processes all objects simultaneously through one backbone network rather than sequentially through multiple stages. This merging approach maintains detection accuracy while significantly reducing the computational burden and associated power consumption, especially when dealing with a large number of instances
3Loss of time
If a single deep learning network is used to simultaneously perform object detection,instance segmentation, and pose estimation, then inference time is maintained constant, but model complexity increases
Solution Approach 1:
The model segments the processing pipeline into distinct functional components: a backbone network for feature extraction, a detection head for object location, a segmentation head for pixel-level segmentation, and a pose estimation head for skeleton detection. This segmentation of functions within a unified architecture allows each component to be optimized independently while maintaining constant inference time through parallel processing
Data Source
AI summary
A video processing method for detecting location(s), pixels, and skeleton(s) of one or more objects and an associated video processing circuit are provided. The video processing method may include: utilizing a backbone network in a predetermined model of a single deep learning network to receive and convert input image data having the object(s) into at least one feature map; and utilizing at least one instance head and a pixel head in the predetermined model of the single deep learning network to receive and convert the aforementioned at least one feature map into one or more object detection results, one or more instance segmentation results and one or more pose estimation results of the object(s).