Vehicle Video Segmentation With Early-Exit Edge Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional models are not fast enough to process video data in real-time for video instance segmentation in vehicles due to constrained computing units and large video sizes, making it difficult for vehicles to analyze driving scenes effectively.
Innovation Solution
A system utilizing an edge server to train a machine learning model with early-exit subnets, allowing vehicles to perform real-time video instance segmentation by leveraging transformer-based models and early-exit subnets that enable similarity score-based early-exit, enabling efficient processing of video frames.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional deep learning models are used for video instance segmentation, then segmentation accuracy can be achieved, but processing speed is too slow for real-time application due to constrained computing units and large video data size
Solution Approach 1:
The model is segmented into multiple early-exit subnets, each capable of independent inference. This allows the system to process video frames through multiple parallel pathways rather than a single sequential model, significantly improving processing speed while maintaining segmentation accuracy through the ensemble of subnet predictions
Solution Approach 2:
The system performs partial processing by allowing early exits at different subnet levels. For simple cases, only the first few subnets are needed, providing fast results. For complex cases, more subnets are processed sequentially, ensuring accuracy is maintained when needed without always incurring the full computational cost
2Reliability
If full deep learning models process all video frames, then comprehensive object identification is achieved, but computational resources are wasted on simple frames that could be processed faster
Solution Approach 1:
The system dynamically adjusts the processing depth for each video frame based on complexity. Simple frames exit early through the first subnet, while complex frames continue through multiple subnets. This dynamic adaptation maintains reliable object identification for all frames while maximizing processing throughput by avoiding unnecessary computation on simple frames
Solution Approach 2:
The system uses feedback from intermediate processing stages to determine whether to continue or exit early. Each subnet provides feedback on the confidence and quality of its segmentation results, allowing the system to make informed decisions about whether additional processing is needed, thus balancing reliability and productivity
Data Source
AI summary
A method for video instance segmentation is provided. The method includes inputting a plurality of video frames collected by a sensor of a vehicle to a trained machine learning model to obtain an n-th output from an n-th layer of the trained machine learning model and an n+1-st output from an n+1-st layer of the trained machine learning model, the trained machine learning model comprising a deep learning model and early-exit subnets, and in response to determining that a difference between the n-th output and the n+1-st output is less than a threshold value, controlling the vehicle based on the n+1-st output, the n+1-st output includes information about instances in the plurality of video frames.