Panoptic Segmentation Network for Occluded Road Scene Objects
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional autonomous vehicle perception systems are unreliable in handling heavy occlusions and complex scenes with complicated shapes, missing small and large objects, and are inflexible, leading to limited accuracy in object classification, dimensions, and orientation prediction.
Innovation Solution
A deep neural network (DNN) performs panoptic segmentation with a single pass, using a common trunk and multiple heads to predict class confidence, instance regression, and depth data, enabling pixel-level class and instance segmentation, which improves detection accuracy in complex environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple separate DNNs are used for different segmentation tasks, then each task can be specialized, but the system becomes slow and complex
Solution Approach 1:
The patent combines multiple separate DNNs into a single unified DNN that performs panoptic segmentation, instance segmentation, and class segmentation simultaneously. This integration reduces computational overhead and processing time while maintaining specialized functionality through shared feature extraction layers and task-specific output heads.
Solution Approach 2:
The unified DNN is designed with multi-functionality to handle multiple segmentation tasks concurrently. It includes a shared backbone network that extracts features universally applicable to all tasks, while task-specific heads provide specialized processing for panoptic, instance, and class segmentation, achieving both speed and accuracy.
2Device complexity
If conventional boundary regressors are used, then the system is simple, but it cannot handle complex scenes with occlusions and complicated shapes
Solution Approach 1:
The patent applies segmentation by dividing the complex scene understanding task into multiple specialized heads within the DNN: panoptic segmentation head for overall scene classification, instance segmentation head for individual object identification, and class segmentation head for object categorization. This segmented approach handles occlusions and complex shapes effectively.
Solution Approach 2:
The patent transitions from 2D boundary regression to 3D spatial reasoning by incorporating depth estimation and instance masking dimensions. This allows the system to handle occlusions and complex shapes by understanding objects in three-dimensional space rather than simple two-dimensional boundaries.
3Ease of manufacture
If individual DNN parts are trained separately, then training is simpler, but accuracy is limited due to lack of holistic scene understanding
Solution Approach 1:
The patent merges multiple training objectives into a unified loss function that trains all segmentation heads simultaneously. This joint training approach allows the DNN to learn holistic scene characteristics and interrelationships between different segmentation tasks, improving accuracy while maintaining training efficiency through shared gradient updates.
Solution Approach 2:
The patent implements feedback mechanisms where predictions from one segmentation head inform and refine predictions from other heads. The loss functions provide feedback signals that guide the shared backbone network to learn features beneficial for all tasks, creating a synergistic training process that improves overall accuracy.
Data Source
AI summary
A deep neural network(s) (DNN) may be used to perform panoptic segmentation by performing pixel-level class and instance segmentation of a scene using a single pass of the DNN. Generally, one or more images and/or other sensor data may be stitched together, stacked, and/or combined, and fed into a DNN that includes a common trunk and several heads that predict different outputs. The DNN may include a class confidence head that predicts a confidence map representing pixels that belong to particular classes, an instance regression head that predicts object instance data for detected objects, an instance clustering head that predicts a confidence map of pixels that belong to particular instances, and/or a depth head that predicts range values. These outputs may be decoded to identify bounding shapes, class labels, instance labels, and/or range values for detected objects, and used to enable safe path planning and control of an autonomous vehicle.