CNN with LSTM Layers for Temporal Semantic Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image segmentation methods are not fast or robust enough to handle various inputs, particularly in applications like autonomous vehicles where timely and accurate object detection and tracking are crucial.
Innovation Solution
The use of a convolutional neural network (CNN) architecture that incorporates Long Short-Term Memory (LSTM) layers to associate temporal information with spatial information, improving processing time and accuracy, and the generation of 'noisy' datasets by propagating labels from oversegmented ground truth images to unlabeled images for training, allowing for more accurate semantic segmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional image segmentation methods are used, then the system is simpler to implement, but the processing speed is too slow and accuracy is insufficient for autonomous vehicle applications
Solution Approach 1:
The patent segments the image processing task into multiple parallel streams: a first CNN processes images at full resolution for detailed segmentation, while a second CNN processes downsampled images for broader context. This segmentation of the processing workload enables faster overall performance without requiring a single overly complex model, resolving the contradiction between speed and complexity.
Solution Approach 2:
The patent introduces temporal dimension by incorporating video sequences and motion information into the segmentation process. By processing images across multiple time frames and combining spatial segmentation with temporal motion analysis, the system achieves faster and more accurate object detection without proportionally increasing computational complexity.
2Measurement precision
If a single high-accuracy segmentation model is used, then object identification accuracy improves, but processing time increases
Solution Approach 1:
The patent divides the segmentation task between two parallel CNN models with different complexity levels. The first CNN handles detailed segmentation for accuracy-critical regions, while the second CNN provides faster, coarser segmentation for less critical areas. This segmentation of the computational task maintains overall accuracy while reducing total processing time.
Solution Approach 2:
The patent applies full-resolution detailed segmentation only where necessary (partial action) rather than uniformly across the entire image. By selectively applying high-computation segmentation to regions containing objects of interest and using faster methods for background areas, the system maintains accuracy for critical detections while reducing overall processing time.
Data Source
AI summary
Systems and methods for segmenting an image using a convolutional neural network are described herein. A convolutional neural network (CNN) comprises an encoder-decoder architecture, and may comprise one or more Long Short Term Memory (LSTM) layers between the encoder and decoder layers. The LSTM layers provide temporal information in addition to the spatial information of the encoder-decoder layers. A subset of a sequence of images is input into the encoder layer of the CNN and a corresponding sequence of segmented images is output from the decoder layer. In some embodiments, the one or more LSTM layers may be combined in such a way that the CNN is predictive, providing predicted output of segmented images. Though the CNN provides multiple outputs, the CNN may be trained from single images or by generation of noisy ground truth datasets. Segmenting may be performed for object segmentation or free space segmentation.