Cross-Modality Descriptor Generation for Unpaired Sensor Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autonomous vehicles (AVs) face challenges in data retrieval when paired data from different sensor modalities are not captured simultaneously, making it difficult to determine which data corresponds to which scene.
Innovation Solution
A method involving encoders and transformers to generate and transform descriptors across different sensor modalities, using autoencoders and contrastive learning to align descriptors, enabling data retrieval and comparison across unpaired modalities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data from different modalities are labelled independently, then each modality can be processed separately, but it becomes impossible to determine which data corresponds to which scene
Solution Approach 1:
The patent introduces a descriptor as an intermediary representation that bridges different modalities. The descriptor is generated from data of a first modality and used to retrieve data of a second modality, enabling scene correspondence without requiring paired training data. This intermediary solves the problem of losing scene correspondence information when modalities are processed independently.
2Reliability
If paired data from different sensor modalities are captured simultaneously, then scene correspondence can be determined, but the system requires complex synchronized capture and processing
Solution Approach 1:
The patent performs preliminary encoding of data from the first modality into a descriptor before retrieval operations. This pre-processing step creates a modality-agnostic representation that can be used to search for corresponding data in other modalities without requiring simultaneous capture. The preliminary encoding action enables reliable scene correspondence while avoiding complex synchronized systems.
Solution Approach 2:
The patent replaces the mechanical synchronization system (simultaneous capture requirements) with an information-based system using descriptors. Instead of relying on temporal synchronization of sensor captures, the system uses learned descriptor representations to establish scene correspondence, substituting a complex mechanical coordination system with a simpler information processing approach.
3Measurement precision
If encoders are trained on paired training data, then accurate cross-modality transformation can be achieved, but the system requires availability of paired training data which may not exist
Solution Approach 1:
The patent inverts the traditional approach by training encoders on unpaired data from each modality separately, rather than requiring paired data. The encoders are trained to create descriptors that capture modality-specific characteristics, and then a transformation mechanism learns to map between these descriptor spaces. This inversion allows the system to work with available unpaired data while still achieving accurate cross-modality transformation.
Data Source
AI summary
The present invention relates to a computer-implemented method of generating a descriptor associated with data of a first modality. The method comprises: receiving first data associated with a first modality, and second data associated of a second modality, wherein the first and second modalities are different; generating respective first and second descriptors corresponding to the respective first and second modalities by encoding the first and second data using respective first and second encoders, the first and second encoders respectively trained based on first training data of the first modality and second training data of the second modality; transforming the first descriptor into a third descriptor, the third descriptor corresponding to the second modality; and storing the third descriptor in a database