Mobile Device Localization Without Sensor Calibration or Sync
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining the location of mobile devices using camera images and inertial sensors face challenges such as requiring precise calibration, failing in texture-poor environments, needing time-synchronization of image and motion data, and requiring training for specific environments, which limits their applicability and accuracy.
Innovation Solution
A computer-implemented method using a convolutional neural network to determine image features, a first recurrent neural network for inertial sensor data, and a second recurrent neural network to combine features for location determination, allowing for robust and reliable location estimation without separate calibration or synchronization, and enabling end-to-end training with raw data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional visual odometry techniques are used to determine location from camera images, then location can be determined, but precise camera calibration is required and the system fails in texture-poor environments
Solution Approach 1:
The patent replaces traditional mechanical calibration procedures with a neural network-based system that automatically learns camera parameters and motion patterns from data. The neural network substitutes the manual calibration process, eliminating the need for precise manual camera calibration while maintaining location determination reliability.
Solution Approach 2:
The system performs self-calibration through the neural network learning process. Instead of requiring external calibration tools or procedures, the system automatically adapts to the specific camera and sensor configuration by training on collected data, enabling the system to calibrate itself without external intervention.
2Measurement precision
If neural networks are used to process camera images for location determination, then training for specific environments is required, but this limits applicability to new environments
Solution Approach 1:
The neural network is designed to process multiple types of input data (camera images, inertial sensor data, barometer data) and adapt to different environments through unified training. This multi-functional approach allows the same network architecture to operate across diverse environments without requiring environment-specific models, enhancing both accuracy and adaptability.
Solution Approach 2:
The system performs preliminary training with diverse training data that encompasses multiple environments and conditions before deployment. This pre-training phase enables the neural network to learn general patterns and adapt to various environments, reducing the need for environment-specific retraining while maintaining high accuracy.
3Reliability
If image data and motion data are combined to determine location, then more information is available for location determination, but time-synchronization is required making the system difficult to set up
Solution Approach 1:
The neural network automatically learns and compensates for timing offsets between camera and sensor data during training. Instead of requiring manual time-synchronization configuration, the system self-adjusts to handle the temporal relationships between different data sources, simplifying setup while maintaining reliable location determination.
Solution Approach 2:
The neural network acts as an intermediary that processes and integrates data from multiple sources with different timing characteristics. It learns to align and combine camera images, inertial sensor data, and barometer readings without requiring precise external synchronization, effectively mediating between the different data streams.
4Measurement precision
If camera and inertial sensors are calibrated separately with respect to each other, then individual sensor accuracy is maintained, but calibration and time-synchronization between sensors is required
Solution Approach 1:
The patent merges the calibration process for camera and inertial sensors into a unified neural network training process. Instead of performing separate calibrations and then synchronizing them, the system combines all sensor data and jointly trains the neural network to handle the integrated system, maintaining measurement accuracy while eliminating the complexity of separate calibration and synchronization procedures.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer-implemented method of determining the location of a mobile device comprising a camera and at least one inertial sensor. A sequence of images are captured over a period of time, and a first neural network determines features indicative of the motion of the device from pairs of consecutive images. Data indicative of the motion of the device is captured, and a second neural network determines features indicative of the motion of the device from the data. A third neural network then determines features indicative of the location of the device from the features determined by the first neural network and the second neural network. The location of the device is the determined from the features determined by the third neural network.