ToF Camera Model Training for Realistic Noise Simulation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing model-based time-of-flight (ToF) simulators struggle to accurately reproduce the error sources and noise of real ToF cameras, requiring costly ground-truth data and failing to account for statistical errors due to manufacturing variations and thermal fluctuations.

Innovation Solution

A data-driven approach using a time-of-flight simulation data training circuitry that obtains and updates a ToF camera model by comparing real data with model data, incorporating a neural network to refine the model and simulate ToF data, reducing the need for ground-truth data and accounting for statistical errors.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of manufacture

If model-based ToF simulators use mathematical models and renderers to simulate ToF camera behavior, then the simulation can be performed without real data, but the accuracy in reproducing error sources and noise of real cameras deteriorates

Engineering Contradiction:

Improveease of simulationVSAvoidaccuracy of error reproduction

Core Design Contradiction:

Ease of manufactureVSMeasurement precision

Solution Approach 1:

The patent creates a digital copy of the real ToF camera by training a neural network on real camera data. This trained neural network model replicates the specific error sources and noise characteristics of the target camera type, allowing accurate simulation without requiring continuous access to real cameras or extensive ground-truth data.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent replaces traditional mathematical physics-based models with a data-driven neural network model. This substitution allows the system to capture complex, non-linear error sources and noise patterns that are difficult to model analytically, thereby improving accuracy while maintaining computational efficiency.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Extent of automation

If model-based simulators use synthetic scenes and camera models as input, then the simulation process can be automated, but the ability to account for manufacturing variations and thermal fluctuations deteriorates

Engineering Contradiction:

Improveautomation of simulationVSAvoidaccounting for statistical errors

Core Design Contradiction:

Extent of automationVSAdaptability or versatility

Solution Approach 1:

The patent performs preliminary training of the neural network model using real camera data that encompasses various manufacturing variations and thermal conditions. This pre-training phase captures statistical error patterns across different operating conditions, enabling the model to automatically account for these variations during subsequent simulations without requiring real-time adjustments.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent employs a feedback mechanism where the neural network model is continuously refined by comparing simulated output with actual real camera measurements. This feedback loop allows the model to adapt and improve its representation of manufacturing variations and thermal effects, enhancing both accuracy and robustness across different camera units and conditions.

Inventive Principle:

Principle #23Feedback

3Measurement precision

If ToF simulators require ground-truth data for training, then the accuracy of simulated data can be improved, but the cost and complexity of data collection increases

Engineering Contradiction:

Improveaccuracy of simulated dataVSAvoidcomplexity of data collection

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent creates a comprehensive digital replica of the ToF camera's error characteristics through neural network training. Once trained on a relatively small dataset, this digital copy can generate accurate simulated data for numerous applications without requiring additional ground-truth data collection, thereby reducing ongoing costs and complexity while maintaining high accuracy.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent transforms the approach from collecting extensive ground-truth data for each simulation scenario to training a versatile neural network model that can adapt to different conditions through parameter adjustments. The model learns underlying error patterns that generalize across various scenes, lighting conditions, and camera parameters, reducing the need for exhaustive data collection.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentEP4002327B1Time-of-flight simulation data training circuitry, time-of-flight simulation data training method, time-of-flight simulation data output method, time-of-flight simulation data output circuitry

Publication Date: 2026.01.28 SONY EUROPE BV

Figure 1~2
Figure 3
Figure 4

AI summary

The present disclosure generally pertains to time-of-flight simulation data training circuitry, configured to: obtain time-of-flight camera model data based on a time-of-flight camera model modelling a real time-of-flight camera of a real time-of-flight camera type; obtain real time-of-flight data from at least one real time-of-flight camera of the real time-of-flight camera type; determine a difference between the real time-of-flight data and the time-of-flight camera model data; and update, based on the determined difference, the time-of-flight camera model for generating simulated time-of-flight data representing the real time-of-flight camera.