AV State Fusion Checkpoints for Edge Latency Masking

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Network latency in wireless edge networks and cloud networks poses challenges for mission-critical vehicle functions like autonomous driving, as the round trip time latency can be significant, making real-time processing of sensor data unreliable.

Innovation Solution

A latency masking system for autonomous vehicles that includes a sensors module, wireless transceiver module, on-board function module, and state fusion and prediction module, which uses a state history data structure with checkpoints to update local state information with remote state information, compensating for latency by predicting and fusing sensor data from both on-board and remote sources.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If sensor data is transmitted to a remote server for processing, then computing resources are leveraged to enhance vehicle functions, but network latency makes the results invalid for real-time mission critical functions

Engineering Contradiction:

Improvecomputing resource leverageVSAvoidnetwork latency

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by maintaining a state history data structure with checkpoints that record previous sensor data and processing results. When remote processing is initiated, the system has already prepared historical state information that can be used to mask the latency delay, allowing the vehicle to continue operations while waiting for remote server responses.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The state history data structure acts as an intermediary between the remote server and the vehicle's real-time control systems. It stores checkpoints of sensor data and processing results, enabling the system to retrieve and use historical state information to compensate for delays in remote server responses, thus masking the network latency effect.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If remote server processing is used, then vehicle functions are enhanced, but the round trip time latency of 70-90 milliseconds or longer makes real-time processing unreliable

Engineering Contradiction:

Improvevehicle function enhancementVSAvoidreal-time processing reliability

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system performs preliminary processing and stores results in the state history data structure with checkpoints. This allows the vehicle to use pre-computed information from historical checkpoints to maintain reliable real-time processing while still benefiting from enhanced remote server capabilities for non-critical functions.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The state history data structure provides a cushion of historical state information that compensates for the 70-90 millisecond or longer round trip latency. By having pre-stored checkpoints of sensor data and processing results, the system can mask the delay and maintain reliable real-time processing for mission critical functions.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

3Loss of information

If sensor data is continuously transmitted to the remote server, then remote state information is derived, but the latency makes the results invalid for autonomous driving perception functions

Engineering Contradiction:

Improveremote state information derivationVSAvoidlatency making results invalid

Core Design Contradiction:

Loss of informationVSLoss of time

Solution Approach 1:

The state history data structure serves as an intermediary that stores checkpoints of sensor data and processing results. When remote server processing completes with latency, the system can retrieve corresponding historical state information from the data structure, matching the delayed remote results with appropriate historical checkpoints to restore validity for autonomous driving perception functions.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system performs preliminary storage of sensor data and processing results in the state history data structure with checkpoints before the delayed remote results arrive. This preliminary action allows the system to have the necessary historical information ready to compensate for the latency when remote results are finally received.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11516636B2Latency masking in an autonomous vehicle using edge network computing resources

Publication Date: 2022.11.29 GM GLOBAL TECHNOLOGY OPERATIONS LLC

AI summary

A latency masking system for use in an autonomous vehicle (AV) system includes a sensors module providing sensor data from a plurality of sensors. The sensor data includes image frames provided by a vehicle camera and vehicle motion data. A wireless transceiver transmits the sensor data to a remote server associated with a network infrastructure and receives remote state information derived from the sensor data. An on-board function module receives the sensor data from the sensors module and generates local state information. A state fusion and prediction module receives the remote station information and the local state information and updates the local state information with the remote state information. The state fusion and prediction module uses checkpoints in a state history data structure to update the local state information with the remote state information.