LLM Environment Reconstruction via Perception-Map Fusion

Solution Overview

Problem

Existing systems for autonomous navigation and mapping face challenges in generating accurate and consistent digital environments, particularly when map and perception data are incomplete, inaccurate, or unavailable.

Innovation Solution

The use of large language models (LLMs) to generate tokenized representations of environments by fusing perception and map data, allowing for the creation of consistent and accurate digital environments even in the absence of complete data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If perception data is used alone for environment reconstruction, then real-time detection capability is maintained, but accuracy and completeness deteriorate when map data is unavailable

Engineering Contradiction:
Core Design Contradiction:
ReliabilityVSDevice complexity

Solution Approach 1:

The patent combines map data and perception data into a unified environment reconstruction framework. The language model processes both data types together, fusing their complementary strengths: map data provides accurate static structure information while perception data provides real-time dynamic information, achieving reliable reconstruction without overloading the perception module alone

Inventive Principle:
Principle #5Merging (Combining)

2Ease of operation

If map data and perception data are processed separately, then module independence is maintained, but consistency between map and perception results deteriorates

Engineering Contradiction:
Core Design Contradiction:
Ease of operationVSStability of the object's composition

Solution Approach 1:

The patent introduces a language model as an intermediary that receives both map data and perception data as inputs and generates a unified tokenized environment description. This mediator translates both data types into a common semantic representation, ensuring consistency while allowing the modules to remain independently operable

Inventive Principle:
Principle #24Intermediary (Mediator)

3Measurement precision

If high accuracy perception is required for all driving objects, then detection precision is improved, but processing time and computational load increase

Engineering Contradiction:
Core Design Contradiction:
Measurement precisionVSLoss of time

Solution Approach 1:

The patent applies partial action by using map data to provide accurate information about static environment elements (lanes, curbs, signs) that would otherwise require intensive perception processing. This allows the perception module to focus computational resources on dynamic objects, achieving high overall accuracy while reducing processing time and latency

Inventive Principle:
Principle #16Partial or excessive action

Data Source

PatentUS20250200283A1Using large language models to augment perception data in environment reconstruction systems and applications
Publication Date: 2025.06.19
  • US20250200283A1 patent drawing
  • US20250200283A1 patent drawing
  • US20250200283A1 patent drawing

AI summary

Approaches presented herein provide for the use of language models to generate tokenized descriptions of physical environments. In at least one embodiment, sensor and/or observational data can be obtained for an environment and used to generate a set of perception data. The perception data can be analyzed, along with approximate positional data within the environment, to identify a set of aligned map data. The aligned map data and perception data can be provided as input to a trained language model, which can be trained to correlate and/or fuse the information to generate a single, consistent representation of the environment. The language model can output a tokenized description of the environment, which can be in a domain-specific language, that is a compact but robust textual description of the environment.