The invention discloses a
robot path navigation method based on multi-
modal perception, relates to the technical field of
intelligent robots, and can solve the problems of insufficient utilization of visual
semantic information, limited
perception capability of a three-dimensional geometric structure and difficulty in migration from
simulation to reality in the prior art. The method comprises the steps of collecting multi-source
original data and generating three-dimensional
Gaussian splashing visual representation and voxelization
geometric representation; constructing a high-
throughput simulation training framework, and integrating the three-dimensional
Gaussian splash model as a plug-and-play renderer to a vectorization physical simulator; extracting visual semantic features through a pre-trained visual
encoder, and extracting three-dimensional geometric features through a grouped
convolutional neural network; fusing the visual semantic features, the three-dimensional geometric features and the ontology
perception features through a
recurrent neural network to generate potential representation; and training the navigation strategy network based on the potential representation and migrating deployment. According to the invention, efficient semantic perception navigation and steady
simulation-to-reality migration of the
robot in a complex environment are realized.