513 results about "Spatial perception" patented technology

The invention provides a bridge girder erection machine intelligent construction control method and system based on machine vision, and relates to the technical field of intelligent control. Image information of a construction site is collected through a plurality of high-definition cameras arranged on a bridge girder erection machine; inputting the image information into a pre-trained deep learning model, identifying and positioning a target in the construction scene, and constructing a digital twinborn model of the construction scene in combination with a graph structure of a construction scene topological structure subjected to spatial perception graph convolution operation; the intelligent decision agent generates an initial construction strategy according to the current construction scene state, an improved Monte Carlo tree is adopted to search and optimize the initial construction strategy, and an optimal construction strategy is generated through multi-objective solution; and after the construction is completed, based on the digital twin model and the whole process data, analyzing the construction quality and efficiency by using a graph neural network, and generating a multi-dimensional evaluation report and optimization suggestions.

According to the medical image segmentation method and device based on spatial perception and frequency domain information, the precision and robustness of medical image segmentation are effectively improved by combining frequency domain information guidance and a multi-head state spatial perception technology. Frequency domain transformation is performed on a medical image, high-frequency and low-frequency components of the image are separated by using a multi-scale decomposition technology, and low-frequency features are extracted to obtain global information. By introducing a learnable noise filtering mechanism, noise and irrelevant background information in a frequency domain are suppressed, so that the model can be focused on a lesion area more accurately. A multi-head perception visual state space module is designed on a bottleneck layer, lesion features of different scales are captured through a multi-scale adaptive feature fusion mechanism, and the capability of segmenting small-size lesions and complex structures is enhanced. A context focusing attention mechanism is introduced into jump connection, fusion of global information and local details is further enhanced, and the accuracy of a segmentation result is ensured; and finally, recovering a high-resolution segmented image through a decoder.

The invention discloses an industrial equipment fault early warning analysis method and system based on a graph neural network, and relates to the technical field of intelligent fault diagnosis and equipment state monitoring, and the method comprises the steps: constructing a time slice data tensor, and forming a graph neural network input structure; on the basis of the graph neural network input structure, feature learning and correlation analysis of the graph neural network are executed, and node-level embedding representation is obtained; and based on the node-level embedding representation, calculating fault relevance response distribution of the nodes, evaluating propagation risks and positioning equipment fault sources. According to the method, the problem that a traditional method cannot effectively express a spatial structure or a cooperative relationship between measuring points in an early modeling stage is solved, the spatial perception capability of overall modeling is improved, so that a data foundation is laid for subsequent feature extraction and anomaly analysis, and the reconstruction accuracy is improved by comparing an embedded reconstruction value with an original observation value and calculating a node-level reconstruction error. Fault influence range evaluation and traceability positioning are realized, and quantitative support is provided for early warning response and maintenance decision.

The invention relates to the field of airline planning, and discloses a digital twin system for low-altitude airline planning and a construction method, and the method comprises the steps: obtaining a spatial perception data set of a preset sensor in a target airspace, carrying out the fusion of the spatial perception data set, obtaining the fusion perception data, obtaining the historical flight hidden danger data of the target airspace, and obtaining the historical flight hidden danger data of the target airspace; performing multi-layer convolution on the fusion perception data by using the historical flight hidden danger data to obtain a risk analysis result, performing fuzzy fusion on preset risk factors, generating a dynamic obstacle risk field according to the risk factors after fuzzy fusion and the risk analysis result, performing flight simulation on the dynamic obstacle risk field, and obtaining a flight simulation result; and collecting flight simulation real-time data in the flight simulation process, generating an air route space cost map according to the flight simulation real-time data, and performing optimal path search on the air route space cost map to obtain a low-altitude air route scheme. In the face of airspace actual risks, the searching efficiency of the optimal path in low-altitude route planning is improved.

The invention belongs to the technical field of computer vision and artificial intelligence, belongs to the cross technical field of target detection, deep learning and image processing, and particularly relates to a YOLOv8 algorithm improvement method based on an unmanned aerial vehicle aerial image small target detection model, which comprises the following steps of: introducing a user-defined feature enhancement module into a YOLOv8 backbone network, a neck part and a detection head part; the self-defined feature enhancement module comprises a context guide self-adaptive fusion module introduced into a backbone network so as to replace part of traditional convolution operation; a space edge sensing feature up-sampling module and a space sensing enhanced convolution module are adopted in the neck fusion network; a fine-grained dynamic pruning detection head is introduced into a detection head detection network. According to the method, the performance of the model in a small target detection scene is effectively enhanced, and the accuracy, robustness and real-time response capability of a detection system are remarkably improved.

The invention relates to the technical field of automatic dispensing, and discloses an automatic dispensing method and system based on machine vision, and the method comprises the steps: firstly, precisely obtaining the pixel coordinates of a preset Mark point on a workpiece through machine vision, and calculating a set of 2D deviation vectors of the preset Mark point and a Mark point at an ideal position; then, the set of the Mark point 2D deviation vectors is sent into a pre-trained Z-axis compensation prediction model; according to the model, a corresponding Z-axis deviation value can be learned and predicted from 2D plane deviation, and complex nonlinear mapping possibly existing between the 2D deviation and the Z-axis deviation is effectively solved. And finally, comprehensively utilizing the calculated 2D deviation vector and the Z-axis deviation value predicted by the model to carry out comprehensive three-dimensional adaptive adjustment on the original dispensing path, thereby overcoming the limitation that the traditional method only focuses on 2D compensation and the resulting problem of misalignment of linear and nonlinear spatial perception dimensions. And the accuracy and the adaptability of automatic dispensing are obviously improved.

The invention discloses an efficient open vocabulary-oriented panoramic segmentation method. The method comprises the following steps of S1, performing visual feature extraction and aggregation based on a multi-scale feature extractor and a lightweight aggregator; s2, encoding any category of vocabularies by using a text encoder to obtain text embedding # imgabs0 # S3, improving semantic understanding of visual aggregation features based on a vocabulary perception selection module, and reducing feature interaction burden of a mask decoder; s4, on the basis of bidirectional dynamic embedding experts, generating instance embedding with semantic perception and spatial perception by dynamically distributing expert weights; s5, based on a lightweight decoder, using the object kernel to perform mask prediction and refinement layer by layer, and using the object kernel and text embedding to perform dot product as category prediction; the method aims at reducing the model calculation overhead and increasing the reasoning speed while achieving considerable performance, and has remarkable practical value and application prospect.

The invention discloses a remote sensing image segmentation method fusing frequency modulation and spatial perception, and the method comprises the steps: obtaining and preprocessing an original remote sensing image, and generating a standardized input image; the image is input into a multi-scale frequency domain enhanced feature extraction network, features are extracted step by step according to a plurality of feature levels, each level realizes frequency adaptive semantic enhancement through frequency domain modulation transformation and spatial feature fusion, and deep feature expression is enhanced through feedforward neural network modeling and residual connection output and cross-level residual fusion introduction; the final multi-scale features are decoded through a decoding module, the spatial resolution is recovered, and a pixel-level segmentation result is generated; and constructing a composite loss function containing classification errors, boundary perception and frequency consistency items, and carrying out optimization training on the network. According to the method, semantic complementarity of a remote sensing image in a frequency domain and a space domain is fully mined, so that segmentation precision and robustness of a ground object target in a complex scene are improved, and the method has good generalization ability and engineering practicability.

The invention relates to the technical field of computer vision and image processing, in particular to a robust image matching system and method based on multi-scale attention feature fusion, and the system comprises a feature extraction module which extracts a multi-resolution feature map containing image features from a to-be-matched image; the position coding module performs spatial position coding on the image features to improve the spatial perception capability of the model; the multi-scale attention module performs multi-scale enhancement processing on the image features, so that the model is more effectively focused on the significant part of the image; the attention module establishes a dependency relationship between image features; the coarse-level matching layer module generates a coarse-level matching result and screens out a matching pair with relatively high confidence coefficient; and the coarse-to-fine module carries out fine processing on the coarse matching result, so that more accurate feature matching is realized. According to the method, the effectiveness and reliability of the method are verified by testing the image matching performance of the to-be-tested picture under the conditions that the to-be-tested picture contains different illumination changes, extreme visual angle changes, repeated textures and the like.

A system for three-dimensional spatial perception underwater that includes a primary submersible vehicle. Frontward facing projectors are used to scan beams of light across a selected field of underwater view. Further, event cameras with pixels triggered by events are used to detect corresponding spots of the scanned beams that are reflected from underwater objects or surfaces. Circuitry is configured to determine positions of the objects based on the pixels that are triggered. Additionally, pilot submersible vehicles the projectors and event cameras are tethered to the primary submersible vehicle to provide extend detection of spots corresponding to reflections from underwater objects or surfaces.

The invention relates to the technical field of civil engineering construction, in particular to a precise positioning civil engineering pile foundation construction system which comprises a multi-source space sensing and positioning module, a space-time multi-dimensional data fusion module, a three-dimensional guiding and correcting module, a hydraulic control module and a closed-loop feedback optimization module. The multi-source space sensing and positioning module obtains pile machine position and attitude information through the Beidou / GNSS high-precision positioning unit, the inertia high-precision navigation unit and the laser horizontal measurement unit, and the space-time multi-dimensional data fusion module processes and fuses the information to generate unified space position data. The three-dimensional guiding and correcting module compares the data with preset pile position coordinates, if the deviation exceeds a threshold value, a correcting instruction is generated, the hydraulic control module executes the instruction and adjusts the position and the posture of a pile machine, the closed-loop feedback optimization module collects the data in the correcting process and optimizes a correcting strategy, all-weather centimeter-level precision space positioning is achieved, and the positioning precision is improved. And the accuracy and efficiency of pile foundation construction are improved.

The invention discloses a VR teaching experience enhancement system and method, and belongs to the technical field of virtual teaching, and the method specifically comprises the steps: collecting the position data and posture data of a trainee in a VR environment, building a multi-user coordination interaction model based on the position data and posture data of the trainee in the VR environment, and carrying out the interaction of the multi-user coordination interaction model. The method comprises the following steps: acquiring training participants, identifying an interaction relationship and an interaction event among the training participants, generating tactile feedback data corresponding to the interaction event based on the interaction event, performing calibration processing on the tactile feedback data, and synchronously sending the calibrated tactile feedback data to VR equipment of the training participants, the calibration processing comprises adjusting the tactile feedback data based on the spatial propagation difference quantity and the tactile feedback time offset value of the trainees; according to the invention, when multiple trainees cooperatively operate the same virtual object or scene, consistent and real-time tactile response and spatial perception can be obtained, and the teaching experience in a multi-person cooperation scene is improved.

The invention relates to the field of traffic target detection, provides a real-time traffic target detection method and system based on adaptive attention perception, and aims to dynamically adjust the attention intensity and attention distribution of each attention head to different features by designing an adaptive scaling factor, enhance the attention to local features and improve the traffic target detection accuracy. The target detection accuracy is improved, a multi-scale offset perception attention module is designed to enhance the spatial perception capability of the sampling offset generation process so as to better pay attention to key areas and improve the target detection accuracy and robustness in a complex scene, and a bidirectional cross-scale feature fusion module is designed to improve the accuracy and robustness of target detection in a complex scene. Cross-layer connection is carried out between different feature levels, the feature fusion capability is enhanced to reduce the information loss of the features, meanwhile, a dynamic up-sampling operator is added, the information loss of the features is further reduced, the accuracy of target detection is improved, and the accuracy and robustness of the traffic target detection method are improved.

The invention discloses an aberration correction and image quality enhancement method for a laminated structure image, and the method comprises the steps: carrying out the deconvolution preprocessing of a to-be-detected marked image according to an aberration priori set, and obtaining an aberration-free image; meanwhile, combining label data to obtain a data set; feature extraction is carried out based on shallow convolution according to the data set, and global feature information is generated through activation function operation; enhancing the feature data by adopting a frequency domain feature and spatial domain feature fusion strategy; the enhanced feature map realizes initial aberration restoration through an aberration correction module; the corrected feature map is processed by a double-channel attention mechanism, and the global context modeling capability of the self-attention mechanism and the spatial perception characteristic of the position attention unit are fused in parallel; and the image resolution is improved through a super-resolution reconstruction module comprising a sub-pixel convolution layer. By adopting the technical scheme of the invention, the accuracy of overlay error detection is improved.

The invention discloses a digital twin interaction control system based on robot vision and tactile fusion, and the system comprises a data collection module which is used for collecting visual image data and tactile sensing data; the preprocessing module is used for preprocessing; the multi-mode perception modeling module is used for executing spatial perception modeling and tactile state modeling; the fusion state vector construction module is used for generating a fusion state vector; the digital twinborn mapping module is used for constructing a digital twinborn body; the simulation interaction control module is used for executing interactive action control and generating a control instruction parameter group; the action execution and feedback module is used for controlling the robot to execute the actual action and updating the fusion state vector; and the dynamic evaluation and self-adaptive adjustment module is used for dynamically optimizing and adjusting the fusion state vector. Visual and touch information of the robot is fused, a digital twinning synchronous control mechanism is constructed, and the interactive optimization method which is fast in operation response, high in execution precision and stable in control process is achieved.

The invention discloses a fruit tree disease and pest detection method and device based on improved YOLOv11, relates to the technical field of automatic machine learning, and aims to solve the problem that YOLOv11 in the prior art is still difficult in apple disease and pest detection. Taking the fruit tree shot image as input, and outputting based on the fruit tree pest detection model to obtain a fruit tree processing graph; and judging whether the fruit tree has diseases and pests and the types of the diseases and pests according to the fruit tree processing graph, wherein the fruit tree disease and pest detection model is obtained based on improvement of a YOLOv11 model. According to the method, advanced feature extraction, layered feature fusion and enhanced spatial perception functions are integrated, efficient and accurate detection of fruit tree diseases and insect pests is realized, the robustness and generalization ability of a detection model are improved, and meanwhile, relatively high performance is kept in a complex environment.

The invention discloses a road disease detection method based on an unmanned aerial vehicle, electronic equipment and a program product. The method is realized based on a trained road disease detection model, a C3k2-MDDSC module is introduced into a backbone network of the model, the feature multiplexing capability is enhanced through gradient shunting and multi-scale fusion, the gradient disappearance problem is relieved, and the robustness of the model is improved by means of jump connection and packet convolution. An ACFP module is introduced into the tail end of the backbone network, dynamic fusion of local and global features is realized by using multi-scale cavity convolution and a channel-space attention mechanism, and the complex scene modeling capability is improved. And the neck network is integrated with an SGF module, so that the spatial perception of the model to a tiny target can be improved. Besides, the ES-FPN proposed based on the SGF module not only can enhance the utilization of shallow spatial information, but also can optimize the complementarity of cross-level features. During training, regression loss, namely fast high-quality intersection-to-union ratio loss, is proposed, and angle punishment is introduced to improve the alignment precision of the rotating frame and the convergence speed of the model.

The invention provides a self-adaptive perception positioning method based on unsupervised fusion BEV, and the method comprises the following steps: collecting point cloud data of environment information nearby a vehicle in real time through employing a laser radar, and collecting image view data of the environment information nearby the vehicle in real time through employing a camera; the method comprises the steps of converting multi-view image view data and point cloud data into the same BEV space, respectively generating camera BEV features and point cloud BEV features, fusing the camera BEV features and the point cloud BEV features in a projection mode to obtain fused BEV features, performing semantic segmentation and instance segmentation on the fused BEV features, and generating a vectorized fused BEV map; learning query is introduced through query-based collaborative self-adaption, interaction between multi-view image view data and the fused BEV map and collaborative self-adaption of the multi-view image view data and the fused BEV map are achieved, and a spatial perception result is generated; and self-checking the spatial perception result, checking the rationality of the image, and completing the closed loop of the perception positioning process. According to the invention, the fusion BEV under unsupervised learning can rapidly detect and obtain the environment information, and autonomous verification is carried out, so that the accuracy of vehicle positioning is ensured, and the experience feeling of the vehicle is improved at the same time.

The invention provides a stack image enhancement processing method, and relates to the technical field of image enhancement, and the method comprises the steps: collecting and preprocessing a stack image, and constructing a training sample set containing a risk level and an inclination angle label; an encoder-classifier dual-stage model is constructed, an encoder extracts features through a multi-scale feature extraction and enhancement module, a space perception attention enhancement module, a dynamic feature pyramid fusion module and a deformation gradient enhancement residual module, and a classifier outputs a risk level and an inclination angle; during training, classification and regression loss weights are dynamically adjusted, an adversarial enhancement sample is generated to enhance robustness, and a gradient is adaptively cut based on a channel attention enhancement vector. According to the method, local illumination interference can be effectively eliminated, multi-scale features are fused, the sensitivity to a sparse risk area is enhanced, and the self-adaptive enhancement processing effect of the heaped image in a complex environment is improved.

The invention discloses a zero sample anomaly detection method and system based on a triple perception learning enhanced visual language model, and relates to the field of computer vision, and the method comprises the steps: extracting global and local visual features from an input image; in the visual coding process, local features in a deep network are corrected through a spatial perception attention enhancement module, and fine-grained attribute text description is generated for abnormal visual features; performing deep semantic alignment on the attribute text description and the general text prompt through an attribute perception guide module; calculating the similarity between the enhanced visual features and the optimized text features, and generating a pixel-level abnormal segmentation map; in the inference stage, the segmented image is converted into a space attention weight through an anomaly perception reconstruction module, the space attention weight is fed back to a visual encoder to generate final global feature representation, and an anomaly score is calculated. According to the method, under the condition that a target domain training sample is not needed, the anomaly detection and positioning accuracy and generalization ability of the model under the scenes of industrial defect detection and the like are remarkably improved.

The invention is applicable to the technical field of medical image processing and mixed reality, and provides a laparoscopic surgery mixed reality navigation method based on deep learning and dynamic point tracking, which comprises the following steps: dynamically registering a three-dimensional model containing kidney, tumor and vessel with an initial frame of a laparoscope through a mixed reality alignment technology; the method comprises the following steps: constructing an operating forceps motion sensing model based on a time sequence deep neural network, realizing real-time control and parameter locking of a three-dimensional model pose, dynamically updating a two-dimensional feature point set by adopting a multi-feature-point combined tracker, constructing a candidate feature combination through a cross-quadrant sampling strategy, and constructing an operating forceps motion sensing model; a candidate feature combination is generated through a four-quadrant division and cross-regional sampling strategy, an optimal camera pose parameter is generated in combination with a re-projection error and pose continuity constraint, and an operation video is dynamically covered with a semitransparent three-dimensional model. The method can significantly enhance the spatial perception capability of the kidney anatomical structure, reduce the registration error of the kidney in the three-dimensional integrated kidney structure model and the laparoscope video, and improve the navigation precision.

The present disclosure provides a method for calibrating a humanoid robot, comprising obtaining a humanoid robot with original kinematic biasing values, controlling the humanoid robot through predetermined poses, capturing image data of body parts using vision sensors mounted on the humanoid robot while moving through the poses, determining revised kinematic biasing values by processing the image data using a bipedal spatial perception model trained using synthetic image data containing keypoints, and replacing the original kinematic biasing values with the revised kinematic biasing values. The bipedal spatial perception model processes captured image data to generate observed keypoint locations on robot components, which are compared with kinematic-based locations from joint encoder measurements to minimize discrepancies through optimization algorithms.

The invention relates to the technical field of artificial intelligence, and particularly discloses a visual space description method, system and product based on depth information fusion, and the method comprises the steps: obtaining an image and a corresponding object text; performing vision-plane position feature extraction on the image to obtain a vision feature and a plane position feature, and performing depth feature extraction on the image to obtain a depth feature; performing text coding on the object text to obtain text features; performing feature fusion on the visual features, the plane position features, the depth features and the text features to obtain fusion features; performing spatial relationship classification on the fusion features to obtain a spatial relationship; and inputting the instruction language, the spatial relationship and the object text into a large language model to obtain a natural statement describing the spatial relationship of the object. According to the method, the spatial relationship between the objects in the image can be accurately described, the spatial perception capability of artificial intelligence is improved, and meanwhile, the increase of redundant manual annotation workload is avoided.

The invention provides a low-illumination track foreign matter intrusion detection method based on improved YOLOv8n, and the method comprises the steps: carrying out the contrast enhancement, gamma correction and image sharpening of a low-illumination image through an adaptive enhancement module, improving the quality of an input image, and improving the precision of subsequent target detection; an SPPF module is replaced with an RWSPPF module in the YOLOv8n model, and a spatial perception attention mechanism is introduced, so that the multi-scale adaptive capacity and the feature extraction effect of the model are enhanced, and the foreign matter detection accuracy is improved; and an improved loss function and an Adam optimizer are adopted, so that the training efficiency and precision are improved. By calculating the intersection-to-union ratio and the shortest distance of the foreign matter and the track area, the danger degree of the foreign matter is evaluated, safety alarm information is generated in real time, and railway traffic safety is guaranteed. On the basis of the detection method, the track foreign matter can be accurately recognized under the low-illumination condition, the detection efficiency and safety are improved, and the method has wide application prospects.

The invention relates to the technical field of alignment of photoetching machines, and discloses an alignment method, device and equipment for a photoetching machine and a storage medium. The method comprises the following steps: carrying out image acquisition and space registration on a plurality of small-size alignment marks preset on the surface of a wafer to obtain a virtual large mark image; performing double-light-source imaging on the target area of the virtual large-mark image by adopting an ultraviolet light source and an infrared light source respectively to obtain an ultraviolet light image group and an infrared light image group; performing multi-level fuzzy discrimination and spatial perception double attention processing on the ultraviolet light image group and the infrared light image group to obtain a multi-level optimization feature set; according to the multi-level optimization feature set, guiding filtering and self-adaptive parameter fusion are executed, an optimal alignment coordinate is obtained, and the photoetching machine is driven to execute alignment operation based on the optimal alignment coordinate, so that the optimal configuration of alignment parameters under different process conditions and environment interference is ensured, and the alignment accuracy is improved. And the alignment accuracy and reliability of the photoetching machine are improved.

The invention provides a method and device for realizing visual and auditory synchronous perception, and the method comprises the steps: obtaining the relative position information between a played video image and an audience and the type of a room where the audience is located under the condition that a video application is started; determining a volume adjustment gain based on the acquired relative position information and the type of the room where the audience is located; adjusting the sound to be played of the video application according to the volume adjustment gain; when the video is played, the relative orientation and distance between the image and the audience can be obtained, the type of the room can be judged, the audio signal to be played is dynamically adjusted according to the orientation and distance between the audience and the image and the type of the room, and the virtual sound image synchronously changing with the image is generated. The audience can obtain the same spatial perception in auditory sense and visual sense, and the immersion of video application is increased.

The invention discloses a robust point cloud registration method and system based on global spatial perception and multistage filtering. The method comprises the following steps: respectively obtaining point cloud initial features and spatial structure features of global spatial perception based on point cloud information; inputting the point cloud initial features and the spatial structure features into a transform network for feature enhancement, and obtaining enhanced features; and performing up-sampling processing and multi-stage filtering on the enhanced features to obtain a transformation matrix, and completing point cloud registration. According to the method, global spatial perception and multi-stage filtering are introduced, so that abnormal values can be effectively eliminated while feature matching is completed finally, and the precision and efficiency of point cloud registration are improved.

The invention discloses a traffic flow prediction method based on multi-scale dual hypergraph fusion. Aiming at the problem that an existing traffic flow prediction model is difficult to capture a multi-scale high-order spatial dependency relationship of a traffic road network, the method comprises the following steps of: firstly, constructing an urban road dual hypergraph with three scales of microscopic individual travel intention, mesoscopic community commuting interaction and macroscopic area flow conduction, and then designing a traffic flow prediction method of space-time perception based on the urban road dual hypergraph. Wherein the spatial perception module extracts and fuses a dependency relationship between a high-order local spatial feature and a global spatial feature in a multi-scale dual hypergraph; the time sensing module captures short-term fluctuation and long-term trend of traffic flow; and residual connection is introduced to enhance spatio-temporal feature fusion, and finally multi-step traffic flow prediction is realized. The invention provides a traffic flow prediction model with multi-scale high-order space perception, which adaptively fuses high-order space features of three scales of microcosmic, mesoscopic and macroscopic, and significantly improves the precision and generalization ability of traffic flow prediction.

The invention discloses an agent action prediction method based on multi-scale spatial perception, which comprises the steps of map spatial structure modeling, historical trajectory feature extraction, feature fusion and multi-modal action prediction. According to the map space structure, map feature extraction is conducted on two-dimensional vector map data in an application scene map through a multi-scale map convolutional neural network, and high-dimensional map feature information is obtained; the historical trajectory feature extraction adopts a convolutional neural network and a feature pyramid network to extract high-dimensional trajectory data features of all agents; according to feature fusion, correlation modeling fusion is carried out on high-dimensional map feature information and high-dimensional trajectory data feature information through a self-attention mechanism, and agent trajectory fusion features with direction information are obtained; according to the multi-modal trajectory prediction, intelligent agent trajectory fusion features are used for prediction regression and confidence scoring, multi-modal complete trajectory coordinates of action prediction and corresponding confidence scores are output, and a reasonable auxiliary decision is provided for intelligent agent actions.

The invention discloses a visual language action large model design method for enhancing spatial perception ability, which comprises the following steps of: acquiring a plurality of frames of task scene images, inputting the images into a two-dimensional image encoder and a visual geometry guidance Transformer VGGT encoder, extracting image features and performing time sequence spatial embedding, encoding a natural language instruction into language embedding representation, and extracting a visual language action large model. Image features and language embedding are fused through a cross attention mechanism, a pre-training large model is input to generate cross-modal representation, the fusion features are input into an action expert module to output an action control sequence in combination with robot body state information, and a robot is driven to execute an operation task. Therefore, damage of additional information to an original pre-training model is avoided, and compared with an original combined structure of a pre-training visual language model and an action expert, utilization of multi-view picture information is enhanced, so that higher understanding ability on space depth is achieved in the task execution process, the task success rate is increased, and the task execution efficiency is improved. And a more efficient and more robust robot sensing and decision-making integrated system is realized.