100 results about "Robot perception" patented technology

Provided is a method for token-based representation and learning of robotic perception data based on a graph neural network, comprising: obtaining a plurality of types of perception data of a robot; performing token-based representation according to types of the plurality of types of perception data; constructing an initial feature graph based on the plurality of types of perception data after the token-based representation; learning a compact representation of the initial feature graph based on an autoencoder and reconstructing a graph structure; after the autoencoder completes learning of the graph structure, fixing the graph structure; and converting the plurality of types of perception data into node feature vectors, constructing a feature graph based on the graph structure, and performing numerical encoding on each of the node feature vectors by utilizing the graph neural network to obtain a representation of high-dimensional feature vectors of the plurality of types of perception data.

The invention discloses a non-modal segmentation method and system based on RGB-D and a 6D object grabbing method and system, and relates to the technical field of computer vision and robot perception. According to the method, RGB and depth modal features are extracted through a double-branch backbone, and cross-channel image features are output; channel alignment and gating selection are carried out through a mutual attention fusion module; and finally, generating four branches of occlusion object segmentation, visible area segmentation, complete non-modal mask complementation and occlusion relation classification through a visible part guide non-modal multi-branch decoding module, and generating a complete target mask with consistent semantics. Three-dimensional geometric information of a target object is extracted based on a complete non-modal mask, a 6D grabbing posture is output through a posture estimation network and converted into a mechanical arm motion instruction to complete grabbing, the problems of serious shielding, sensor noise and cross-modal dislocation can be effectively solved, the non-modal instance segmentation precision is improved, and the grabbing efficiency is improved. The method can be widely applied to high-precision robot operation scenes such as autonomous storage and industrial operation.

The invention discloses an SLAM-oriented multi-sensor space-time synchronization and online calibration method, and relates to the technical field of computer vision and robot perception. Comprising the steps that S100, initialization is carried out, and an initial pose and an external parameter transformation matrix initial value are obtained through coarse alignment; s200, data acquisition and preprocessing: acquiring IMU data, LiDAR data and Odom data in real time, and performing data filtering, distortion removal, time sequence correction and point cloud downsampling processing on the acquired data; and S300, performing time synchronization processing, estimating the time offset of the LiDAR and Odom data in real time, and performing unified correction to generate a unified timestamp. According to the method, an online time offset estimation and compensation mechanism with an IMU as a main clock is established, accurate synchronization of sensor data of different frequencies is achieved, and the problem of data alignment caused by sampling frequency difference and clock drift is effectively solved; and meanwhile, tight coupling fusion is realized by using an IMU pre-integration factor, a LiDAR residual factor and an Odom constraint factor, so that the space-time consistency of multi-sensor data is remarkably improved.

The invention relates to the technical field of robot perception and data generation technologies, in particular to a first-view-angle drilling method and device based on memory enhancement and a storage medium, and the method comprises the steps: extracting a plurality of target frames from a historical video stream stored in a space intelligent machine, obtaining a plurality of memory elements based on the plurality of target frames, performing three-dimensional reconstruction on each target frame, constructing a target world model, planning a first visual angle track of the target robot in the target world model, performing imaging simulation on the target robot, performing real-time rendering on a first visual angle video of the target robot, and performing real-time rendering on a second visual angle video of the target robot based on the same time axis and action script as the first visual angle video. A third-person video corresponding to the space intelligent machine is generated, and a drilling result is output; according to the method, the long-term memory data of the space intelligent machine is ingeniously used, high-quality drilling data can be quickly generated, the reliability of the first-view video is improved, and a reliable basis is provided for training and testing of a robot algorithm.

The invention belongs to the technical field of robot perception and decision making, and discloses a multi-modal fused intelligent target perception method and system, and the method comprises the steps: obtaining multi-modal data in a distribution network operation scene, carrying out the preprocessing, fusing the knowledge of the distribution network operation field, and generating knowledge-enhanced multi-modal features; carrying out cross-modal alignment processing on the knowledge-enhanced multi-modal features, carrying out graph structure-based joint semantic and space alignment on isomorphic modals, and carrying out feature projection-based binding alignment on heterogeneous modals to obtain consistent aligned multi-modal features in a shared semantic space; based on a dynamic adaptive strategy, screening and fusing the aligned multi-modal features to generate unified fusion features; and inputting the fusion features into a target perception model, and outputting an image segmentation result and a point cloud semantic segmentation result of the distribution network operation target. According to the method, high-precision segmentation and positioning of the target in a complex distribution network scene are realized, and safe and efficient operation of the robot is effectively supported.

The embodiment of the invention provides a robot and an operation method and device thereof, a storage medium and a program product. The method comprises the steps that natural language instruction information and multiple robot sensing data are received; performing spatial feature extraction processing based on at least part of the robot sensing data to obtain spatial feature information of the corresponding robot sensing data; performing time sequence feature fusion based on the spatial feature information of the perception data of the at least two robots to generate space-time semantic implicit representation information; performing cross-modal interaction based on the natural language instruction information and the space-time semantic implicit representation information to generate space target representation information; and path planning and / or action control are / is carried out based on the spatial target representation information, so that the robot is controlled to execute operation corresponding to the natural language instruction information. Therefore, high information integrity can be kept under the condition of low computing power, serious information loss caused by white list detection is reduced, and the universality of robot operation control is remarkably improved.

The invention discloses a robot data analysis decision method and system based on a neural network, and relates to the technical field of robot production, and the method comprises the steps: obtaining the state data of a local robot; acquiring state data of other local robots, and selecting the robot with the highest comprehensive score as an auxiliary robot; collecting local data and auxiliary data; performing modal noise reduction, feature extraction and fusion on the local data and the auxiliary data by using the neural network preprocessing layer to obtain an environment representation vector; determining a bottom risk by using a Bayesian network; analyzing a risk evolution trend by using a knowledge graph; and inputting the underlying risk and the risk evolution trend into the reinforcement learning model, and outputting an optimal decision. According to the method, the deficiency of single-robot sensing is made up through multi-robot cooperative data acquisition, noise reduction and deep feature extraction of interfered data are realized by means of neural network preprocessing, and the environment sensing accuracy, risk assessment precision and decision reliability of the robot in the interference environment are improved.

The invention discloses a cross-scene self-adaption oriented on-line optimization method for perception parameters of a body-equipped robot, and relates to the technical field of robot perception control. The method comprises the steps of obtaining a historical scene type and environment data of the robot with the body, identifying a current scene type, and if the scene types are the same, collecting voice data and calling a special environment analysis model to determine current environment data; and if not, directly matching standard sensing parameters, calculating an environment change degree based on historical and current environment data, and constructing a sensing parameter optimization strategy, including determining an optimization depth and a parameter adjustment space, and finally performing iterative optimization by taking a minimum sensing deviation as a target, evaluating parameter performance by using an integrated prediction plug-in, and outputting an optimal adaptive parameter. According to the method, adaptive switching of the sensing parameters among different scenes and dynamic optimization of the sensing parameters in the same scene are realized, and the sensing robustness and task execution reliability of the robot in a complex environment are improved.

The invention discloses a robot perception enhancement method based on a vision-touch fusion codec, and belongs to the technical field of robot perception and artificial intelligence. Aiming at the problems that a single visual perception scheme cannot directly observe contact information, the robustness is poor under environment change and multi-modal fusion is difficult, the method is characterized by comprising the following steps: synchronously collecting visual and tactile data, and carrying out time sequence alignment and preprocessing; respectively converting the image blocks and the tactile signals into feature sequences and adding position codes; feature interaction is carried out through cross-modal attention in a unified encoder, and training is carried out by adopting a joint random masking and reconstruction mechanism to obtain shared representation; the shared characterization is used for a downstream robotic operation task. According to the method, the observability of the contact behavior is realized, the perception robustness under shielding and illumination changes is improved, and more stable perception input is provided for a smart operation task.

The invention discloses a self-supervised monocular depth estimation method based on an enhanced multi-scale pose network, and relates to the field of computer vision. The problems that in an existing self-supervision monocular depth estimation method, the pose network structure is simple, the time sequence modeling capacity is insufficient, and geometric constraints are missing are solved. According to the method, a self-supervision joint training framework composed of a depth estimation sub-network and an enhanced pose estimation sub-network is constructed; wherein the pose estimation sub-network extracts multi-scale spatial structure features through a layer-by-layer feature fusion encoder, and adopts a context fusion decoder based on time sequence attention to model a motion dependency relationship between continuous frames; meanwhile, a self-supervised pose consistency loss function is introduced, geometric continuity of a camera track is enhanced through forward and reverse transformation consistency constraint and closed-loop geometric constraint, and collaborative optimization of depth prediction and pose estimation is realized. The method is also suitable for the application fields of automatic driving, robot perception, augmented reality and the like.

The invention discloses a real-time AR data superposition and interaction system for an industrial inspection robot, and relates to the technical field of industrial man-machine interaction, and the system comprises the steps: collecting multi-dimensional inspection context data obtained by a sensor carried by the inspection robot, and building a unified space reference system; updating of the AR terminal in the industrial scene is completed; and the corresponding control command is issued to the inspection robot for execution. According to the invention, high-precision, low-delay and semantic-consistent closed-loop coordination between data sensed by the inspection robot and AR terminal display is realized, and the information presentation intuition, the man-machine coordination fluency and the field decision accuracy in the industrial inspection process are improved; the defects of interaction lag, virtual-real dislocation, operation interruption and the like existing in a traditional system are effectively overcome, and the urgent requirement for efficient and immersive man-machine interaction equipment in a high-risk complex industrial scene is met.

The invention discloses a three-dimensional occupancy perception method and system suitable for multiple tasks, and belongs to the field of robot perception, and the method comprises the steps: carrying out the manual labeling of point cloud data and image data, obtaining a point-by-point label and a pixel-by-pixel label, carrying out the data verification, converting the point cloud data with the labels into a 3D semantic occupancy label of a scene through 3D semantic reconstruction, and carrying out the recognition of the 3D semantic occupancy label. Forming sample data; training a three-dimensional occupancy perception model by using the sample data and performing three-dimensional occupancy perception prediction; sequentially performing feature enhancement extraction, feature dimension conversion, feature time sequence fusion and feature compensation correction on the image data based on a feature extraction module to obtain multi-scale voxel features; the feature information of the multi-scale voxel features is independently adjusted through a semantic segmentation head and a target detection head, then a 3D semantic occupancy result and an instance detection result are output, the two detection results are converted into an instance segmentation result and a target tracking result through a post-processing module, and low-cost and high-efficiency multi-task three-dimensional occupancy perception is achieved.

The application discloses a robot complex scene perception system and method based on environment adaptive multi-modal fusion, relates to the technical field of robot perception, and comprises four modules of a flexible sensing matrix, environment feature self-calibration, cross-modal causal reasoning and scene prediction.The flexible sensing matrix collects environment parameters and multi-modal data of vision, inertia and touch.The self-calibration module corrects data errors through environment-error mapping and gradient descent algorithm.The causal reasoning module realizes data fusion and failure completion based on a Bayesian network.The scene prediction module outputs a prediction result through a pre-trained LSTM model and feeds back an optimized collection strategy.The application breaks through the bottleneck of traditional perception technology, improves the environment adaptability, system fault tolerance and decision initiative of complex unstructured scenes, is suitable for scenes such as disaster rescue and outdoor inspection, and helps robots to work autonomously and safely.

The application provides a three-dimensional map construction and camera trajectory estimation method based on edge information fusion, and belongs to the field of computer vision and robot perception. An image sequence in a scene is collected, and a two-dimensional edge intensity map is generated after edge detection; a three-dimensional Gaussian point cloud is initialized or updated: an initial point cloud is generated by back projection of the color, depth and edge intensity value of the first frame; a point cloud is added in the required area according to the rendering difference in subsequent frames; the camera pose of the current frame is initialized, a prediction map is generated based on the pose and the point cloud of the previous frame, a loss function is constructed to optimize the camera pose; the point cloud attributes are optimized again based on the optimized pose and the point cloud of the previous frame, and the point cloud of the current frame is generated; the sequence is processed frame by frame, the optimized point cloud is accumulated to form a global three-dimensional map, and the camera motion trajectory is output by accumulating the camera pose. The application improves the SLAM precision, stability and reconstruction quality in various complex scenes, especially in low-texture and strong-structure environments.

The invention discloses a semantic map construction method based on a camera-laser radar, and belongs to the technical field of robot perception and autonomous navigation, and the method comprises the steps: carrying out the hardware configuration, so as to achieve the perception and time synchronization of the camera and the laser radar; performing sensor calibration on the camera and the laser radar; constructing a local overlapping frame point cloud according to the camera image and the laser radar point cloud; open set 2D semantic extraction of an open word list is realized under single frame and time sequence joint constraint; and carrying out 2D-3D assignment and cross-frame instance fusion to construct a semantic map. In order to solve the problems that in the prior art, geometric advantages of a laser radar are not fully utilized, open set targets of indoor intelligent scenes are difficult to adapt, and the like, the invention provides an overall scheme capable of realizing engineering landing. The method has the advantages that the open set semantic coverage capacity is high, 2D-3D assignment noise is low, cross-frame fusion is stable, semantic retrieval and interaction are more efficient, and semantic robustness is improved under complex working conditions.

The present application relates to the field of ocean engineering and intelligent detection technology, specifically to a kind of multi-robot collaborative underwater oil and gas pipeline weld intelligent detection method and system, it includes obtaining and fusing each robot perception and task state data, assesses communication and perception ability, predicts task load trend and optimizes scheduling, adjusts multimodal fusion priority, re-plans task path and optimizes execution strategy, generates collaborative detection feedback data.The present application improves the collaborative ability of multi-robot in communication limited and ocean current disturbance environment through the dynamic task scheduling mechanism of global collaborative state perception;Combined with the time sequence alignment of multimodal sensing data and the dynamic adjustment of fusion priority, the stable recognition ability in complex underwater environment is enhanced;Through the joint optimization of task path planning and data fusion strategy, the detection coverage and defect recognition accuracy are improved, and the adaptability and reliability of the system in deepwater long-distance pipeline weld detection are enhanced.

The invention relates to the technical field of robot sensing and intelligent, in particular to a robot self-adaptive sensing method and system. The method comprises the following steps: in response to a task instruction, analyzing to obtain a perception modal demand corresponding to task semantics and an execution stage; dynamically adjusting configuration strategies of various modal sensors in a task execution process according to the sensing modal requirements; monitoring the confidence of each sensor data stream in real time; and when it is monitored that the confidence coefficient of the main sensor data flow is lower than a first threshold value, dynamically adjusting fusion priorities of different sensor data in the perception decision. According to the method, fine scheduling of sensing resources is realized, the environment robustness of a robot sensing system is greatly improved, and meanwhile, the decision accuracy and generalization ability are effectively improved.

The application discloses a kind of collaborative robot system and service process for active power grid marketing, belong to artificial intelligence and electric power marketing service technical field.System includes: robot perception interaction end, cloud intelligent analysis hub and dynamic customer memory bank, robot perception interaction end deploys lightweight vision model, real-time identification customer emotion, body state and explicit feature, and initiatively trigger service;Cloud intelligent analysis hub uses power grid knowledge enhanced large language model, carries out depth demand analysis and individualized strategy generation to multi-modal data;Dynamic customer memory bank stores customer historical interaction record and portrait information in the form of knowledge graph, realizes the continuity and memory of service.The application is through edge small model+cloud big model collaborative architecture, general AI capability and power grid business knowledge are deeply fused, so that robot can actively identify customer demand, provide marketing service, improve customer experience and marketing conversion rate of electric power business hall.

This invention discloses a robot sensor calibration method, apparatus, robot, and storage medium. The method includes: determining the transformation relationship between the calibration board coordinate system and the radar coordinate system based on geometric similarity, according to the scanning data of the robot's radar sensor on the calibration board and the attribute information of the calibration board; determining the second position of the target label in the radar coordinate system based on the first position of the target label in the calibration board coordinate system and the transformation relationship; determining the third position of the target label in the image coordinate system based on the image data collected by the robot's image sensor on the calibration board; and calibrating the image sensor and the radar sensor based on the second and third positions. The technical solution of this invention can determine a more accurate transformation relationship between the calibration board coordinate system and the radar coordinate system, improving the performance of the robot's perception system and providing a new scheme for robot sensor calibration.

The application discloses a kind of fusion ToF and monocular vision's robot joint layer perception system, belong to robot perception field, system uses Eye-in-Hand architecture, integrates ToF camera and RGB camera;Through joint bilateral filtering and Kalman filtering, improve depth map quality;Based on SURF feature matching and the ICP registration of KD-Tree acceleration is realized target six degree of freedom pose estimation;Combined with improved RRT algorithm and hierarchical collision detection planning capture path.The robot joint layer perception system of fusion ToF and monocular vision provided in the application, through the space alignment and fusion of two kinds of visual information, make up for respective limitations, improve the robustness and precision of overall perception system;In single-arm and double-arm capture experiment, system positioning accuracy reaches ±3mm, dynamic scene capture success rate is high.It is suitable for industrial automation, logistics sorting and the like.

The invention relates to the technical field of robot model deployment and heterogeneous computing system architecture, in particular to a robot sensing-reasoning inter-node communication system and method based on a page lock memory. The system comprises a sensing node, a shared memory management module and a reasoning node. And the shared memory management module is used for constructing and managing a page lock memory pool and mapping the page lock memory pool to process address spaces of the sensing node and the reasoning node. And the sensing node is used for acquiring the memory block from the pool and directly writing the acquired sensor data into the pool to realize zero-transfer writing. And the reasoning node is used for asynchronously loading the data to the graphics processor video memory through direct memory access operation according to the address information when the data of the target memory block is ready. According to the invention, high-bandwidth, low-delay and high-determinacy data transmission without redundant copy between robot sensing-reasoning nodes is realized.

The invention relates to the technical field of robot sensing, and discloses a micro-texture regulation and control method of an anodic alumina-based robot tactile sensor and the tactile sensor, and the tactile sensor comprises an induction sensing unit, a reference sensing unit and a signal analysis unit. The method comprises the following steps: acquiring an original capacitance change signal generated by the induction sensing unit due to pressure and an original reference capacitance signal of the reference sensing unit; performing differential preprocessing on the original capacitance change signal based on the original reference capacitance signal to obtain a net capacitance signal representing pure pressure response; extracting dynamic response characteristics from the net capacitance signal in real time; inputting the net capacitance signal into a preset pressure-capacitance mapping model to obtain an initial pressure estimation value; and generating a real-time correction value for the initial pressure estimation value according to the dynamic response characteristics, correcting the initial pressure estimation value based on the correction value, and outputting a pressure sensing result. According to the invention, the effect of accurately outputting the pressure result according to the micro texture features of the tactile sensor is realized.

The invention discloses a camera / laser radar joint calibration method based on vanishing point constraint, and belongs to the technical field of robot perception and computer vision, and the method comprises the following steps: S1, obtaining a circle center coordinate and an orthogonal vanishing point in a camera coordinate system, and a circle center coordinate and a direction vector of an orthogonal straight line in a laser radar coordinate system; s2, determining a rotation matrix from a camera coordinate system to a laser radar coordinate system; and S3, estimating a translation vector from the camera coordinate system to the laser radar coordinate system. According to the camera / laser radar joint calibration method based on vanishing point constraint, through a rotation-translation decoupling form, direction information of strong constraint is provided by using an orthogonal groove and an outer frame, and rotation is preferentially locked through vanishing point and straight line fitting, so that the influence of a circle center error on rotation is significantly weakened, and the calibration precision is improved. And the overall precision and stability are improved.

The embodiment of the application provides a kind of ground feature data measurement method, device, electronic equipment and storage medium, belong to robot perception technical field.The method is applied to the robot tactile perception system based on vision, including elastomer, the outer surface of elastomer is in contact with ground, and the inner surface of elastomer is provided with mark array;Method includes: obtaining the motion image of elastomer based on mark array, obtains target image data;Image feature extraction is carried out to target image data, and motion image feature data is obtained;Displacement field construction is carried out based on motion image feature data, and three-dimensional displacement data is obtained;Friction evaluation and elasticity evaluation are carried out to ground based on three-dimensional displacement data, and friction feature data and elasticity feature data are obtained;Characteristic information of ground is generated based on friction feature data and elasticity feature data, and target ground feature data is obtained.The embodiment of the application can obtain rich ground feature data, and improve the measurement accuracy of ground feature data.

The application relates to the field of intelligent agricultural machinery and automatic driving technology and discloses a livestock industry feed pushing robot feed pushing control system based on deep learning. The system comprises a perception layer, a decision layer and an execution layer. The perception layer fuses a laser radar, ultrasonic waves and a camera; an edge computing unit runs an instance segmentation model to fit a feed edge; the decision layer utilizes an extended Kalman filtering algorithm containing a dynamic adjustment mechanism of an observation noise covariance matrix, adjusts a weight in real time according to a satellite signal quality, realizes seamless navigation indoors and outdoors, and executes a double closed loop PID strategy of a visual position loop and a torque limiting loop, and adaptively adjusts a pushing plate and a vehicle speed according to visual feedback and motor current. The application solves the problems of weak perception ability of the feed pushing robot, unstable navigation switching and poor operation flexibility, and realizes precise and efficient unmanned feed pushing.

The invention discloses a robot teleoperation method based on fusion of virtual reality and mixed reality. The robot teleoperation method comprises the following steps: sensing a far-end environment by observing a robot; constructing a near-end interactive virtual environment, and presenting the virtual environment to a near-end operator through a virtual reality device; constructing three-dimensional point cloud data based on a far-end environment through three-dimensional reconstruction; in the virtual environment, pose estimation is carried out on the far-end operation robot and the target object, and three-dimensional registration is carried out in the virtual environment; working space boundary condition constraints and space operation pose change constraints of the working robot are determined, path planning and trajectory planning are conducted on the working robot, and visualization is conducted in the virtual environment; and generating a control program of the operation robot to move along the planned track. By constructing the interactive virtual environment, the working space range and the operation pose of the operation robot are visualized in the interactive virtual environment, and the trajectory is planned and visualized in the interactive virtual environment, so that the safety of teleoperation is greatly improved.

The present application relates to the technical field of robot autonomous exploration, and particularly relates to a method for air-ground heterogeneous robot cooperative exploration based on a topological graph and hierarchical planning, which comprises the following steps: a robot explores to obtain exploration environment information; a ground base station receives the exploration environment information, divides the exploration environment into a plurality of map blocks, and extracts a total set of unknown regions and a total set of boundaries of the map blocks; a topological graph is constructed according to the total set of unknown regions and the total set of boundaries; the ground base station sends the topological graph to the robot, the robot autonomously plans a route, the robot updates the topological graph in real time, and the route planning and the topological graph are returned to the ground base station; the above steps are repeated until the robot exploration is completed. Through the proposed air-ground heterogeneous robot cooperative autonomous exploration system based on a topological graph and hierarchical planning, the fusion of air-ground robot perception and exploration information can be effectively realized, and the efficiency of air-ground robot cooperative autonomous exploration is improved.

The application discloses an indoor-oriented three-dimensional Gaussian diffusion model point cloud repairing method, expresses a missing point cloud as a three-dimensional point set, introduces a point cloud centroid as a spatial reference center, constructs a progressive-axis distance component of each point relative to the centroid, and calculates a second-order statistic in each axis direction as a directional scale signal; according to the deviation of the directional scale signal from a reference scale, the noise intensity is re-calibrated in each axis direction to form a diagonal covariance form of a directional adaptive three-dimensional Gaussian forward diffusion model; a denoising network is used to predict noise components in each diffusion step and update according to the progressive-axis diffusion intensity, while an observation consistency constraint is introduced; finally, a diffusion standard loss function is used to train the error between the real noise injected in the forward diffusion model and the predicted noise, and an optimized diffusion model is obtained. The method can be applied to indoor three-dimensional reconstruction, digital twinning, robot perception and indoor point cloud dataset enhancement scenes.

The application discloses a kind of based on double robot perception positioning's square tube fence variable parameter welding system and method, belong to welding automation technical field, it includes center control system, conveying and variable position mechanism, double machine welding execution mechanism, three-dimensional perception unit and welding power supply.This application identifies weld gap in real time by perception positioning unit, and according to gap size automatically matches expert database to adjust welding current, speed and swing amplitude;With single vertical square tube as basic welding unit, control double robot opposite synchronous welding, build symmetric heat field.This application can effectively guide galvanized steam directional escape, significantly reduce porosity defects, automatically compensate the gap fluctuation caused by blanking error, while ensuring weld forming consistency, inhibit structural deformation, greatly improve the production efficiency and intelligent level of square tube fence.