2972 results about "Pose" patented technology

The invention discloses a visual inertial positioning method based on dynamic target detection and semantic information constraint, and belongs to the field of motion estimation and dynamic environment processing. According to the method, a dynamic target detection mechanism is introduced, the dynamic target is effectively detected based on the target detection network, inertial navigation information and geometric constraints, the dynamic target is effectively recognized in the image processing process, the corresponding dynamic feature points are screened out, the mismatching rate of the dynamic features is remarkably reduced, and high-quality observation input is provided for back-end optimization. In the back-end sliding window optimization stage, a semantic information consistency constraint method is constructed, and the estimation stability of the system in a weak texture area or a repeated texture area is enhanced by utilizing the consistency of feature points in the same semantic area on a geometric structure. Visual inertia pose estimation is realized based on dynamic target detection and semantic information constraint, and high-robustness and high-precision pose estimation can still be realized in a complex environment with dynamic interference of pedestrians, vehicles and the like and severe scene change.

The invention relates to the technical field of robot control, and discloses a heavy-load robot motion trail method and system based on machine learning. The method comprises the steps that historical movement track data of the heavy-load robot in a working scene are collected, and the data comprise a joint position sequence, an end effector pose sequence and environment obstacle distribution information; the data is preprocessed, track features are extracted, a space-time correlation matrix is constructed, and the matrix is used for representing the dynamic coupling relation between joint movement and the tail end pose; training a trajectory prediction model containing a long and short-term memory network and an attention mechanism based on the matrix, and generating a collaborative mapping relation between a joint position and a tail end pose; obtaining a current task target pose sequence and an environment constraint condition in real time, and outputting a candidate track set meeting dynamic constraint through a model; and adopting a multi-objective optimization algorithm to screen candidate tracks, generating an optimal track instruction and issuing the optimal track instruction to an execution mechanism. The method adapts to the complex characteristics and variable working conditions of the heavy-load robot, and the track adaptability is improved.

The invention discloses a self-adaptive three-dimensional scene reconstruction method and system based on a single panorama, and belongs to the field of computer vision processing. The method comprises the following steps: firstly, generating a depth map through an indoor scene panorama and constructing an initial three-dimensional grid; then generating a multi-view image and a shielding mask thereof based on view conversion, forming a training sample pair for fine tuning of the diffusion completion model, and injecting scene prior information for the model; then extracting a grid boundary contour and calculating a central axis, and adaptively constructing a camera pose set; after the integrity of the grid is optimized through iteration completion, the grid is converted into a 3D Gaussian sputtering field, and Gaussian point parameters are optimized through multi-view data; in the optimization process, an up-sampling strategy of rendering error feedback and edge Gaussian points is adopted, and finally a scene model with complete geometric textures is output. According to the method, a high-quality three-dimensional scene is reconstructed from a single panorama through a structure self-adaptive completion and Gaussian refining technology, and the method is particularly suitable for immersive roaming reconstruction of indoor scenes.

The invention relates to the technical field of intelligent manufacturing, and discloses a target structure automatic detection method, device and equipment and a medium, and the method comprises the steps: obtaining scanning path planning data of a target detection structure, driving an ultrasonic probe to execute surrounding scanning motion, and collecting an ultrasonic image sequence and spatial pose data; dynamically adjusting a pressure application angle and a scanning speed based on force feedback information, fusing spatial pose data and an image sequence to perform three-dimensional reconstruction, constructing a three-dimensional geometric model of a target detection structure, extracting feature distribution data by applying an intelligent analysis model, generating a feature decision set, and performing feature extraction; and mapping the feature decision set to a three-dimensional coordinate system to construct an analysis report containing the feature type marks and the topological relation. Through fusion of force control scanning, image reconstruction and intelligent analysis, standardization and intelligence of a detection process are realized, image consistency and structure identification precision are improved, manual dependence is reduced, and comprehensiveness and reliability of lesion identification are enhanced.

The invention discloses an end-to-end underwater three-dimensional reconstruction method and system based on an underwater imaging model, and belongs to the technical field of underwater image processing. According to the method, deep learning pose estimation is combined with an underwater imaging model; firstly, a multi-frame underwater image sequence is collected as input, a deep neural network is constructed, and the network mainly comprises two core sub-modules: a pose estimation network; secondly, a three-dimensional reconstruction network is adopted, dense point cloud or voxel reconstruction is completed according to the predicted pose and image content, pose estimation and the three-dimensional reconstruction process are integrated in the same system, and overall joint optimization is achieved; and in combination with a self-adaptive underwater imaging model, modeling is performed on physical processes such as underwater illumination attenuation and scattering, so that the reality sense and the accuracy of a reconstruction result are improved. According to the method, high-quality three-dimensional reconstruction of images in a complex underwater environment is realized, and the method can be widely applied to ocean engineering, underwater robots, submarine topography surveying and mapping and underwater cultural relic protection.

The invention discloses a building engineering progress automatic identification early warning system based on computer vision, which relates to the field of building engineering informatization and comprises a synchronous calibration module, a joint calibration module, a pose generation module, a mapping construction module, a resampling module, a mapping registration module and a comparison early warning module. Clock synchronization and rolling readout calibration are carried out on a camera and an inertial measurement unit, a continuous time pose is established in a frame, sampling is carried out according to rows, and an equivalent global shutter frame is generated in combination with plane and pixel-level geometric mapping; outputting a camera pose track and a local measurement map by adopting back-end optimization containing a rolling mechanism, and registering with the building information model; component detection, instance segmentation and state discrimination are completed on an equivalent global shutter frame, pixel domain measurement is unified into engineering quantity under pose and registration constraints, the engineering quantity is mapped to a work decomposition structure and a progress plan, deviation is calculated, and graded early warning is output according to a multi-threshold rule. Long-term stable and reliable operation and evidence traceability of the system are guaranteed through whole-course quality monitoring and threshold grading.

The invention provides a three-dimensional human body posture estimation method and system based on multi-view visual information fusion and a storage medium, and the method comprises the steps: 1, designing the front half part of a model into Ender Layers with the same layer number as a Transform decoder at a multi-view feature fusion layer, carrying out the data enhancement of an input multi-view original image, and carrying out the reconstruction of the Ender Layers in the multi-view feature fusion layer; inputting the CNN Backbone with the shared weight to extract an initial feature map; 2, introducing a micro-reprojection optimization mechanism, deeply fusing the multi-view geometric consistency constraint into a model training process, and guiding the model to predict a three-dimensional attitude end to end; and step 3, constructing a dynamic projection compensation module. The method has the beneficial effects that the method is particularly suitable for capturing human body posture information in a multi-person interaction scene, the shielding problem and depth estimation ambiguity in a single view angle can be effectively overcome, and the robustness, precision and efficiency of three-dimensional human body posture estimation are remarkably improved.

The invention provides a mechanical arm grabbing method and system based on multi-modal information fusion, and belongs to the technical field of robot intelligent control. Comprising the steps that the conversion relation between a camera coordinate system and a mechanical arm base coordinate system is established through camera calibration, a deep learning neural network is used for conducting grabbing pose estimation on an obtained RGB-D image, and multiple candidate grabbing poses are determined; analyzing a natural language instruction input by a user based on a multi-modal large model, and recognizing a target object region from the RGB-D image by combining a target detection and image segmentation technology; based on the obtained candidate grabbing poses and the target object area, an optimal grabbing pose is screened through a scoring mechanism and mapped to a mechanical arm base coordinate system; and then a dynamic grabbing path is generated by adopting an imitation learning algorithm, and the mechanical arm is controlled to execute grabbing operation. Through multi-modal semantic understanding, accurate grabbing of the mechanical arm in a complex environment can be achieved.

The invention discloses a cross-source data three-dimensional reconstruction method and system based on improved Gaussian sputtering, and the method comprises the steps: collecting an unmanned plane inclined image and a ground panoramic image of a target region, and constructing a time-space correlation data set; based on multi-view geometric constraints, space-time coding matching point pairs are established through an adaptive feature pyramid, intelligent incremental cross-source data sparse reconstruction is carried out, and point cloud and camera parameters are output; adopting improved Gaussian sputtering, compressing a three-dimensional Gaussian kernel into a two-dimensional Gaussian primitive through double tangent vector constraint, and fitting surface geometry to realize multi-scale reconstruction; and optimizing primitive parameters by using a differentiatable renderer, completing multi-scale fine reconstruction through gradient back propagation, and generating a high-precision three-dimensional model. According to the method, multi-scale accurate geometric prior input and accurate camera poses are provided for three-dimensional reconstruction, the dependence on professional manual operation in a traditional three-dimensional reconstruction method is greatly reduced, and meanwhile, the geometric accuracy and visual fidelity of a reconstruction result are remarkably improved.

The invention relates to the technical field of robot visual perception and motion control, and discloses a motion control method and device based on physical reference, equipment and a medium, and the method comprises the steps: obtaining instruction information, a multi-view image and movable assembly pose information; processing the multi-view image according to the instruction information to generate target segmentation information; generating a scale normalization point cloud and a model estimation baseline; determining a physical reference baseline and generating a scale calibration factor; converting the scale normalization point cloud into a physical space point cloud by using a scale calibration factor; extracting a three-dimensional relative position of the target object relative to the movable component in combination with the target segmentation information; an action instruction is generated based on the multi-modal input. According to the method, physical scale alignment of the point cloud is realized through physical reference baseline calibration, so that a visual reconstruction result has real space significance, an accurate action instruction is generated, and the robot space understanding and operation precision is improved.

The invention discloses a ring main unit inspection robot autonomous navigation method and system based on SLAM, particularly relates to the technical field of robot autonomous navigation and intelligent inspection, and is used for solving the problem of positioning drift caused by repeated features of an existing ring main unit scene. Semantic feature analysis and topological constraints are introduced into an SLAM processing flow, acquired image data and point cloud data are processed through a deep learning model, objects such as an electrical cabinet, a corridor channel and a cable trench are identified, and a semantic feature set with category labels and spatial position information is generated; and constructing a topological graph containing node spacing, connectivity and directivity constraints based on the semantic features, adding the topological graph as a constraint factor into SLAM back-end optimization, and performing joint optimization in combination with vision, a laser odometer and inertial prior information, thereby avoiding only depending on repeated geometric feature positioning, and improving the positioning accuracy. The problems of loopback misjudgment and drifting caused by feature confusion are reduced, and the pose resolving stability in the ring main unit environment is improved.

The invention belongs to the technical field of artificial intelligence, and provides a target pose sensing estimation method and system for mechanical arm grabbing operation and a robot. According to the technical scheme, multi-modal data of a target object in the mechanical arm grabbing process under a historical language instruction is obtained; based on the obtained multi-modal data of the target object in the historical mechanical arm grabbing process under the historical language instruction, the end-to-end large model is trained, and a trained large model is obtained; according to real-time environment information and task requirements, the weight of each sub-target in multi-target optimization in the large model is dynamically adjusted to obtain an inference model, an action sequence is generated based on a vector field predicted by the inference model, the generation process is optimized, and meanwhile, multi-modal information is fused for real-time decision making; and the action sequence of the mechanical arm executing the grabbing task in the current environment is determined. And the grabbing efficiency and safety of the mechanical arm are improved.

The invention discloses a visual prediction method for space-time manifold and implicit state deduction of a robot, and relates to the technical field of robot space visual perception. The motion state and the visual image of the robot are collected; calculating carrier pose offset and generating a reverse compensation matrix; modeling an image target into a visual cone probability manifold, and extracting an explicit manifold observation vector; monitoring the confidence coefficient in real time through an observation quality evaluation network, and activating an implicit deduction mode during observation degradation; reading historical time sequence characteristics by using an improved Transform model, and performing prediction and deduction on a future nonlinear motion state of the target; performing coordinate correction and probability field decoding on a prediction result in combination with the reverse compensation matrix, and outputting a three-dimensional prediction trajectory and a spatial covariance ellipsoid; according to the method, the problem that traditional visual tracking fails under the conditions of violent shaking of a carrier and target shielding is solved, and continuous and foresight physical scale positioning and safety decision making of a robot on a dynamic target are achieved.

The invention provides a robot skill learning and control method and device oriented to rich contact operation tasks. The method provided by the invention comprises the following steps: collecting human demonstration data; the human demonstration data at least comprises end effector pose data and stiffness matrix data; on the basis of the end effector pose data and the stiffness matrix data, a generalization motion track and a generalization stiffness track are obtained through time alignment, probability distribution modeling and new state constraint adaptation; constructing an obstacle model through environmental perception, taking the generalized motion trajectory and the generalized rigidity trajectory as reference sampling candidate trajectories, screening a collision-free trajectory in combination with motion consistency, rigidity consistency and smoothness cost, and performing iterative optimization to obtain an optimized motion trajectory and an optimized rigidity trajectory; and converting the optimized motion track and the optimized rigidity track into a robot joint torque instruction based on variable impedance control, and controlling the robot to execute a rich contact operation task based on the robot joint torque instruction.

The invention discloses a human motion posture recognition method and system based on multi-modal data fusion, and relates to the technical field of posture recognition, and the method comprises the steps: firstly, synchronously obtaining a human motion posture video frame and an IMU segment, then carrying out the feature extraction of the two kinds of heterogeneous data in a shunt parallel mode, and carrying out the feature extraction of the two kinds of heterogeneous data; and respectively capturing visual space attitude information and dynamic inertial characteristics of the IMU. Afterwards, specific features of the modals are uniformly expressed through a mixed Token and embedding mechanism, and a space-inertia collaborative attention fusion mechanism is further introduced to realize dynamic association and deep fusion of cross-modal information; and finally, classifying the multi-modal fusion representation vector obtained by fusion so as to realize accurate recognition of the human motion posture. In this way, the defect that traditional fusion is insufficient in capturing subtle action differences can be overcome, and the accuracy and stability of action recognition in a complex scene are improved.

The invention relates to the technical field of robot control, and discloses a master-slave cooperation-based control method, device, equipment and medium, and the method comprises the steps: collecting a spatial pose, a contact force and a dual-view image, and constructing a multi-modal demonstration data set; executing time alignment and normalization to form a standardized input sequence; action features, force features and visual features are extracted through the action modal coding network, the mechanical modal coding network and the visual coding network; fusing to form a multi-modal tensor sequence, and inputting the multi-modal tensor sequence into a Transform decoder to generate a joint time sequence representation; outputting a training action prediction vector from the joint time sequence representation through an action predictor, constructing a supervision loss function in combination with expert demonstration annotation to update each network, and obtaining an optimization control model; and processing the real-time input control driving execution mechanism based on the optimization control model. According to the method, the control model is trained and optimized through multi-modal sensing and time sequence modeling, the joint control instruction is generated in deployment, and stable execution of a complex interaction task is achieved.

The invention provides an assembly action intelligent identification and evaluation method and system based on an image sequence, and relates to the technical field of intelligent manufacturing, and the method comprises the steps: obtaining an assembly image sequence, calculating three-dimensional space information, constructing a workpiece state transition diagram, predicting a workpiece pose, generating a standard action trajectory, and comparing the deviation between the actual trajectory and the standard trajectory. And calculating an action coordination index, judging an assembly state, analyzing action defects, and realizing assembly quality evaluation. According to the method, real-time monitoring, anomaly detection and quality evaluation of the assembly process can be realized, and the assembly precision and efficiency are improved.

The invention discloses a dexterous grabbing method and system for a robot, and aims to solve the technical problems that an existing robot grabbing system is high in dependence on hand-eye calibration precision (especially translation vectors), and the grabbing capacity of objects without priori knowledge is limited. According to the method, a reference-transformation-correction control framework is provided, the pose of an unknown object is accurately recognized through a multi-stage visual segmentation strategy, and a preliminary grabbing target is generated based on the reference relation of one-time teaching; most importantly, the system starts closed-loop visual servo correction, and iteratively corrects the position by detecting the real-time pose of a tail end visual mark, so that translation errors in hand-eye calibration are eliminated, and high-precision alignment is realized; in addition, the system also integrates the functions of object posture adjustment and rapid rotation calibration. Through cooperative work of open-loop target generation and closed-loop position correction, dependence on hardware accurate calibration is effectively reduced, and the grabbing success rate and robustness are improved.

The invention discloses a human body posture estimation method, system and device and a medium, and relates to the technical field of computer vision, and the method comprises the steps: obtaining a to-be-estimated picture containing a human body; the method comprises the following steps of: performing feature extraction and down-sampling on a to-be-estimated picture, performing dimension raising, depth separable convolution operation, channel aggregation operation and dimension reduction on a feature map with down-sampling resolution in an output branch after down-sampling to obtain a local feature map, performing up-sampling reconstruction on the local feature map by adopting dynamic weight interpolation, and fusing an output branch which is not down-sampled to obtain a local feature map; obtaining a first fusion feature; taking the first fusion feature as an initial feature, repeating the step of obtaining the first fusion feature, obtaining a second fusion feature, carrying out fusion to obtain a dual-scale fusion feature, extracting a depth perception feature in the dual-scale fusion feature, carrying out human body posture estimation through the depth perception feature, and obtaining a human body key point heat map. According to the invention, multi-scale and global information is obtained through a lightweight structure, and accurate key point positioning is obtained.

The invention belongs to the technical field of simulation modeling, and provides a high-fidelity three-dimensional reconstruction method fusing attitude prior and geometric constraint, which comprises the following steps: acquiring an original image, and determining an initial camera attitude based on a laser radar inertial odometer; determining error constraint information and relative attitude constraint information, and optimizing the initial camera attitude according to the error constraint information and the relative attitude constraint information to obtain a fine camera attitude; inputting the original image into an image reasoning model obtained by pre-training to obtain a surface normal map output by the image reasoning model; and constructing a three-dimensional Gaussian distribution model, determining geometric constraint information, and performing iterative optimization on the three-dimensional Gaussian distribution model by using the geometric constraint information according to the surface normal map and the fine camera attitude to obtain a three-dimensional scene model. According to the method, the overall reconstruction efficiency is improved through attitude prior and geometric constraints, and the authenticity, accuracy and efficiency of the scene reconstruction process are improved.

The invention discloses a three-dimensional human body posture estimation method and system, and relates to the technical field of computer vision, and the method comprises the steps: extracting two-dimensional human body posture key points from a monocular video picture sequence, and generating a two-dimensional human body posture key point sequence; projecting the two-dimensional key point sequence to a feature space through nonlinear high-dimensional mapping to generate a high-dimensional feature space matrix; and inputting the high-dimensional feature matrix into a three-dimensional human body posture recognition model fusing motion constraints and frequency division spatial-temporal features to obtain a three-dimensional human body posture key point sequence, and realizing three-dimensional human body posture estimation through three-dimensional coordinates. According to the method, the robustness and the detection precision of the monocular three-dimensional human body posture estimation method are improved. Error values of relative movement speed, skeleton length and skeleton direction of the key points are calculated, so that training is easier to converge, and the training process is more stable.

The invention discloses a target positioning and capturing method and system based on multi-modal semanteme, and relates to the technical field of image recognition and mechanical control, and the method comprises the steps: obtaining a two-dimensional image and a natural language interaction instruction; visual-language feature alignment processing is carried out, if semantic ambiguity exists in a natural language interaction instruction in the alignment processing process, a reverse question statement is generated based on a generative interaction mechanism, and a unique target is determined; collecting a multi-view two-dimensional image of a target, and fusing to obtain a three-dimensional point cloud model; inputting a spatial pose generation network to obtain candidate six-degree-of-freedom spatial poses; performing semantic common sense filtering and geometric interference filtering in sequence; calculating a posture score of the filtered candidate six-degree-of-freedom space posture, and selecting an optimal target three-dimensional posture; generating a grabbing operation parameter sequence; and executing the grabbing operation parameter sequence. According to the method provided by the invention, the problem of unsuccessful grabbing caused by semantic ambiguity and shielding is solved.

The invention provides a semantic aerial view visual relocation method and device in a non-exposed scene, electronic equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a multi-view image sequence under a non-exposed scene (such as a tunnel, an underground pipe gallery or an underground parking lot); semantic recognition is carried out based on a pre-trained semantic target detection model, and spatial consistency semantic features are extracted through a semantic-geometric dual-channel fusion mechanism combining a semantic mask and geometric constraints; the method comprises the following steps of: realizing three-dimensional reconstruction by using a voxel micro-renderable modeling method (VGGT), and generating a dense three-dimensional semantic point cloud fusing semantics and a geometric structure; two-dimensional semantics are mapped to a three-dimensional space through a projection and back projection relation, and point cloud semantics are endowed; main structure planes such as the ground, the left wall surface and the right wall surface are extracted, and a two-dimensional semantic aerial view with semantic annotation is generated; and pose estimation is carried out based on a reciprocal matching strategy guided by a semantic mask, so that visual repositioning with high precision, high robustness and semantic interpretability is realized. The method breaks through the problems of low precision, sparse features and poor semantic consistency of traditional visual repositioning in a non-exposed environment, and can be widely applied to the fields of intelligent transportation, underground inspection and unmanned system positioning.

The invention discloses a multi-modal large model three-dimensional perception method based on Riemannian manifold prior guidance, and relates to a computer vision technology. The method comprises the following steps: firstly, constructing a multi-modal large model and point cloud sensing network coordinated cross attention reinforcement learning joint training framework, so that the large model obtains a three-dimensional scene space topology understanding capability; secondly, a conformal property on a Riemannian manifold is utilized to construct a three-dimensional contour and topological association, and multi-scale invariance of homotopy mapping learning contours is introduced, so that the challenges of shape diversity and great size difference of objects are solved; in addition, a lightweight three-dimensional attention gate filtering mechanism is introduced into a point cloud encoder, and more effective global and local point cloud geometric semantic association is established. And finally, when the motion pose quality is evaluated, fusing prior knowledge of the three-dimensional physical relationship to form mixed physical measurement so as to overcome label noise caused by single evaluation scale. And universal understanding and high-precision perception of the robot on complex three-dimensional environments and objects can be realized.

The invention belongs to the technical field of intelligent robot navigation and positioning, and discloses an intelligent robot autonomous mapping method in a GNSS rejection environment, which comprises the following steps of: monitoring a GNSS signal state in real time, and switching to pure laser SLAM mapping when detecting that the number of available satellites, PDOP or pseudo-range residual exceeds a threshold; the method comprises the following steps: acquiring three-dimensional point cloud data, filtering and denoising, obtaining a filtered and denoised current frame point cloud, performing laser radar mapping and positioning registration, selecting a key frame point cloud, constructing lightweight neural implicit scene representation, and training an MLP network; loopback detection is carried out, a sparse factor graph is constructed, key frame poses are corrected, then the key frame poses are applied to original three-dimensional point cloud data, and a two-dimensional grid map is generated. According to the invention, under the condition that the GNSS signal is blocked or fails, the environment map with high precision and low drift is continuously generated, and the real-time performance and robustness of autonomous mapping are remarkably improved.

The invention discloses a robot arm control method and device, equipment, a medium and a product, and the method comprises the steps: obtaining a pre-training dynamic model and a real visual depth map of a real robot arm visual angle, and the pre-training dynamic model is used for completing a specified task; determining a joint control value according to the real vision depth map and a pre-training kinetic model; hybrid action control is carried out based on the joint control value, and remote target navigation is carried out on the response action of the real robot arm through a visual navigation model; and the controlled simulation target image and the actual image are obtained, feature matching and closed-loop estimation are carried out based on the simulation target image and the actual image, and pose error compensation is carried out on the action of the real robot arm. The motion is observed and deduced through a real visual depth map, then real and simulated mixed motion control is carried out to reduce a visual and dynamic gap, and finally pose error compensation is carried out. And the pose error of the arm is reduced, and a start pose guarantee is provided for downstream control.

The invention discloses a pose estimation method and device based on point pair feature matching, a medium and a product. The method comprises the following steps: performing coarse registration on a fusion scene point cloud and each template point cloud, and calculating point pair features; according to the corresponding hash key values, retrieving from each hash table to establish registration point pairs; calculating a translation vector and a rotation quaternion matched with each registration point pair according to the reference point coordinate system and the template coordinate system, and performing clustering processing on each registration point pair to obtain each initial corresponding set; screening out a target pose candidate transformation matrix and a matched target template point cloud from the pose candidate transformation matrixes matched with the initial corresponding sets respectively; a target optimization function between the fusion scene point cloud and the target template point cloud is constructed, iterative optimization is performed on parameters in the target pose candidate transformation matrix, and the target pose of the fusion scene point cloud is obtained at the end of iteration, so that the precision, real-time performance and reliability of six-dimensional pose estimation are improved.

The invention relates to the field of robots, and discloses a robot ordering method and system, and the method comprises the steps: obtaining a depth map and a color map of a current scene; obtaining a detection frame of a target object, and cutting out three-dimensional point cloud data of the corresponding object from the depth map according to a detection result; reasoning by taking the three-dimensional point cloud data as input of a reasoning model, and predicting candidate grabbing poses; and according to the predicted candidate grabbing poses, trajectory planning and motion execution are carried out. Reasonable action information can be generated according to information provided by the visual language model in combination with specific task requirements, and therefore control over the mechanical arm is achieved.

The invention provides a real-time monitoring system and method for sorting of a digital twin-driven Delta mechanical arm. According to the system, firstly, a physical experiment platform is built, and a virtual model is built through SolidWorks, 3DMAX and Unity 3D. A communication framework is designed, and wireless communication of the PLC, the sensor, the mechanical arm, the conveying belt and the digital twin system is achieved. Positive and inverse kinematics analysis and Cartesian space trajectory planning are carried out on a Delta robot, and a related algorithm is packaged into an MATLAB function and compiled into a DLL library for C # calling. The system collects joint angle data in real time based on a rotary encoder, transmits the joint angle data to a digital twin system through an OPC UA protocol, and drives a virtual mechanical arm model after resolving, so that high-precision and low-delay virtual-real synchronous pose reconstruction is realized. In addition, the system also develops a collision detection function based on a bounding box algorithm, and performs visual detection by using camera calibration and a YOLO model.

The present disclosure provides a method for calibrating a humanoid robot, comprising obtaining a humanoid robot with original kinematic biasing values, controlling the humanoid robot through predetermined poses, capturing image data of body parts using vision sensors mounted on the humanoid robot while moving through the poses, determining revised kinematic biasing values by processing the image data using a bipedal spatial perception model trained using synthetic image data containing keypoints, and replacing the original kinematic biasing values with the revised kinematic biasing values. The bipedal spatial perception model processes captured image data to generate observed keypoint locations on robot components, which are compared with kinematic-based locations from joint encoder measurements to minimize discrepancies through optimization algorithms.