71 results about "Object perception" patented technology

The invention discloses a three-dimensional dynamic scene graph construction method based on 3D Gauss, and belongs to the field of computer body intelligence. The implementation method comprises the following steps of: realizing object perception and semantic feature extraction of open vocabularies by utilizing a visual basic model; a 3D Gaussian scene with high fidelity and continuous object semantics is constructed through multi-view multi-dimension optimization of 3D Gaussian representation; constructing a multi-level three-dimensional scene graph, extracting spatial levels and semantic relationships among objects by using 3D spatial positions and semantic tags of instance objects existing in a semantic Gaussian graph, and constructing a multi-level spatial semantic topology to accurately represent an environment layout; according to the method for realizing local updating for the Gaussian scene graph based on the environmental structural similarity, environmental change detection is carried out through real-time RGB-D observation and the structural similarity between high-quality rendering views of the Gaussian scene graph, and corresponding local updating is carried out by using rapid training and differentiable rendering of 3D Gaussian representation. And the capability of adapting to a complex dynamic environment of the 3D Gaussian scene graph is improved.

The invention provides an image text description generation method, electronic equipment and a readable storage medium. According to the method, the object perception prototype learning module and the global context feature extraction module are introduced, so that fine-grained information and global semantic understanding in the image are effectively balanced. The visual backbone network module can extract multi-scale and multi-level image features and perform fusion, thereby enhancing the expression ability of the image features. The object perception prototype learning module further extracts an object prototype from the fusion features to ensure that the model can accurately capture key objects and attributes thereof in the image, and the global context feature extraction module ensures that the overall context of the image is fully understood. On the basis, the encoding and decoding module combines the global context and the object prototype to generate the text description, so that the semantic splitting phenomenon in the traditional method is avoided, and the detail information in the image is effectively reserved, thereby improving the accuracy and integrity of the image description.

Embodiments of the present disclosure provide a method, device, and medium for training a large scale object foundation model. The method comprises obtaining a training dataset comprising a plurality of subsets for a plurality of object perception tasks, wherein a sample in the training dataset comprises an image with an object, a prompt indicating the object, and labeled object perception information of the image. The method further comprises generating, by the image encoder, an image feature based on the image. The method further comprises generating, by the text encoder or the visual prompt encoder, a prompt embedding based on the prompt. The method further comprises generating, by the object decoder, object perception information of the object based on the image feature and the prompt embedding. In addition, the method further comprises training the object processing model based on the generated object perception information and the labeled object perception information.

The invention discloses a dexterous hand environment article sensing and modeling algorithm based on body sensing. The algorithm comprises the steps that a mechanical arm drives a dexterous hand to approach an object; tactile-spatial data acquisition: a three-axis force sensor is arranged at the tail end of each finger of the dexterous hand, and the system is in contact with an object through multiple fingers of the dexterous hand to acquire multi-dimensional information of the appearance, surface characteristics and local topology of the object; active exploration sampling, multi-finger movement of the mechanical arm and the dexterous hand, and active penetration of unknown object surface and enhanced sampling of local features through the dexterous hand by the system; the method comprises the following steps: modeling an object attribute, analyzing a mechanical response sequence obtained by tactile-spatial data acquisition and active exploration sampling, modeling a target object in three dimensions of elasticity, hardness and local geometric morphology, and realizing accurate identification and modeling of an unknown object appearance form and an object attribute in an environment. The distribution uniformity of the tactile point cloud and the model integrity are improved, the hardware dependency is reduced, and the environmental adaptability is enhanced.

A method for (3D) object detection includes: receiving an input image with respect to a 3D space, an input point cloud with respect to the 3D space, and an input language with respect to a target object in the 3D space; using an encoding model to generate candidate image features of partial areas of the input image, a point cloud feature of the input point cloud, and a linguistic feature of the input language; selecting a target image feature corresponding to the linguistic feature from among the candidate image features based on similarity scores of similarities between the candidate image features and the linguistic feature; generating a decoding output by executing a multi-modal decoding model based on the target image feature and the point cloud feature; and detecting a 3D bounding box corresponding to the target object by executing an object detection model based on the decoding output.

A method for monitoring a performance of an object perception system of an automated driving system of a vehicle and related aspects is disclosed. The object perception system is configured to ingest sensor data samples generated by vehicle-mounted sensors and to output object perception data indicative of detected objects in a surrounding environment of the vehicle and one or more attributes of the detected objects. The method includes outputting reference data indicative of detected objects in the surrounding environment of the vehicle and of one or more attributes of the detected objects. The method further includes comparing the object perception data with the reference data and assigning confidence values to the object perception data based on the comparison. The method further includes controlling the vehicle, the object perception system, and / or downstream ADS functions configured to ingest the object perception data, based on the assigned one or more confidence values.

The invention discloses a sparse view angle three-dimensional Gaussian language field construction method and system for humanoid robot grabbing. Comprising the following steps: acquiring an RGB image set and a user natural language instruction under a sparse view angle; reconstructing a three-dimensional point cloud through stereo matching, and initializing a three-dimensional Gaussian primitive field after noise elimination and measurement alignment; two-dimensional semantic features are embedded into the field, joint optimization is carried out through a double-path semantic supervision module, and a three-dimensional Gaussian language field with consistent semantics is constructed; wherein the dual-path semantic supervision comprises an object perception path and a global context path, the local semantic consistency and the overall semantic relationship are constrained respectively, and the final semantic representation of each Gaussian primitive is obtained through weighted fusion; and finally, candidate grabbing postures are generated, semantic reordering is carried out in combination with a language instruction, and the optimal grabbing posture is screened through geometric-semantic joint scoring. Under the sparse image set input condition, the execution accuracy of a robot grabbing task under a complex instruction can be improved.

The invention provides an object sensing method and system based on proximity vision and tactile deformation, which are applied to the technical field of tactile sensing, and the method comprises the steps: obtaining a binocular image collected by a visual tactile sensor after a target object is in contact with a preset visual tactile sensor; performing visual feature extraction based on the binocular image to obtain a binocular feature point two-dimensional coordinate of the target object; extracting a tactile mark based on the binocular image to obtain a two-dimensional coordinate of a tactile mark point of the target object; through a three-dimensional reconstruction model based on ray tracing, based on the two-dimensional coordinates of the binocular feature points, determining approximate three-dimensional visual perception of the target object in the non-contact area; and determining contact three-dimensional deformation perception of the target object in the contact area through a three-dimensional reconstruction model based on ray tracing and based on the two-dimensional coordinates of the binocular feature points. According to the invention, the perception capability of the visual tactile sensor on the object state and the interaction degree can be improved.

The invention discloses a neural implicit vision SLAM (Simultaneous Localization and Mapping) method based on dynamic perception. The method aims at solving the core technical problems that an existing visual SLAM method is insufficient in robustness, poor in global consistency, large in calculation overhead and the like in challenging environments such as dynamic scenes, weak texture areas and violent illumination changes. According to the method, the feature processing capability of deep learning, efficient dynamic object perception, advanced neural implicit mapping and a global optimization mechanism are integrated, so that more accurate camera pose estimation and higher-quality static environment map construction are realized. In the tracking module, a six-step workflow based on mask guidance is adopted, dynamic objects are filtered from the source, frame-level pre-screening is carried out, and the robustness and the calculation efficiency of the system are remarkably improved. In a dynamic local mapping module, a pixel-level fusion method based on transmission probability and inverse variance weight is innovatively adopted, texture blurring and geometric distortion at the boundary of a plurality of sub-maps are effectively inhibited, and the visual quality of a global map is improved. Besides, by introducing a loop candidate frame reordering strategy based on pose uncertainty weighting in loop detection, visual similarity and geometric credibility can be combined, the false detection rate is effectively reduced, and global consistency and long-term precision of the map are ensured.

The invention provides an object-oriented hierarchical memory network device. The object-oriented hierarchical memory network device comprises a low-level object sensing module, a space-time memory construction module and a long-term space-time memory modeling module, the low-level object sensing module is used for extracting object features of an input video through a segmentation basic model; the space-time memory construction module is used for extracting object interaction and scene information by analyzing an inter-frame space and time relationship so as to splice object features and input original video frames to generate a space representation of target perception; and the long-term space-time memory modeling module is used for storing historical information through a memory bank, capturing a global dependency relationship of a video level, and splicing a plurality of adjacent video frames output by the space-time memory construction module on a channel dimension to form local multi-frame space-time representation. And the purpose of enhancing the capturing capability of the model on fine-grained space-time dynamic in the long video is achieved.

The invention discloses a visual language large model perception enhancement method and system based on visual prompt, and relates to the field of visual language large models. The problem that how to deploy a small-scale large-language model under the condition that resources are limited in the prior art becomes an urgent problem to be solved is solved. The method comprises the following steps of: segmenting an original image by adopting a segmentation component to generate a mask and an object segmentation list; respectively processing an original image and an image mask generated by a semantic segmentation device by using a visual encoder, so as to extract multi-level visual features which highlight the position and boundary of an object; layer normalization and MLP layer processing are carried out to form visual features; taking the generated mask and the segmentation result list of the object as a text instruction, and inputting the extracted multi-level visual features which highlight the position and the boundary of the object and the visual features into a visual language large model to perform autoregressive semantic generation; the method is also suitable for the technical field of improving the object perception and question-answering ability of the visual language large model without adding extra training parameters.

The application discloses a threat object detection method based on reinforcement learning, and aims to improve the threat object detection accuracy in the field of automatic driving. The technical scheme is to construct a threat object reasoning and detection system based on reinforcement learning, which is composed of a strategy model, a reward value calculation module, a group advantage calculation module and a strategy model updating module. The strategy model in the system is trained to obtain a trained threat object reasoning and detection system with threat reasoning and threat object perception capabilities. Finally, the trained threat object detection system is used for threat object detection to obtain the boundary box and threat level of the threat object. The application enables the multi-modal large language model to realize fine-grained object-level threat reasoning based on the context information of the driving scene, significantly improves the understanding ability of the automatic driving system to threat semantics and the perception ability of the threat object, and improves the threat object detection accuracy in the field of automatic driving.

The invention discloses a sorting system and method based on machine vision, and the sorting system based on machine vision comprises an image collection module, a data processing module and a sorting execution module. The image acquisition module is used for synchronously acquiring a hyperspectral image and three-dimensional point cloud data of a target area. And the data processing module is in communication connection with the image acquisition module and comprises a multi-modal feature fusion unit and a dynamic sorting decision unit. According to the invention, through multi-modal information fusion, chemical components and geometric space information of an object can be obtained at the same time, and more comprehensive object perception is realized; a fusion network based on an attention mechanism is adopted, different modal features are weighted in a self-adaptive mode, and the recognition precision is improved; physical constraints and task targets are comprehensively considered, an optimal grabbing point and a movement track are generated, and the sorting success rate is increased; the method has a self-learning capability, and can continuously optimize the system performance from practical experience.

Systems and methods for cooperative perception are described. In some examples, the system can comprise a first subsystem comprising a first sensor, a first communication device, and a processor, which can cause the system to: detect, by the first sensor, first point cloud data, apply a data preprocessing process to the first point cloud data to generate first preprocessed sensor data, apply a feature encoding process to the first preprocessed sensor data to generate first feature data; apply an adaptive feature filtering process to the first feature data to select a first subset of features from the first feature data; apply a cooperative feature aggregation process to fuse the first subset of features with other subsets of features, to generate a fused feature map; and apply an object perception model to the fused feature map to generate object perception data.

The invention discloses a space object sensing method based on a privacy protection camera and a radar, and the method comprises the steps: obtaining the three-dimensional space characteristics of a current space through the radar, judging whether a human body exists in the current space or not through the three-dimensional space characteristics, and starting the privacy camera to collect a global image if the human body exists in the current space; obtaining two-dimensional space features of the current space through the global image, and fusing the two-dimensional space features and the three-dimensional space features of the current space to obtain fused space features of the current space; inputting the fusion feature of the current space into a pre-trained object recognition model to obtain an object type of the current space; interference source labels and universal furniture labels are marked according to the object types of the current space, the current space objects with the interference source labels are filtered, and universal furniture distribution of the current space is obtained; and according to the current space general furniture distribution, radar detection parameters are adjusted according to detection areas, and detection thresholds of different detection areas are obtained.

The invention discloses a self-supervised learning method of a point cloud combining partial-overall perception, and belongs to the technical field of artificial intelligence. Aiming at the problem that the invariant perception of point cloud features is generally enhanced by expanding point cloud data and the influence of part of information on object perception is ignored in the existing method, the method comprises the following steps: preprocessing the point cloud data, extracting enhanced point cloud and part of point cloud feature vectors, and then enhancing the similarity condition of the point cloud using comparison loss evaluation feature vectors; a part of feature vectors of the point cloud are obtained through a point cloud feature extractor network and a point cloud feature projection head network, the part of feature vectors are put into an implicit field to learn implicit representation of the part, and the feature vectors of the enhanced point cloud are integrated to obtain an integrated overall feature vector and an integrated partial feature vector; and comparing the difference between the two feature vectors, and performing back propagation to optimize the network according to the difference condition. According to the method, the relation between local and overall point cloud is fully combined, and richer point cloud information representation is obtained.

The embodiment of the invention provides a scene recognition method and device based on contrast invariant risk minimization, a storage medium and electronic equipment, and belongs to the technical field of scene recognition. The method comprises the steps of: dividing a sample into a plurality of environments; obtaining image features and object perception features; extracting verb invariant features and role specific features; respectively setting verb classifiers for different environments, and introducing invariant risk minimization constraint and contrast learning loss; identifying and positioning each entity in the scene, and generating a corresponding target category and a bounding box result; and through training, carrying out joint optimization on verb classification loss, role recognition loss and auxiliary loss. The method can effectively improve the accuracy of the model on action recognition, noun recognition and position positioning, and has better cross-domain generalization performance.

The invention provides a target object sensing method, a communication device and a program product, relates to the technical field of communication, and can at least solve the problem of low target sensing accuracy in related technologies. The method comprises the following steps: acquiring perception information of a target object perceived by a plurality of perception devices at different positions; and selecting at least two pieces of target perception information from the multiple pieces of perception information, and confirming the behavior or position of the target object according to the at least two pieces of target perception information. The target sensing accuracy can be improved.

The embodiment of the invention provides a target detection method and device based on vision and radar, a vehicle and a medium. The method comprises the steps that first target object data and second target object data of a vehicle driving environment at the current moment are acquired, and the first target object data and the second target object data are fused to generate third target object data; and finally, determining fourth target object data according to the third target object data and historical third target object data generated at the previous moment. Wherein the first target object data is acquired through a camera, and the second target object data is acquired through a radar. The method is used for achieving the technical effect of improving the sensing precision and robustness of the target object.

A communication system, apparatus, method, and one or more non-transitory computer-readable storage devices for cooperative perception in communication perception integration employ the steps of: determining a type of a first communication node for object perception; and notifying the first communication node of the determined type. The step of determining the type of the first communication node comprises the following steps of: determining the type of the first communication node as a sensing transceiving node in a sensing transceiving set so as to receive or send a sensing signal for sensing an object, or determining the type of the first communication node as a sensing activation node in a sensing activation set so as to receive or send a sensing signal for sensing an object; wherein the sensing activation set comprises the sensing transceiving set.

The application discloses a sparse-view three-dimensional Gaussian language field construction method and system for humanoid robot grasping. The method comprises the following steps: acquiring an RGB image set under sparse-view and a natural language instruction of a user; reconstructing a three-dimensional point cloud through stereo matching, initializing a three-dimensional Gaussian primitive field after noise elimination and metric alignment; embedding two-dimensional semantic features in the field, jointly optimizing through a double-path semantic supervision module, and constructing a semantic-consistent three-dimensional Gaussian language field; wherein the double-path semantic supervision comprises an object perception path and a global context path, which respectively constrain local semantic consistency and overall semantic relationship, and the final semantic representation of each Gaussian primitive is obtained through weighted fusion; finally, candidate grasping postures are generated, semantic reordering is performed in combination with the language instruction, and the optimal grasping posture is selected through geometric-semantic joint scoring. The method can improve the execution accuracy of the robot grasping task under complex instructions under the condition of sparse image set input.

An object processing system within a trailer for a tractor trailer is disclosed. The object processing system includes an input area of the trailer at which objects to be processed may be presented, a perception system for providing perception data regarding objects to be processed, and a primary transport system for providing transport of each object in one of at least two primary transport directions within the trailer based on the perception data.

This application provides a vehicle control method and apparatus, storage medium, and electronic device, relating to the fields of smart mines, autonomous driving, and unmanned vehicles. The method includes: fusing the perception results of a first target object acquired by a lidar and the perception results of a second target object acquired by a millimeter-wave radar to obtain a target object perception queue; and fusing the target object perception queue with a first target object tracking queue using a target fusion strategy to obtain a second target object tracking queue. The target fusion strategy includes: for each perception element, if a matching first tracking element exists, determining whether its source includes both lidar and millimeter-wave radar; if both are included, determining the target sensor speed information based on the respective speed information of the lidar and millimeter-wave radar, and then updating the first tracking element to obtain the second tracking element. This application ensures the reliability of target object detection and improves the safety and efficiency of autonomous driving in mines.

The application discloses a tunnel target object sensing method based on 4D millimeter wave radar and camera fusion, comprising the following steps: judging whether a vehicle enters a tunnel or exits the tunnel; when judging that the vehicle enters the tunnel, tracking the target object until the vehicle exits the tunnel; collecting image data of the target object in a 2D space through a camera, converting the pixel depth information of the image data into a 3D space detection result and outputting; collecting point cloud information through a 4D millimeter wave radar, denoising and clustering the point cloud information to obtain a 3D target detection result and output; and integrating the detection result and the 3D target detection result to generate target object information. The application realizes redundant design of a sensing end, improves environmental sensing capability, and in the case that the misrecognition rate is low, can stably identify stationary vehicles or obstacles in the tunnel, improves the robustness and safety of an autonomous vehicle, and ensures passenger safety.

The invention provides a two-dimensional code attached video identification analysis method based on object perception, and the method comprises a mobile terminal which is provided with a two-dimensional code scanning verification module. The method further comprises the following steps: S1, real-time video acquisition and preprocessing; s2, feature extraction; s3, feature fusion and classification; s4, outputting a result and performing anti-cheating judgment; the method overcomes the defects that an existing two-dimensional code inspection technology cannot identify the physical attribute of a carrier, the response speed is low, and cheating is easily caused by stored and pre-stored pictures / videos, and the like, and the mobile terminal is forced to shoot a short video containing a target two-dimensional code in real time (generally 2-3 seconds). And the dynamic visual features (total time consumption is less than or equal to 5 seconds) of the two-dimensional code area and the surrounding background in the video frame sequence are quickly analyzed, and whether the two-dimensional code is attached to the surface of a three-dimensional object or only exists on a two-dimensional plane medium (picture or screen) is judged, so that cheating behaviors implemented by using the pre-stored two-dimensional code picture / video are prevented from the source.

The application discloses a dynamic object perception system and method based on multi-source sensor information fusion, relates to the field of dynamic object perception, and through parallel processing of multi-source heterogeneous data such as RGB-D images, 3D point clouds and millimeter wave radars, and introduction of a cross-modal inconsistency measurement mechanism, differences and conflicts between different sensor features are actively analyzed and quantified to evaluate the perception confidence of each sensing mode under the current environment in real time. Based on this, an adaptive attention mechanism is driven to dynamically generate the fusion weight of each mode, and finally, intelligent weighted fusion of each mode feature vector is realized, and a high-robustness fusion target state is output. In this way, low-quality or conflicting sensor information can be intelligently suppressed, and the contribution of high-quality information can be enhanced, so that more robust and accurate dynamic object perception can be realized in a complex and changeable environment.

The invention discloses a threat object detection method based on reinforcement learning, and aims to improve the detection accuracy of threat objects in the field of automatic driving. According to the technical scheme, the threat object reasoning detection system based on reinforcement learning is constructed and composed of a strategy model, a reward value calculation module, a group advantage calculation module and a strategy model updating module. And training a strategy model in the system to obtain a trained threat object reasoning detection system with threat reasoning and threat object sensing capabilities. And finally, performing threat object detection by adopting the trained threat object detection system to obtain a bounding box and a threat level of the threat object. According to the method, the multi-mode large language model can realize fine-grained object-level threat reasoning based on the context information of the driving scene, the ability of an automatic driving system to understand threat semantics and the ability of the automatic driving system to perceive threat objects are remarkably improved, and the detection accuracy of the threat objects in the field of automatic driving is improved.

In various examples, certain objects typically found in predictable locations within an environment can be detected more reliably by utilizing known information about the locations and / or the objects themselves. For instance, the disclosed systems and methods can determine the locations of target areas within a coordinate system associated with a target region in an environment and use the target areas to detect and track specific objects that might otherwise be difficult to detect. For example, a target region might be a parking space for a machine, and the coordinate system might specify target areas corresponding to wheel stops, curbs, ground barriers, or other objects commonly associated with parking spaces.The systems can scan various points, which represent sensor feedback, to determine whether a target object is located in a target area, and in some cases also to track the target object.

A communication method and a related device, in the method, after a first communication device sends first information for requesting position information of a target object, the first communication device may receive second information, and determine the position information of the target object based on the second information. In this way, different communication devices can determine the position of the target object in a mutual cooperation mode so as to realize perception of the target object. In some implementations, a first communication device does not have an angle measurement capability or the angle measurement capability of the first communication device satisfies a first condition such that the first communication device is able to implement object perception through cooperation of other communication devices in the event that there is a lack of some or all of the angle measurement capabilities. The flexibility and robustness when the communication equipment executes the sensing task can be enhanced.

The application provides an object perception method and device based on binaural hearing signal expression and a hearing aid, and belongs to the technical field of hearing aids. The object perception method based on binaural hearing signal expression integrates radar perception, object analysis, target identification, spatial sound effect simulation and hearing aid broadcasting and the like, realizes intuitive perception of a target object to a surrounding environment in a vision-limited or hearing-limited scene, and enables the target object to understand the dynamics of the surrounding environment in real time without relying on visual information, thereby enhancing the auxiliary ability of a wearable device to a user in a complex scene. Compared with traditional visual display or voice prompt, the feedback mode of the hearing signal is more suitable for the needs of vision-limited users, hearing-limited users or in a specific complex scene, and can effectively improve the use experience of the target object.