69 results about "Commonsense reasoning" patented technology

The invention provides a natural language control method for a humanoid robot. By constructing the simple skill library and the complex skill library of the robot, the dependence on a large amount of operation data in a specific field is effectively reduced. According to the method, a large language model is used as a base of a task planner, and the capabilities of scene learning, semantic understanding and common sense reasoning of the task planner are brought into full play. By introducing a visual model in the field of computer vision, high-level semantic information and low-level spatial information of the environment can be obtained, and an accurate scene graph is constructed to serve as environment state representation. According to the voice interaction interface constructed by the invention, the naturalness of man-machine interaction is improved, so that a user can more conveniently communicate with the robot and input instructions, the user experience is effectively improved, and popularization of the humanoid robot in an actual application scene is facilitated.

The invention belongs to the technical field of computer vision and natural language processing, and discloses a video authenticity detection method and system based on a large model and a retrieval enhancement generation technology. Jointly constructing a to-be-detected text by combining the video title, the brief introduction, the text obtained by audio conversion and the video description text; completing the construction of an external knowledge base; performing fact verification based on a retrieval enhancement generation technology (RAG); performing common sense reasoning based on a large model: constructing cue words, performing common sense and logic reasoning on the to-be-detected text by utilizing the large model ChatGLM, detecting whether common sense mistakes and logic conflicts exist or not, and judging whether news is true or false according to the common sense mistakes and the logic conflicts; and comprehensively judging the authenticity of the to-be-detected video according to the fact check and common sense reasoning. Through the combination of the large model and the RAG technology, the accuracy and interpretability of video content authenticity detection are improved; the detection efficiency is improved; false news videos can be found and processed in time, and the health of a network environment and public trust can be maintained.

An exemplary method for detecting deepfake images and providing customized analysis comprises: receiving, from a user, a textual user inquiry regarding an image; inputting the textual inquiry and the image into a deepfake detection model, wherein the deepfake detection model comprises: an image encoder for generating a plurality of image embeddings based on the image; a text encoder for generating a plurality of textual embeddings based on the textual inquiry; one or more layers for generating a plurality of answer embeddings; and a language model for generating a textual analysis based on the plurality of answer embeddings; and outputting the textual analysis, wherein the textual analysis includes a classification result of whether the image is fake and further includes one or more visual features in the image and one or more attributes of the one or more visual features that contribute to the classification result.

The invention relates to the technical field of artificial intelligence, and discloses a text emotion deep reason recognition method and device based on common sense reasoning, and the method comprises the steps: firstly recognizing potential emotion clauses and potential reason clauses in a text, and constructing a corresponding emotion clause graph and a corresponding reason clause graph; then, searching emotional and causal common senses related to the clauses to expand a graph structure, and obtaining two graph neural networks for representing context semantics and common sense knowledge of an emotional and reason graph respectively; and utilizing an interactive attention mechanism to find out an emotion-reason combination clause pair with the maximum relevancy, and further decoding a deep reason with a relatively strong causal relationship and a corresponding emotion. Due to the fact that a text grammar structure is fully described and related external common senses are fused, deep emotional reasons can be better decoded, potential deep emotional reasons behind text surface emotional reason events are mined, and the accuracy and reliability of emotional reason inference are effectively improved.

Embodiments of the present disclosure relate to fine-tuning a neural network model using a weight-decomposed low-rank adaptation (DoRA). DoRA reduces the number of parameters that are fine-tuned, thereby reducing memory and the time needed to fine-tune the parameters. the Pre-trained weights are decomposed into two components, magnitude and direction, which are separately fine-tuned. The magnitude components are fine-tuned while the direction components remain unchanged (frozen). Then low-rank adaptation (LoRA) is used to fine-tune the direction components, efficiently minimizing the number of trainable parameters. Compared with using LoRA to fine-tune the weights directly, using DoRA exhibits a closer resemblance to full fine-tuning's learning behavior and improves upon LoRA in commonsense reasoning and visual instruction tuning tasks. By employing DoRA, both the learning capacity and training stability of LoRA is enhanced. The fine-tuned decomposed magnitude and direction components may be merged into the pre-trained weights to avoid any additional inference overhead.

The embodiment of the invention discloses a knowledge graph and multi-modal fusion government affair information recommendation method and equipment, belongs to the technical field of information recommendation, and solves the problem that the matching degree between a recommendation result and a demand is relatively low when a user obtains government affair information. Comprising the steps of generating a user preference vector based on a user historical retrieval behavior, and performing coarse-grained retrieval on a government affair knowledge graph to obtain candidate knowledge sub-graphs; fusing the candidate knowledge sub-graphs with multi-modal data uploaded by the user to obtain a multi-modal semantic vector; performing multi-hop information aggregation processing on the multi-modal semantic vector, and performing fine-grained retrieval based on an aggregated graph node vector and a user query vector; based on a reordering mechanism, reordering the retrieval results, and determining recommended government affair knowledge sub-graphs; and based on the multi-modal data and the recommended government affair knowledge subgraph, performing visual question and answer and visual common sense reasoning processing to obtain a multi-modal reasoning result, and based on user requirements, sending the multi-modal reasoning result as recommended content to the user.

The invention belongs to the field of intelligent planning control, and relates to an intelligent agent instruction execution method and system based on multi-level re-planning. The method comprises the following steps: preliminarily generating a sub-target sequence according to a natural language instruction by utilizing a high-level planning review module based on a large language model, and continuously and dynamically correcting sub-targets in combination with real-time semantic feedback of a current environment; a middle-layer semantic search module is utilized to construct a multi-layer instance semantic map, and ordered search and accurate positioning of all related instances are realized through the multi-layer instance semantic map and common sense reasoning driving; a low-layer action error correction module is used for receiving the RGB image of the current view angle of the intelligent agent in real time, the feasibility of the next preset action is judged, and if potential failure actions are found, the potential failure actions are automatically replaced with safer and more appropriate actions, so that execution failure or collision is avoided. The task success rate and the adaptive capacity of the intelligent agent for executing the natural language instruction in the complex three-dimensional environment can be improved.

The invention provides a large model reasoning system, method and equipment based on Monte Carlo tree search, and relates to the technical field of data processing. According to the large model reasoning system based on Monte Carlo tree search, adaptive task disassembly is carried out for different tasks to obtain candidate thinking cards, the optimal effective reasoning path corresponding to each candidate thinking card is used as an explicit thinking chain type guiding signal, reasoning is carried out for a to-be-reasoned problem through a large model, and the reasoning accuracy of the to-be-reasoned problem is improved. And efficient reasoning is realized through cooperative operation between the two models. The system is expected to improve the performance of a large model in a complex reasoning task, and overcomes the limitation of the current model in the aspects of mathematics, logic and common sense reasoning.

The invention discloses a robot common sense reasoning method based on a graph neural network and a spatial knowledge graph, and the method specifically comprises the steps: S1, constructing an indoor common sense knowledge graph composed of entities and relationships through collecting and sorting common sense in an indoor home environment scene; s2, constructing an instance knowledge graph by using a visual relationship recognition method, and generating a triple from an article to a spatial relationship to an article in real time aiming at an actual article spatial relationship existing in a robot operation environment; s3, learning knowledge representation by using a propagation aggregation method in the graph neural network, representing entities and relationships in the knowledge graph into a continuous feature space, and realizing semantic representation of the entities and relationships in the spatial domain knowledge graph; and S4, reasoning indoor common sense by utilizing spatial domain knowledge graph representation, or designing related entity query, and reasoning similar entities by utilizing entity representation in the graph neural network model. By introducing a propagation and aggregation method of a graph neural network, the efficiency of common sense reasoning is improved.

The invention discloses a method, a system and a device for designing a large model reasoning path verifier based on a preference tree. Aiming at the application of a large language model in a complex reasoning task, the invention provides a novel reasoning path verifier, namely a preference tree-based verifier. According to the method, a preference tree is constructed through a best-first search algorithm, and step-level pairing data is collected to perform preference learning training on a verifier, so that the problem that an existing verifier is only trained on a binary annotation reasoning path and cannot fully utilize a relative relationship of intermediate steps is solved. Compared with a traditional binary classification training method, the verifier carefully captures nuisance between reasoning steps through step-level preference, and therefore the whole reasoning path is evaluated more accurately. The method effectively improves the accuracy and reliability of the verifier, especially has excellent performance in arithmetic and common sense reasoning tasks, and can be further used for assisting math teaching and other tasks.

A method for visual commonsense reasoning (VCR) to infer information from an image is provided. The method may separate a VCR matter into a visual commonsense understanding (VCU) matter and a visual commonsense inference (VCI) matter. The method may provide a visual content of the image using a VCU model. The method may provide conclusions based on content of the image using a VCI model.

The invention discloses a tactile common sense reasoning-oriented tactile-language large model modeling method. The method comprises the steps of S1, preparing a first data set and a second data set; s2, constructing a tactile-language large model, wherein the tactile-language large model comprises a tactile encoder, a tactile-language adapter and a large language model LLM; s3, performing two-stage training on the tactile-language large model, and training the tactile-language large model by adopting the first data set in a first training stage; in a second training stage, training the tactile-language large model after the first training stage by adopting a second data set to obtain a trained tactile-language large model; and S4, in a test stage, inputting the second data set into the trained tactile-language large model to obtain text description. According to the tactile-language large model modeling method, the parameter coupling risk is reduced, robust fusion of efficient convergence and cross-modal representation is realized, and the training stability is remarkably improved.

The invention provides an irony detection method and system based on common sense reasoning, and relates to the technical field of natural language processing, and the method comprises the steps: obtaining an entity of an input text, searching background content related to the entity, and carrying out the mixed filtering processing of the background content; constructing a retrieval enhancement prompt based on the mixed filtering content, taking the input prompt as the input of a preset large language model, and outputting common sense content related to the input text; constructing a syntactic dependency graph based on the input text and the common sense content; optimizing the syntactic dependency graph by using a graph refining strategy to obtain a refined graph; the method comprises the following steps: carrying out inconsistent reasoning skeleton processing on a refined diagram, carrying out joint representation on obtained inconsistent sub-diagrams and the refined diagram, obtaining inconsistent fusion features as input of a preset irony classifier, optimizing the irony classifier by introducing confrontation contrast learning, and outputting an irony probability distribution result by using the optimized irony classifier. The irony detection accuracy is effectively improved.

An exemplary method for detecting deepfake images and providing customized analysis comprises: receiving, from a user, a textual user inquiry regarding an image; inputting the textual inquiry and the image into a deepfake detection model, wherein the deepfake detection model comprises: an image encoder for generating a plurality of image embeddings based on the image; a text encoder for generating a plurality of textual embeddings based on the textual inquiry; one or more layers for generating a plurality of answer embeddings; and a language model for generating a textual analysis based on the plurality of answer embeddings; and outputting the textual analysis, wherein the textual analysis includes a classification result of whether the image is fake and further includes one or more visual features in the image and one or more attributes of the one or more visual features that contribute to the classification result.

The invention belongs to the technical field of image restoration and computer vision, and particularly relates to a multi-mode face restoration and expression recognition system and method based on semantic guidance of a facial action unit. According to the system, multi-scale features are extracted through visual coding, and an AU activation probability is detected by using a graph neural network; the semantic conversion module converts the numerical probability into an interpretable biomechanical structured text; the multi-modal reasoning module fuses vision and text information, introduces the common sense reasoning ability of a multi-modal large language model, and improves the student network performance through knowledge distillation; and finally, the conditional generation module realizes image restoration by taking the semantic features as guidance. The facial action unit is used as a biomechanical medium, the multi-modal reasoning ability is converted into restoration constraint, the defects that in the prior art, restoration of physiology is distorted, recognition depends on image quality, and two tasks are isolated are overcome, collaborative enhancement of face restoration and expression recognition is achieved, and it is ensured that the restoration result is clear in vision and conforms to the physiological law of facial muscles.

The invention discloses a knowledge learning method for intelligent machine common sense reasoning. The method comprises the following steps of 1, normalizing common sense reasoning data; 2, constructing a combination normal form; step 3, multi-dimensional feature common knowledge learning; and 4, model testing and common sense reasoning. According to the method, the problems that a traditional common sense reasoning method only depends on learned shallow semantics, deep common sense semantics contained in sentences are difficult to master and the like can be solved, and common sense knowledge is captured by utilizing a combined normal form and multi-dimensional features, so that the common sense reasoning capability of the model is enhanced.

The embodiment of the present application discloses a government information recommendation method and device that integrates knowledge graph and multimodality, belonging to the field of information recommendation technology, and solving the problem of low matching between recommendation results and needs when users obtain government information. The method includes generating a user preference vector based on the user's historical search behavior, performing a coarse-grained search on the government knowledge graph to obtain a candidate knowledge subgraph; fusing the candidate knowledge subgraph with the multimodal data uploaded by the user to obtain a multimodal semantic vector; performing multi-hop information aggregation processing on the multimodal semantic vector, and performing a fine-grained search based on the aggregated graph node vector and the user query vector; reordering the search results based on a reordering mechanism to determine the recommended government knowledge subgraph; performing visual question answering and visual common sense reasoning processing based on the multimodal data and the recommended government knowledge subgraph to obtain a multimodal reasoning result, and sending the multimodal reasoning result to the user as recommended content based on the user's needs.

The invention mainly provides a text common sense reasoning method for selecting an expert model based on dynamic top-k, which comprises a data set construction method, a model training method and a model reasoning method, so that a lower video memory usage amount and a better model output effect can be achieved in a model reasoning process; and the model is more intelligent under the condition of lower resource consumption. Specifically, different text data needs to be firstly obtained and analyzed, and a data set is formed through recognition; when the model is trained, additionally assigning loss so that the model can realize load balancing and other capacities; determining the number (k value) of experts needing to be called by each layer by using the idea of reinforcement learning; and finally, designing a model reasoning step based on the method for dynamically allocating the experts.

The invention particularly relates to an unknown environment zero sample object target navigation method based on common sense reasoning, which comprises the following steps of: firstly, extracting a scene semantic tag by utilizing a pre-trained open vocabulary target detection model, and constructing a global spatial memory map to realize spatial representation and memory; and meanwhile, extracting global context information in the global spatial memory map based on a large language model so as to infer the position of a target object in an unknown area, and finally outputting a robot navigation action through path planning until the robot finds the target object. According to the method, the robot can adapt to a new object and a new environment without additional training, and the robot can understand semantic information of a target object category in an unknown environment and find an instance of the target object category without seeing a specific instance of the target object, so that the target object category can be identified. The problems that an existing object target navigation method depends on a large amount of labeled training data for training, the sample efficiency is low, and the generalization ability is poor are solved.

The invention provides a Chinese statement error correction method and system based on semantic recognition, and relates to the technical field of natural language processing, and the method comprises the steps: obtaining a statement to be corrected; constructing a semantic graph of the statement to be corrected; taking the semantic graph nodes as retrieval targets, and retrieving and constructing common sense sub-graphs of the semantic graph; graph energy describing semantic consistency between the semantic graph nodes and the related common sense sub-graph nodes is calculated, and the graph energy and the semantic consistency are in negative correlation; under the condition that the graph energy is greater than preset graph energy, positioning error nodes in the statement to be corrected through a gradient deviation detection algorithm in combination with semantic consistency; and error correction is carried out on error nodes by combining the common sense sub-graph and graph energy. According to the Chinese statement error correction method based on semantic recognition, the deep meaning of a sentence can be more accurately understood by introducing a semantic graph and a common sense sub-graph, and the defects of a traditional method in the aspects of context understanding and common sense reasoning are overcome.

This invention discloses a visual commonsense reasoning method and system based on graph attention networks, relating to the interdisciplinary fields of computer vision and natural language processing. The method includes: training a constructed visual commonsense reasoning model using multiple sets of corresponding sample images, sample questions, and labeled sample answer candidates as a training set; the training process includes: acquiring visual and linguistic features of the sample images, sample questions, and sample answer candidates; using visual features as visual nodes and updating visual features using semantic information based on a graph attention mechanism; employing multimodal fusion to fuse linguistic features and updated visual features to obtain a joint visual and linguistic feature representation, thereby training the visual commonsense reasoning model; inputting the image to be reasoned, the question to be reasoned, and multiple reasoning answer candidates into the trained visual commonsense reasoning model, outputting the reasoning result, achieving accurate reasoning, and improving the performance of visual commonsense reasoning tasks.

This invention belongs to the field of natural language processing and dialogue emotion recognition, specifically involving an emotion classification method based on a hybrid expert model and a large-scale model collaboration. The method first encodes the dialogue text to construct basic features. Then, it extracts semantic, contextual, and knowledge-enhanced features through heterogeneous expert networks, and uses a dynamic routing gating network to weightedly fuse the features output by different experts, forming a unified emotion feature representation. Next, this feature representation is mapped to a soft cue vector, concatenated with the word embedding sequence of the original text, and input into a pre-trained large-scale model for emotion inference. During the training phase, cross-entropy loss is used as the target, updating only the parameters of the expert network, routing gating network, and soft cue mapping network, achieving efficient end-to-end parameter training. This invention significantly improves the accuracy of emotion classification by fusing multi-dimensional features and combining the common-sense reasoning ability of a large-scale model, while also achieving higher training efficiency.

The application relates to the technical field of artificial intelligence, and discloses a text emotion deep reason recognition method and device based on common sense reasoning, which comprises the following steps: first, potential emotional clauses and potential reason clauses in the text are recognized, and corresponding emotional clause graphs and reason clause graphs are constructed; then, emotional and causal common sense related to the clauses is searched to expand the graph structure, and two graph neural networks are obtained to represent the context semantics and common sense knowledge of the emotional and reason graphs; and an interactive attention mechanism is used to find out the emotional-reason combination clause pair with the largest correlation, and further decode the deep reason with a strong causal relationship and the corresponding emotion. Since the text grammar structure is fully depicted and the related external common sense is fused, the application can better decode the deep emotional reason, realize the mining of the potential deep emotional reason behind the surface emotional reason event of the text, and effectively improve the accuracy and reliability of the emotional reason inference.

The invention relates to a model training method, a vehicle control method, a device, equipment, a medium and a product, and the method comprises the steps: generating a plurality of second control instructions according to the scene information (sample driving information and sample user information) of a sample vehicle, a first control instruction and the powerful common sense reasoning capability of a large model; the scene information and the first control instruction serve as training samples, the multiple second control instructions serve as labels, model training is carried out, and a control instruction generation model is generated; and generating the capabilities of a plurality of target control instructions associated with the functions corresponding to the initial control instruction. In practical application, only one control instruction is needed, some vehicle control operations can be completed, equipment which is not known by a user in the vehicle can be controlled, and the efficiency and comprehensiveness of vehicle control are improved.

The present invention discloses a multi-task learning model that combines image-text matching and visual reasoning, a visual common sense reasoning method, and a computer device. In the visual common sense reasoning part, a pre-trained model is used to extract the features of images and texts, obtaining a joint representation of the visual modality and the text modality, and then a multi-class cross-entropy loss function is used for optimization training; then the original visual common sense reasoning dataset is processed for use in the image-text matching module; in the image-text matching part, the pixel-level features of the image are first extracted as global features, and the region-level features of the image are extracted as local features, and then the image-text matching is optimized and trained through a contrastive learning loss function; finally, two-way promotion of image-text matching and visual common sense reasoning is achieved through parameter sharing. The present invention improves the model's ability to jointly reason about diverse visual content and advanced text semantics, enabling the model to better align the two modalities. Through the two-way promotion of image-text matching and visual reasoning, the performance of the visual common sense reasoning task is improved.

An exemplary method for detecting deepfake images and providing customized analysis comprises: receiving, from a user, a textual user inquiry regarding an image; inputting the textual inquiry and the image into a deepfake detection model, wherein the deepfake detection model comprises: an image encoder for generating a plurality of image embeddings based on the image; a text encoder for generating a plurality of textual embeddings based on the textual inquiry; one or more layers for generating a plurality of answer embeddings; and a language model for generating a textual analysis based on the plurality of answer embeddings; and outputting the textual analysis, wherein the textual analysis includes a classification result of whether the image is fake and further includes one or more visual features in the image and one or more attributes of the one or more visual features that contribute to the classification result.

The invention relates to the technical field of standing field detection and language processing, in particular to a standing field detection method and system based on common sense reasoning and an MMOE framework, and the method comprises the following steps: introducing a standing field detection text set and an emotion detection data set, setting a COMET common sense reasoning model to analyze the standing field detection text set to obtain a sentence set with different relations, obtaining an emotion detection word and sentence set based on the emotion detection data set; setting an encoder to process the different relation sentence set and the emotion detection word and sentence set to obtain a feature vector e1 of the different relation sentence set and a feature vector e2 of the emotion detection word and sentence set; establishing an expert model and a gating network mechanism to respectively process the feature vector e1 and the feature vector e2 and obtain weighted vector results of different feature vectors, and decoding the weighted vector results to obtain a vertical vector; and constructing a loss function to analyze a loss value of the vertical vector, and updating parameters of the encoder, the expert network and the gating network based on the loss value to obtain a vertical detection result.

Disclosed in the present invention is a knowledge learning method for commonsense reasoning in an intelligent machine. The method comprises the following steps: step 1, normalizing commonsense reasoning data; step 2, constructing a combination paradigm; step 3, performing multi-dimensional feature commonsense knowledge learning; and step 4, performing model testing and commonsense reasoning.

The present invention discloses a generative visual common sense reasoning and explanation method based on scene graph generation. The method includes: adding a single-head attention and gating fusion mechanism to the language large model Flan-T5 for multimodal fusion; in the scene graph generation stage, using an image patch sequence and a prompt template containing object information to generate a scene graph triple sequence; in the visual common sense reasoning and explanation generation stage, combining the scene graph with the question to construct a prompt template, and designing an automatic scene graph selection module based on confidence scores, enabling the model to automatically select appropriate triples and generate answers and explanations. Better metric scores are obtained on the VCR dataset than models such as e-UG. The present invention addresses the problem of the single modality of the language large model, combines multimodal fusion techniques, and through constructing a prompt template and a confidence-based scene graph selection mechanism, gives full play to the semantic understanding characteristics of the large model, endows it with the ability to deeply understand visual information, and thus conducts reasoning and explanation.

The invention provides a multi-modal question inconsistency detection method and system based on common sense reasoning, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining knowledge clues related to post multi-modal content; constructing a deep common sense reasoning question for verifying the authenticity of the knowledge clues, answering the common sense reasoning question based on the post multi-modal content to obtain a first answer, and performing open domain question answering on the common sense reasoning question to obtain a second answer; comparing whether the first answer and the second answer are consistent or not, and if yes, determining that the post does not have contradictions and false parts; if not, determining that the post has contradictions and false parts, and obtaining all potential inconsistent features in the post through answer comparison; extracting the multi-modal features of the posts with contradictions and false parts, fusing the multi-modal features and the inconsistency features to obtain fused features, inputting the fused features into a preset question inconformity label prediction model, and outputting question inconformity labels of the posts.