52 results about "Commonsense reasoning" patented technology

The invention provides a natural language control method for a humanoid robot. By constructing the simple skill library and the complex skill library of the robot, the dependence on a large amount of operation data in a specific field is effectively reduced. According to the method, a large language model is used as a base of a task planner, and the capabilities of scene learning, semantic understanding and common sense reasoning of the task planner are brought into full play. By introducing a visual model in the field of computer vision, high-level semantic information and low-level spatial information of the environment can be obtained, and an accurate scene graph is constructed to serve as environment state representation. According to the voice interaction interface constructed by the invention, the naturalness of man-machine interaction is improved, so that a user can more conveniently communicate with the robot and input instructions, the user experience is effectively improved, and popularization of the humanoid robot in an actual application scene is facilitated.

An exemplary method for detecting deepfake images and providing customized analysis comprises: receiving, from a user, a textual user inquiry regarding an image; inputting the textual inquiry and the image into a deepfake detection model, wherein the deepfake detection model comprises: an image encoder for generating a plurality of image embeddings based on the image; a text encoder for generating a plurality of textual embeddings based on the textual inquiry; one or more layers for generating a plurality of answer embeddings; and a language model for generating a textual analysis based on the plurality of answer embeddings; and outputting the textual analysis, wherein the textual analysis includes a classification result of whether the image is fake and further includes one or more visual features in the image and one or more attributes of the one or more visual features that contribute to the classification result.

Embodiments of the present disclosure relate to fine-tuning a neural network model using a weight-decomposed low-rank adaptation (DoRA). DoRA reduces the number of parameters that are fine-tuned, thereby reducing memory and the time needed to fine-tune the parameters. the Pre-trained weights are decomposed into two components, magnitude and direction, which are separately fine-tuned. The magnitude components are fine-tuned while the direction components remain unchanged (frozen). Then low-rank adaptation (LoRA) is used to fine-tune the direction components, efficiently minimizing the number of trainable parameters. Compared with using LoRA to fine-tune the weights directly, using DoRA exhibits a closer resemblance to full fine-tuning's learning behavior and improves upon LoRA in commonsense reasoning and visual instruction tuning tasks. By employing DoRA, both the learning capacity and training stability of LoRA is enhanced. The fine-tuned decomposed magnitude and direction components may be merged into the pre-trained weights to avoid any additional inference overhead.

The embodiment of the invention discloses a knowledge graph and multi-modal fusion government affair information recommendation method and equipment, belongs to the technical field of information recommendation, and solves the problem that the matching degree between a recommendation result and a demand is relatively low when a user obtains government affair information. Comprising the steps of generating a user preference vector based on a user historical retrieval behavior, and performing coarse-grained retrieval on a government affair knowledge graph to obtain candidate knowledge sub-graphs; fusing the candidate knowledge sub-graphs with multi-modal data uploaded by the user to obtain a multi-modal semantic vector; performing multi-hop information aggregation processing on the multi-modal semantic vector, and performing fine-grained retrieval based on an aggregated graph node vector and a user query vector; based on a reordering mechanism, reordering the retrieval results, and determining recommended government affair knowledge sub-graphs; and based on the multi-modal data and the recommended government affair knowledge subgraph, performing visual question and answer and visual common sense reasoning processing to obtain a multi-modal reasoning result, and based on user requirements, sending the multi-modal reasoning result as recommended content to the user.

The invention belongs to the field of intelligent planning control, and relates to an intelligent agent instruction execution method and system based on multi-level re-planning. The method comprises the following steps: preliminarily generating a sub-target sequence according to a natural language instruction by utilizing a high-level planning review module based on a large language model, and continuously and dynamically correcting sub-targets in combination with real-time semantic feedback of a current environment; a middle-layer semantic search module is utilized to construct a multi-layer instance semantic map, and ordered search and accurate positioning of all related instances are realized through the multi-layer instance semantic map and common sense reasoning driving; a low-layer action error correction module is used for receiving the RGB image of the current view angle of the intelligent agent in real time, the feasibility of the next preset action is judged, and if potential failure actions are found, the potential failure actions are automatically replaced with safer and more appropriate actions, so that execution failure or collision is avoided. The task success rate and the adaptive capacity of the intelligent agent for executing the natural language instruction in the complex three-dimensional environment can be improved.

The invention provides a large model reasoning system, method and equipment based on Monte Carlo tree search, and relates to the technical field of data processing. According to the large model reasoning system based on Monte Carlo tree search, adaptive task disassembly is carried out for different tasks to obtain candidate thinking cards, the optimal effective reasoning path corresponding to each candidate thinking card is used as an explicit thinking chain type guiding signal, reasoning is carried out for a to-be-reasoned problem through a large model, and the reasoning accuracy of the to-be-reasoned problem is improved. And efficient reasoning is realized through cooperative operation between the two models. The system is expected to improve the performance of a large model in a complex reasoning task, and overcomes the limitation of the current model in the aspects of mathematics, logic and common sense reasoning.

The invention discloses a robot common sense reasoning method based on a graph neural network and a spatial knowledge graph, and the method specifically comprises the steps: S1, constructing an indoor common sense knowledge graph composed of entities and relationships through collecting and sorting common sense in an indoor home environment scene; s2, constructing an instance knowledge graph by using a visual relationship recognition method, and generating a triple from an article to a spatial relationship to an article in real time aiming at an actual article spatial relationship existing in a robot operation environment; s3, learning knowledge representation by using a propagation aggregation method in the graph neural network, representing entities and relationships in the knowledge graph into a continuous feature space, and realizing semantic representation of the entities and relationships in the spatial domain knowledge graph; and S4, reasoning indoor common sense by utilizing spatial domain knowledge graph representation, or designing related entity query, and reasoning similar entities by utilizing entity representation in the graph neural network model. By introducing a propagation and aggregation method of a graph neural network, the efficiency of common sense reasoning is improved.

A method for visual commonsense reasoning (VCR) to infer information from an image is provided. The method may separate a VCR matter into a visual commonsense understanding (VCU) matter and a visual commonsense inference (VCI) matter. The method may provide a visual content of the image using a VCU model. The method may provide conclusions based on content of the image using a VCI model.

The invention discloses a tactile common sense reasoning-oriented tactile-language large model modeling method. The method comprises the steps of S1, preparing a first data set and a second data set; s2, constructing a tactile-language large model, wherein the tactile-language large model comprises a tactile encoder, a tactile-language adapter and a large language model LLM; s3, performing two-stage training on the tactile-language large model, and training the tactile-language large model by adopting the first data set in a first training stage; in a second training stage, training the tactile-language large model after the first training stage by adopting a second data set to obtain a trained tactile-language large model; and S4, in a test stage, inputting the second data set into the trained tactile-language large model to obtain text description. According to the tactile-language large model modeling method, the parameter coupling risk is reduced, robust fusion of efficient convergence and cross-modal representation is realized, and the training stability is remarkably improved.

The invention provides an irony detection method and system based on common sense reasoning, and relates to the technical field of natural language processing, and the method comprises the steps: obtaining an entity of an input text, searching background content related to the entity, and carrying out the mixed filtering processing of the background content; constructing a retrieval enhancement prompt based on the mixed filtering content, taking the input prompt as the input of a preset large language model, and outputting common sense content related to the input text; constructing a syntactic dependency graph based on the input text and the common sense content; optimizing the syntactic dependency graph by using a graph refining strategy to obtain a refined graph; the method comprises the following steps: carrying out inconsistent reasoning skeleton processing on a refined diagram, carrying out joint representation on obtained inconsistent sub-diagrams and the refined diagram, obtaining inconsistent fusion features as input of a preset irony classifier, optimizing the irony classifier by introducing confrontation contrast learning, and outputting an irony probability distribution result by using the optimized irony classifier. The irony detection accuracy is effectively improved.

An exemplary method for detecting deepfake images and providing customized analysis comprises: receiving, from a user, a textual user inquiry regarding an image; inputting the textual inquiry and the image into a deepfake detection model, wherein the deepfake detection model comprises: an image encoder for generating a plurality of image embeddings based on the image; a text encoder for generating a plurality of textual embeddings based on the textual inquiry; one or more layers for generating a plurality of answer embeddings; and a language model for generating a textual analysis based on the plurality of answer embeddings; and outputting the textual analysis, wherein the textual analysis includes a classification result of whether the image is fake and further includes one or more visual features in the image and one or more attributes of the one or more visual features that contribute to the classification result.

The invention belongs to the technical field of image restoration and computer vision, and particularly relates to a multi-mode face restoration and expression recognition system and method based on semantic guidance of a facial action unit. According to the system, multi-scale features are extracted through visual coding, and an AU activation probability is detected by using a graph neural network; the semantic conversion module converts the numerical probability into an interpretable biomechanical structured text; the multi-modal reasoning module fuses vision and text information, introduces the common sense reasoning ability of a multi-modal large language model, and improves the student network performance through knowledge distillation; and finally, the conditional generation module realizes image restoration by taking the semantic features as guidance. The facial action unit is used as a biomechanical medium, the multi-modal reasoning ability is converted into restoration constraint, the defects that in the prior art, restoration of physiology is distorted, recognition depends on image quality, and two tasks are isolated are overcome, collaborative enhancement of face restoration and expression recognition is achieved, and it is ensured that the restoration result is clear in vision and conforms to the physiological law of facial muscles.

The embodiment of the present application discloses a government information recommendation method and device that integrates knowledge graph and multimodality, belonging to the field of information recommendation technology, and solving the problem of low matching between recommendation results and needs when users obtain government information. The method includes generating a user preference vector based on the user's historical search behavior, performing a coarse-grained search on the government knowledge graph to obtain a candidate knowledge subgraph; fusing the candidate knowledge subgraph with the multimodal data uploaded by the user to obtain a multimodal semantic vector; performing multi-hop information aggregation processing on the multimodal semantic vector, and performing a fine-grained search based on the aggregated graph node vector and the user query vector; reordering the search results based on a reordering mechanism to determine the recommended government knowledge subgraph; performing visual question answering and visual common sense reasoning processing based on the multimodal data and the recommended government knowledge subgraph to obtain a multimodal reasoning result, and sending the multimodal reasoning result to the user as recommended content based on the user's needs.

The invention mainly provides a text common sense reasoning method for selecting an expert model based on dynamic top-k, which comprises a data set construction method, a model training method and a model reasoning method, so that a lower video memory usage amount and a better model output effect can be achieved in a model reasoning process; and the model is more intelligent under the condition of lower resource consumption. Specifically, different text data needs to be firstly obtained and analyzed, and a data set is formed through recognition; when the model is trained, additionally assigning loss so that the model can realize load balancing and other capacities; determining the number (k value) of experts needing to be called by each layer by using the idea of reinforcement learning; and finally, designing a model reasoning step based on the method for dynamically allocating the experts.

The invention particularly relates to an unknown environment zero sample object target navigation method based on common sense reasoning, which comprises the following steps of: firstly, extracting a scene semantic tag by utilizing a pre-trained open vocabulary target detection model, and constructing a global spatial memory map to realize spatial representation and memory; and meanwhile, extracting global context information in the global spatial memory map based on a large language model so as to infer the position of a target object in an unknown area, and finally outputting a robot navigation action through path planning until the robot finds the target object. According to the method, the robot can adapt to a new object and a new environment without additional training, and the robot can understand semantic information of a target object category in an unknown environment and find an instance of the target object category without seeing a specific instance of the target object, so that the target object category can be identified. The problems that an existing object target navigation method depends on a large amount of labeled training data for training, the sample efficiency is low, and the generalization ability is poor are solved.

The invention provides a Chinese statement error correction method and system based on semantic recognition, and relates to the technical field of natural language processing, and the method comprises the steps: obtaining a statement to be corrected; constructing a semantic graph of the statement to be corrected; taking the semantic graph nodes as retrieval targets, and retrieving and constructing common sense sub-graphs of the semantic graph; graph energy describing semantic consistency between the semantic graph nodes and the related common sense sub-graph nodes is calculated, and the graph energy and the semantic consistency are in negative correlation; under the condition that the graph energy is greater than preset graph energy, positioning error nodes in the statement to be corrected through a gradient deviation detection algorithm in combination with semantic consistency; and error correction is carried out on error nodes by combining the common sense sub-graph and graph energy. According to the Chinese statement error correction method based on semantic recognition, the deep meaning of a sentence can be more accurately understood by introducing a semantic graph and a common sense sub-graph, and the defects of a traditional method in the aspects of context understanding and common sense reasoning are overcome.

This invention discloses a visual commonsense reasoning method and system based on graph attention networks, relating to the interdisciplinary fields of computer vision and natural language processing. The method includes: training a constructed visual commonsense reasoning model using multiple sets of corresponding sample images, sample questions, and labeled sample answer candidates as a training set; the training process includes: acquiring visual and linguistic features of the sample images, sample questions, and sample answer candidates; using visual features as visual nodes and updating visual features using semantic information based on a graph attention mechanism; employing multimodal fusion to fuse linguistic features and updated visual features to obtain a joint visual and linguistic feature representation, thereby training the visual commonsense reasoning model; inputting the image to be reasoned, the question to be reasoned, and multiple reasoning answer candidates into the trained visual commonsense reasoning model, outputting the reasoning result, achieving accurate reasoning, and improving the performance of visual commonsense reasoning tasks.

This invention belongs to the field of natural language processing and dialogue emotion recognition, specifically involving an emotion classification method based on a hybrid expert model and a large-scale model collaboration. The method first encodes the dialogue text to construct basic features. Then, it extracts semantic, contextual, and knowledge-enhanced features through heterogeneous expert networks, and uses a dynamic routing gating network to weightedly fuse the features output by different experts, forming a unified emotion feature representation. Next, this feature representation is mapped to a soft cue vector, concatenated with the word embedding sequence of the original text, and input into a pre-trained large-scale model for emotion inference. During the training phase, cross-entropy loss is used as the target, updating only the parameters of the expert network, routing gating network, and soft cue mapping network, achieving efficient end-to-end parameter training. This invention significantly improves the accuracy of emotion classification by fusing multi-dimensional features and combining the common-sense reasoning ability of a large-scale model, while also achieving higher training efficiency.

The application relates to the technical field of artificial intelligence, and discloses a text emotion deep reason recognition method and device based on common sense reasoning, which comprises the following steps: first, potential emotional clauses and potential reason clauses in the text are recognized, and corresponding emotional clause graphs and reason clause graphs are constructed; then, emotional and causal common sense related to the clauses is searched to expand the graph structure, and two graph neural networks are obtained to represent the context semantics and common sense knowledge of the emotional and reason graphs; and an interactive attention mechanism is used to find out the emotional-reason combination clause pair with the largest correlation, and further decode the deep reason with a strong causal relationship and the corresponding emotion. Since the text grammar structure is fully depicted and the related external common sense is fused, the application can better decode the deep emotional reason, realize the mining of the potential deep emotional reason behind the surface emotional reason event of the text, and effectively improve the accuracy and reliability of the emotional reason inference.

The invention relates to a model training method, a vehicle control method, a device, equipment, a medium and a product, and the method comprises the steps: generating a plurality of second control instructions according to the scene information (sample driving information and sample user information) of a sample vehicle, a first control instruction and the powerful common sense reasoning capability of a large model; the scene information and the first control instruction serve as training samples, the multiple second control instructions serve as labels, model training is carried out, and a control instruction generation model is generated; and generating the capabilities of a plurality of target control instructions associated with the functions corresponding to the initial control instruction. In practical application, only one control instruction is needed, some vehicle control operations can be completed, equipment which is not known by a user in the vehicle can be controlled, and the efficiency and comprehensiveness of vehicle control are improved.

The present invention discloses a multi-task learning model that combines image-text matching and visual reasoning, a visual common sense reasoning method, and a computer device. In the visual common sense reasoning part, a pre-trained model is used to extract the features of images and texts, obtaining a joint representation of the visual modality and the text modality, and then a multi-class cross-entropy loss function is used for optimization training; then the original visual common sense reasoning dataset is processed for use in the image-text matching module; in the image-text matching part, the pixel-level features of the image are first extracted as global features, and the region-level features of the image are extracted as local features, and then the image-text matching is optimized and trained through a contrastive learning loss function; finally, two-way promotion of image-text matching and visual common sense reasoning is achieved through parameter sharing. The present invention improves the model's ability to jointly reason about diverse visual content and advanced text semantics, enabling the model to better align the two modalities. Through the two-way promotion of image-text matching and visual reasoning, the performance of the visual common sense reasoning task is improved.

An exemplary method for detecting deepfake images and providing customized analysis comprises: receiving, from a user, a textual user inquiry regarding an image; inputting the textual inquiry and the image into a deepfake detection model, wherein the deepfake detection model comprises: an image encoder for generating a plurality of image embeddings based on the image; a text encoder for generating a plurality of textual embeddings based on the textual inquiry; one or more layers for generating a plurality of answer embeddings; and a language model for generating a textual analysis based on the plurality of answer embeddings; and outputting the textual analysis, wherein the textual analysis includes a classification result of whether the image is fake and further includes one or more visual features in the image and one or more attributes of the one or more visual features that contribute to the classification result.

Disclosed in the present invention is a knowledge learning method for commonsense reasoning in an intelligent machine. The method comprises the following steps: step 1, normalizing commonsense reasoning data; step 2, constructing a combination paradigm; step 3, performing multi-dimensional feature commonsense knowledge learning; and step 4, performing model testing and commonsense reasoning.

The present invention discloses a generative visual common sense reasoning and explanation method based on scene graph generation. The method includes: adding a single-head attention and gating fusion mechanism to the language large model Flan-T5 for multimodal fusion; in the scene graph generation stage, using an image patch sequence and a prompt template containing object information to generate a scene graph triple sequence; in the visual common sense reasoning and explanation generation stage, combining the scene graph with the question to construct a prompt template, and designing an automatic scene graph selection module based on confidence scores, enabling the model to automatically select appropriate triples and generate answers and explanations. Better metric scores are obtained on the VCR dataset than models such as e-UG. The present invention addresses the problem of the single modality of the language large model, combines multimodal fusion techniques, and through constructing a prompt template and a confidence-based scene graph selection mechanism, gives full play to the semantic understanding characteristics of the large model, endows it with the ability to deeply understand visual information, and thus conducts reasoning and explanation.

The invention provides a multi-modal question inconsistency detection method and system based on common sense reasoning, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining knowledge clues related to post multi-modal content; constructing a deep common sense reasoning question for verifying the authenticity of the knowledge clues, answering the common sense reasoning question based on the post multi-modal content to obtain a first answer, and performing open domain question answering on the common sense reasoning question to obtain a second answer; comparing whether the first answer and the second answer are consistent or not, and if yes, determining that the post does not have contradictions and false parts; if not, determining that the post has contradictions and false parts, and obtaining all potential inconsistent features in the post through answer comparison; extracting the multi-modal features of the posts with contradictions and false parts, fusing the multi-modal features and the inconsistency features to obtain fused features, inputting the fused features into a preset question inconformity label prediction model, and outputting question inconformity labels of the posts.

The application belongs to the technical field of service software, and discloses an AI-based service software system and method, which comprises the following steps: a data collection module collects multi-source data; a data processing module pre-processes the data; a data privacy protection module encrypts sensitive information at each link; a model construction module constructs a model according to specific application tasks and data characteristics; a model optimization module optimizes the model; an algorithm strengthening module strengthens the model algorithm; an external interaction module realizes seamless integration of the AI system and a traditional database; a performance monitoring module monitors the system performance in real time; and a security protection module timely repairs discovered security vulnerabilities. The application can improve data quality; through model compression, hardware adaptation, explainability enhancement and generalization ability improvement of the model optimization module, the model running efficiency can be improved; and the algorithm strengthening module can realize the technical effects of counter-attack defense and common sense reasoning enhancement on the model algorithm.

The invention belongs to the technical field of service software, and discloses an AI-based service software system and method, and the method comprises the steps: collecting multi-source data through a data collection module; the data processing module preprocesses the data; the data privacy protection module encrypts sensitive information of each link; the model construction module constructs a model according to specific application tasks and data characteristics; the model optimization module is used for optimizing the model; the algorithm strengthening module strengthens the model algorithm; the external interaction module realizes seamless joint when the AI system is integrated with the traditional database; the performance monitoring module monitors system performance in real time; and the security protection module timely repairs the found security vulnerabilities. The data quality can be improved; the model is subjected to optimization such as model compression, hardware adaptation, interpretability enhancement and generalization ability improvement through the model optimization module, the model operation efficiency can be improved, and the technical effects that the algorithm enhancement module carries out attack defense and common sense reasoning enhancement on the model algorithm are achieved.

The invention discloses a big language model common sense reasoning enhancement method and device based on knowledge filtering. The reasoning enhancement method comprises the following steps: acquiring a data set comprising a plurality of questions aiming at common knowledge and a plurality of corresponding answers; inputting the plurality of questions into a large language model based on knowledge filtering to obtain corresponding common knowledge so as to construct a first question-knowledge pair; by evaluating the influence degree of the corresponding common knowledge on the corresponding answers of the multiple questions, classifying the first question-knowledge pair and generating a corresponding level label, and obtaining second question-knowledge pair data containing the level label; inputting the second question-knowledge pair data and the data set into a preset reward model to obtain corresponding reward value output; and according to the reward value output, adjusting each parameter of the preset reward model to obtain a trained reward model so as to realize reasoning enhancement of the large language model based on knowledge filtering.

The application discloses a visual common sense reasoning method and system based on a knowledge-aware attention network, relates to the cross technical field of computer vision and natural language processing, and comprises the following steps: training a visual common sense reasoning model by using multiple groups of sample images, sample questions and annotated sample answer candidates; the training process comprises the following steps: acquiring visual features and initial context sentence representations of the sample images, the sample questions and the sample answer candidates; learning knowledge-based sentence representations and knowledge-aware attention vectors of the sample questions and the sample answer candidates according to visual context guidance and knowledge embedding; fusing the knowledge-aware attention vectors and the visual features to obtain fused features, so as to train the visual common sense reasoning model; inputting an image to be reasoned, a question to be reasoned and multiple reasoning answer candidates into the trained visual common sense reasoning model, and outputting a reasoning result, so that accurate reasoning is realized, and the performance of a visual common sense reasoning task is improved.

The invention discloses a civil and commercial case classification method and system based on machine learning. The method comprises the steps of data acquisition, data preprocessing, case classification model construction, hyper-parameter optimization and civil and commercial case classification. The invention relates to the technical field of civil and commercial case data processing, in particular to a civil and commercial case classification method and system based on machine learning. A data preprocessing method of data cleaning, phrase segmentation and data set segmentation is adopted; a double-interaction self-attention network model is adopted to classify civilian and commercial cases, legal backgrounds and provision details in the cases are deeply understood by combining generated legal common sense reasoning, and the limitation of purely depending on case texts is supplemented; an optimization algorithm is adopted to carry out model hyper-parameter optimization, and hyperbolic tangent chaotic mapping and a Cauchy variation strategy are combined, so that premature convergence in a search process is avoided, and the diversity in the search process is enhanced while the global search capability is improved.