61 results about "Duplicate content" patented technology

Provided in the embodiments of the present disclosure are a task processing method, a text processing method, an automatic question-answering method, a task processing model training method, an information processing method based on a task processing model, and a cloud training platform, which are applied to the technical field of computers. The task processing method comprises: acquiring task data of a target task; and inputting the task data into a task processing model, so as to obtain a task processing result of the target task, wherein the task processing model is obtained by means of performing training on the basis of a plurality of pieces of target sample data and target sample results of the plurality of pieces of target sample data, the target sample data is obtained by means of performing screening on the basis of detection results of repeated content of a plurality of pieces of sample data, and the content of the target sample results is not repeated. Target sample data is obtained by means of performing screening on the basis of detection results of repeated content, and the content of target sample results is not repeated, such that the repetition illusion of a model is reduced without damaging the processing capability of the model itself and relying on external knowledge, thereby improving the accuracy and integrity of a task processing result.

The invention discloses a medical text data synthesis method based on role driving, and relates to the technical field of text data synthesis and artificial intelligence, and the method comprises the following steps: S100, generating an original problem; s200, screening original problems; and S300, synthesizing the medical text data. According to role description and interest content generation, the adaptability to different medical role requirements is enhanced, semantic duplicate removal and correlation evaluation are realized by utilizing a text coding model and a vector database, semantic duplicate content is removed, correlation is accurately evaluated by utilizing a large language model, the data quality and diversity are improved, and the method is suitable for popularization and application. Through generation of variant questions and answers, diversity and practicability of medical text data are enriched. According to the method, the personalization and pertinence of the medical text data are improved, the quality and duplicate removal efficiency of the medical text data are enhanced, the problem types are enriched, the generated content is optimized, and various information requirements of the medical text data are comprehensively covered.

The invention discloses a content interaction method for AI online education, which is applied to a content interaction system for online AI education, and the content interaction system for online AI education comprises a teacher end, a server end and a student end. According to the AI online education content interaction method, through interaction among the teacher terminal, the server terminal and the student terminal, students can play back at any time point through the student terminal, timely and repeatedly listen to important knowledge points, and can also join in a live course again without missing follow-up real-time explanation content; meanwhile, the playback efficiency is improved through the judgment of the wrong playback operation at the student end. According to the playback dynamic state of the student end, the teacher end generates a message prompt of a playback hotspot interval to assist a teacher to adjust an explanation strategy in a targeted manner, so that the content is fully understood and mastered, and the teaching effect is effectively improved; and meanwhile, the repeated content explanation is identified and the classroom rest time period is reminded, so that the time is effectively saved when the student side plays back.

The invention discloses a bidding document duplicate checking method and system based on artificial intelligence, and comprehensive analysis and accurate duplicate checking of bidding document content are realized through multi-modal analysis (text, image and table), a dynamic weight model and a visual report. The method breaks through the limitation of traditional text comparison, and supports similarity detection of non-text elements such as images and tables; capturing deep semantic similarity by adopting a bag-of-word vector and assignment dimension reduction technology; the image matching accuracy is improved through a self-adaptive partitioning algorithm; semantic matching of the table data is achieved based on the attribute item weight; the strategy pertinence is enhanced through dynamic weight distribution; the visual report visually presents repeated contents; the analysis precision is improved through anti-interference preprocessing; and multi-format documents are compatible to meet diversified requirements. According to the method, the comprehensiveness, accuracy and efficiency of duplicate checking of the bidding document are remarkably improved, the problems of complex formats and semantic understanding are effectively solved, and the high requirements of the bidding and tendering field for fairness and originality are met.

The invention relates to the technical field of aeronautical text data processing, and discloses an aeronautical text content cleaning and labeling method, system, device and medium wherein the method comprises: noise filtering: identifying and removing noise in an aeronautical text in combination with static cleaning and a general large model; format standardization: converting the aviation text after noise removal into a standardized format text; duplicate removal and error correction: detecting duplicate contents based on a hash algorithm, and correcting spelling errors and grammar errors based on a general large model to obtain an aviation text subjected to duplicate removal and error correction; entity identification: key entities are extracted based on the general large model, and the extracted key entities are labeled; active learning: screening high-value samples in the marked key entities based on an uncertainty query strategy; and dynamic optimization: performing verification and iterative optimization on the marking result in combination with the aviation knowledge base. According to the method, the automation level, the labeling accuracy and the system self-adaptive capability of aviation text processing can be remarkably improved.

The invention relates to the technical field of semantic analysis, and discloses an intelligent semantic analysis system and method based on a data large model, and the method comprises the steps: converting an audio file into a text, dividing the text into a character group and a word group, numbering the character group and the word group, and extracting repeated content; training a deep learning model to process the character group and the word group to form a standard text; analyzing the audio to determine a language region, obtaining a dialect database and extracting dialect content; combining the dialects, generating a to-be-analyzed text, and performing semantic analysis and classification; and finally, obtaining the expression meaning of the standard text according to the analysis result. The existing marked dialects are combined and substituted into the standard text, so that potential multiple explanations caused by dialect differences in the text can be covered and revealed, all expression meanings existing in the standard text can be obtained, and the process can ensure that any possible expression meaning is not missed, so that the accuracy of the expression meanings is improved. And the final semantic analysis is more comprehensive and accurate.

The invention discloses a science and technology project document duplicate checking method and device based on multi-level abstract generation and a medium, belongs to the technical field of document duplicate checking, and solves the problem of how to improve the reliability and accuracy of science and technology project document duplicate checking. The method comprises the following steps: firstly, carrying out structured analysis on a project document to be subjected to duplicate checking, introducing a fine-tuned abstract generation model to obtain a multi-level abstract of a project to be subjected to duplicate checking, and extracting a feature vector of the multi-level abstract through a fine-tuned and trained text feature extraction model; secondly, performing similarity search in a vector database based on feature vectors corresponding to the dimensions of the main abstract, further calculating cosine similarities based on the dimensions of the subordinate abstract, and sequencing the weighted similarities to obtain a similarity rank; and finally, according to the structured analysis information and the similarity ranking result, LLM similarity analysis is performed by using a large language model, a document duplicate checking report is formed, the originality of the science and technology project is ensured, repeated subsidization is avoided, and the method has important practical application value for science and technology project management.

The embodiment of the invention provides an online man-machine conversation optimization method, system and device, electronic equipment, a storage medium and a program product. In the scheme provided by the embodiment, when a first dialogue generation model executes a streaming response based on user input data, a currently generated first dialogue text fragment is subjected to repeated detection, and when it is detected that repeated content exists in the first dialogue text fragment, the first dialogue generation model is controlled to interrupt execution of the streaming response; in addition, reply generation constraint information is determined according to the repeated content in the first reply text segment, and then reply generation cue words are generated based on the reply generation constraint information and the non-repeated content in the first reply text segment. And after the dialogue generation cue word is input into the first dialogue generation model, the first dialogue generation model is triggered to determine a dialogue generation starting point according to the non-repeated content in the first dialogue text fragment, and streaming response is continuously executed from the dialogue generation starting point.

In some aspects, a content-extraction system can receive a query from a client device and generate a result set of digital content responsive to the query. For instance, the content-extraction system can obtain, from a search system, a set of digital content matching one or more keywords. The content-extraction system can exclude digital content items lacking core content, digital content items with duplicative content, or both. In some aspects, the content-extraction system can determine, for one or more remaining digital content items, a content attribute score. The content-extraction system can select, as the result set of digital content, a subset of digital content based on the content attribute scores. The content-extraction system can output the result set to the client device.

The present application discloses a question-and-answer method for code information and related equipment; the embodiment of the present application can generate answer content for code processing questions; obtain the current generated answer content; determine at least one reference content unit sequence based on the content unit sequence at the preset detection position in the generated answer content; and count the number of occurrences of each reference content unit sequence at the preset detection position in the generated answer content; when sampling candidate content units belonging to the preset detection position during the answer generation process, the target content unit sequence composed of each candidate content unit and associated content unit to be sampled is matched with each reference content unit sequence, so as to sample the answer content from each candidate content unit according to the number of occurrences of the sequence corresponding to the target content unit sequence. In the embodiment of the present application, the occurrence of repeated content in the answer is reduced. Therefore, this scheme can improve the quality of generated answers.

The invention discloses a multi-document comparison method combining similarity calculation and accurate matching, and belongs to the technical field of information processing, and the method comprises the following steps: S1, text preprocessing, S2, blocking and positioning, S3, vector embedding, S4, similarity matrix construction, S5, Top-K pairing extraction, S6, similarity classification, S7, accurate matching, S8, visual highlighting, and S9, similarity evaluation. Through a sentence-level semantic vector and Top-K matching mechanism, the influence of document structure change is effectively overcome, the detection accuracy and robustness of content reconstruction type plagiarism are remarkably improved, matching dislocation caused by repeated content in a long document is avoided, the comparison stability is ensured, the problem of full-text vector semantic dilution is solved by embedding fine grit according to sentences, and the accuracy and robustness of content reconstruction type plagiarism detection are improved. According to the method, the recognition sensitivity of sentence pattern rewriting and local plagiarism is enhanced, and meanwhile, the abstract similarity is converted into traceable evidence by combining double-layer quantitative indexes with visual highlight positioning, so that the result interpretability and the practical value are greatly improved.

The present disclosure relates to the field of image processing technology, and in particular to a method, system, device and storage medium for obtaining duplicate content between multiple images. The method constructs a training data set containing multiple training images, and uses the training data set to train a model that can detect and distinguish between marked areas in multiple training images and whether the marked areas are the same, and then uses the trained model to distinguish whether the marked areas detected in the target image are the same, and finally quickly finds the duplicate content between the target images based on the detected marked areas and the distinction results.

The application discloses a project budget generation method and device, terminal equipment and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: acquiring project construction content; performing repeated content processing on the project construction content to obtain de-duplicated project construction content; performing budget estimation on the de-duplicated project construction content to obtain a budget amount of the project; and performing budget examination on the budget amount of the project to obtain a final budget result of the project. The application can perform repeated content processing on large-scale and complex information system projects through an automatic means, and thus the processing efficiency can be improved.

The embodiment of the invention provides a task processing method, a text processing method, an automatic question answering method, a task processing model training method, an information processing method based on a task processing model and a cloud training platform, which are applied to the technical field of computers, and the task processing method comprises the following steps: obtaining task data of a target task; the task data are input into a task processing model, a task processing result of the target task is obtained, the task processing model is obtained through training on the basis of multiple pieces of target sample data and target sample results of the multiple pieces of target sample data, and the target sample data are obtained through screening on the basis of repeated content detection results of the multiple pieces of sample data; the target sample result content is not repeated. The target sample data is obtained by screening based on the repeated content detection result, and the content of the target sample result is not repeated, so that the repeated illusion of the model is relieved on the premise of not damaging the processing capability of the model and not depending on external knowledge, and the accuracy and integrity of the task processing result are improved.

To provide a method, computer device, and computer program for eliminating duplicate content in a large-scale recommendation system.SOLUTION: A method for eliminating duplicate content in a large-scale recommendation system includes steps of: generating, for each piece of video content, a video hash using multiple segmented images generated from the video content; and identifying, based on the video hash, duplicate video content and eliminating duplication.SELECTED DRAWING: Figure 3

Embodiments of the present disclosure relate to a test case processing method, device, equipment and medium, wherein the method comprises: obtaining a first test case file in table format; performing format conversion on first data corresponding to the first test case file to obtain second data in mind map data structure; performing deduplication and integration processing on element node data of multiple test cases in the second data to obtain third data; and converting the third data into a second test case file in mind map format through a mind map tool. With the above technical solution, in the conversion process of the test case file from the table format to the mind map format, through the deduplication and integration processing, the data redundancy problem existing in the converted mind map format is avoided, the user can quickly find the required data, the readability of the data is improved, and in subsequent maintenance, the repeated work of repeated content is avoided, and the maintenance efficiency is improved.

The invention relates to an electric power planning report three-level title structured analysis generation method and device and a medium, and the method calls a large language model to execute the following steps: obtaining a to-be-processed electric power planning report and carrying out report paragraph text extraction to obtain an original text; segmenting the original text into overlapped text blocks, wherein the overlapped part comprises a three-level title context; constructing a power planning professional cue word, for each text block, identifying a three-level title in a specific format, classifying sub-section contents into a corresponding three-level title text, and outputting an analysis result; and detecting repeated contents in analysis results of adjacent text blocks, eliminating repetition based on text tail fix matching, integrating the analysis results of all the text blocks, generating a structured result containing three-level titles, corresponding texts, analysis time and text statistical information, and performing structural optimization. Compared with the prior art, the method has the advantages of high three-level title recognition accuracy, high structured data format compliance rate and the like.

The invention discloses an optimization method and device for a machine learning model for text extension, and the method comprises the steps: inputting a first text into a preset first machine learning model for extension, and obtaining a second text; obtaining a sample set of repeated texts in the second text, and comparing the sample set with a first preset text to obtain a paired data set; training a reward model according to the paired data set, and optimizing the first machine learning model according to the reward model to obtain a second machine learning model; and evaluating whether the repetition problem of the output text of the second machine learning model is relieved or not, and determining the second machine learning model as a target machine learning model under the condition that the repetition problem of the output text of the second machine learning model is relieved. According to the method, the quality of the extended text can be remarkably improved, the problem of low quality of the extended text caused by repeated content is reduced, the user experience is improved, and the efficiency can be improved.

The invention relates to the technical field of knowledge base management, in particular to a knowledge base management system based on a large model, which comprises a data acquisition module, a data processing module, a knowledge base construction module and a verification module. According to the method, the text data is continuously scanned to remove repeated content and avoid word segmentation errors, the source data in the social security field is classified to collect the unstructured data in the source data to perform data conversion, namely, the text in the unstructured data is processed to identify entities and concepts in the text, and the data conversion efficiency is improved. And identifying and recording the relationship between entities, improving the availability and query efficiency of data, verifying the accuracy of word segmentation processing by combining feedback data of a user, automatically identifying whether a wrong word segmentation result appears or not by recording query data of the user, and further adjusting a word segmentation processing flow and model training parameters. The social security service system is effectively managed.

The present invention discloses a multi-document summary generation method for crisis help information; the method is divided into: an extractive summary stage and a generative summary stage. In the extractive summary stage, the sentence graph structure is used to fuse contextual information in a deep neural network, and the subgraph structure is used to optimize the information extraction to effectively extract key information from numerous documents; in the generative summary stage, the powerful sequence generation ability of the BART model and the characteristics of the pointer generation network in accurately copying key information and avoiding duplicate content generation are used to further streamline and summarize the extracted information, and generate an accurate and compact summary of the crisis help information. The method of the present invention is suitable for processing and summarizing massive help information in crisis scenarios. The two-stage method of the present invention can quickly and accurately provide clear information summaries for rescue and emergency management, greatly improving the efficiency and effectiveness of crisis response.

This application discloses an agent-based electronic medical record (EMR) data governance method and related apparatus, relating to the field of data processing technology. After acquiring the EMR to be governed, the method first processes each chapter of the EMR by removing garbled characters and duplicate content errors, obtaining the chapter governance result, thus obtaining the chapter-governed EMR. Then, for each chapter in the chapter-governed EMR, consecutive error-free EMR text in each field is copied, and EMR text with inconsistencies between field names and content is generated, obtaining the chapter's field governance result. The field governance results of all chapters are combined to obtain the field-governed EMR, which is the data-governed EMR. This solution can both handle errors in EMR data and improve the efficiency and quality of EMR data governance.

Provide a system. 【Solution means】 Means for collecting existing documents and converting them into text, Means for analyzing text data, classifying and tagging it, Means for detecting and integrating duplicate content, Means for generating a unified document, Means for collecting business result data and matching it with the document content, Means for automatically updating the document, Means for verifying user authentication and permissions, Means for allowing users to provide feedback, A system including means for revising the document based on the feedback.

The invention discloses a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of computers, character strings are divided in multiple dynamic division modes with different character lengths, palintext character strings in sub-character strings are found out, the division mode containing the most palintext character strings is used as a target division mode, and the target division mode is used for dividing the palintext character strings in the sub-character strings. According to the method, repeated contents in the data can be found to the greatest extent by maximizing the number of the palindromic character strings, so that repeated parts in the character strings can be accurately identified and removed, and the characteristics of the palindromic character strings are fully utilized to optimize a data storage structure. Therefore, the technical problem of high cost of data storage equipment caused by increase of the total data amount can be solved, and the technical effects of improving the storage utilization rate, reducing the number and capacity requirements of the storage equipment and further reducing the operation cost of enterprises are achieved.

The invention relates to a method and a device for generating a page identifier. The method comprises the following steps: acquiring page data, wherein the page data comprises a page address and page content; the method further includes determining a target portion of the page address based on the page data, the target portion including at least one of a first query parameter related to the user and a second query parameter unrelated to page content. In addition, the method further includes generating a page identifier based on the target portion. In this way, normalization processing can be carried out on a plurality of pages with different page addresses and the same page content, so that the pages with the same page content have unique page identifiers. Therefore, under application scenes of webpage indexing, data statistical analysis, webpage data caching and security monitoring, webpages with the same page content of multiple different network addresses and related data of the webpages can be identified, repeated content can be eliminated, processing efficiency can be improved, forged page address attacks can be prevented, and system security can be improved.

The invention provides a science and technology document duplicate checking method and device, equipment and a medium, and the method comprises the steps: obtaining a fine tuning sample set according to an obtained sample science and technology document set, carrying out the fine tuning of a candidate large model, and obtaining a fine-tuned target large model; constructing a corresponding target science and technology knowledge graph, and obtaining a similar document set of the document object to be subjected to duplicate checking based on the target science and technology knowledge graph; and calling the model capability of the target large model, and obtaining a target duplicate checking result of the document object based on the similar document set through the model capability. According to the technical scheme, the document range for duplicate checking of the document object is narrowed through the target science and technology knowledge graph, the adaptation degree of the target large model and the industry to which the science and technology document belongs is improved through fine adjustment of the large model, the target duplicate checking result of the science and technology document is obtained based on the target large model, the duplicate checking efficiency and accuracy of the science and technology document are improved, and the practicability is high. And the duplicate checking method of the science and technology document is optimized.

The invention discloses a method for deleting repeated data based on fuzzy clustering in a distributed storage system. The method comprises a partitioning module, a superblock module, a clustering center module, a to-be-stored file S, a similarity Ri module, a threshold value delta module, a fingerprint module, an index table module and a fingerprint index updating module. According to the fuzzy clustering-based duplicated data deletion method in the distributed storage system, a file mapping table is constructed by applying file types and a division strategy to similar clustering, files are classified according to duplicated contents, blocking is carried out according to the file types, a similar tree structure is created, and the duplicated data deletion efficiency is improved. According to the method, by layering and gradually reducing the search range, the search efficiency is improved, and the metadata management overhead is reduced while the repeated data recognition rate is improved, so that the system performance is kept, and the data throughput and the system fault-tolerant capability can also be ensured without sacrificing the partial duplicate removal efficiency.

The systems and methods disclosed herein obtain (e.g., via a user interface) a collection of unstructured data, where each document includes a content set. Using a first AI model set, multiple summaries are generated by categorizing each document into clusters based on vector comparisons of content sets and summarizing the content for each cluster. A second AI model set (same as or different from the first AI model set) identifies duplicate content within the unstructured data by generating similarity values between pairs of summaries and determining if the similarity values meet a predefined threshold. A report is generated (e.g., on the user interface) indicating the duplicate content sets and / or the collection of unstructured data.

The present application discloses a method for identifying songs with duplicate content, including: performing song matching based on multiple audio clips of a song to be identified to obtain a target song; determining the offset time position of each audio clip in the target song among the multiple audio clips; wherein the offset time position is used to characterize the time point at which the audio clip matches the target song; calculating the difference between the distribution curve to be identified formed by all offset time positions and the standard time distribution curve; if the difference is greater than a threshold, the song to be identified is determined to be a song with duplicate content. The difference between multiple distribution curves to be identified and the standard time distribution curve is calculated. If the difference is greater, the song to be identified is determined to be a song with duplicate content, rather than using a model to identify songs with duplicate content, thereby improving the efficiency of identifying songs with duplicate content. The present application also discloses a server and a computer-readable storage medium, which have the above beneficial effects.

The invention provides a sequence frame digital content generation method, device and equipment and a computer readable medium, and the method comprises the steps: obtaining a basic material set which comprises a plurality of basic materials which are not repeated; the method comprises the following steps: acquiring a basic material set, generating a metadata file used for recording a reference relationship between a sequence frame bit and the basic material according to a material identifier of the basic material, at least one basic material being referenced by a plurality of sequence frame bits, packaging according to the basic material set and the metadata file to generate a sequence frame digital content, and sending the sequence frame digital content to the basic material set. According to the method and the device, shared reference of the basic materials in different sequence frame positions can be realized based on the reference relationship recorded by the metadata file, and the repeated content of the basic material set is removed, so that repeated storage of materials such as the same picture can be avoided, the quantity of the materials such as the picture can be greatly reduced, and the size of a final resource packet is further reduced.