49 results about "Duplicate content" patented technology

Provided in the embodiments of the present disclosure are a task processing method, a text processing method, an automatic question-answering method, a task processing model training method, an information processing method based on a task processing model, and a cloud training platform, which are applied to the technical field of computers. The task processing method comprises: acquiring task data of a target task; and inputting the task data into a task processing model, so as to obtain a task processing result of the target task, wherein the task processing model is obtained by means of performing training on the basis of a plurality of pieces of target sample data and target sample results of the plurality of pieces of target sample data, the target sample data is obtained by means of performing screening on the basis of detection results of repeated content of a plurality of pieces of sample data, and the content of the target sample results is not repeated. Target sample data is obtained by means of performing screening on the basis of detection results of repeated content, and the content of target sample results is not repeated, such that the repetition illusion of a model is reduced without damaging the processing capability of the model itself and relying on external knowledge, thereby improving the accuracy and integrity of a task processing result.

The invention discloses a content interaction method for AI online education, which is applied to a content interaction system for online AI education, and the content interaction system for online AI education comprises a teacher end, a server end and a student end. According to the AI online education content interaction method, through interaction among the teacher terminal, the server terminal and the student terminal, students can play back at any time point through the student terminal, timely and repeatedly listen to important knowledge points, and can also join in a live course again without missing follow-up real-time explanation content; meanwhile, the playback efficiency is improved through the judgment of the wrong playback operation at the student end. According to the playback dynamic state of the student end, the teacher end generates a message prompt of a playback hotspot interval to assist a teacher to adjust an explanation strategy in a targeted manner, so that the content is fully understood and mastered, and the teaching effect is effectively improved; and meanwhile, the repeated content explanation is identified and the classroom rest time period is reminded, so that the time is effectively saved when the student side plays back.

The invention discloses a bidding document duplicate checking method and system based on artificial intelligence, and comprehensive analysis and accurate duplicate checking of bidding document content are realized through multi-modal analysis (text, image and table), a dynamic weight model and a visual report. The method breaks through the limitation of traditional text comparison, and supports similarity detection of non-text elements such as images and tables; capturing deep semantic similarity by adopting a bag-of-word vector and assignment dimension reduction technology; the image matching accuracy is improved through a self-adaptive partitioning algorithm; semantic matching of the table data is achieved based on the attribute item weight; the strategy pertinence is enhanced through dynamic weight distribution; the visual report visually presents repeated contents; the analysis precision is improved through anti-interference preprocessing; and multi-format documents are compatible to meet diversified requirements. According to the method, the comprehensiveness, accuracy and efficiency of duplicate checking of the bidding document are remarkably improved, the problems of complex formats and semantic understanding are effectively solved, and the high requirements of the bidding and tendering field for fairness and originality are met.

The invention relates to the technical field of aeronautical text data processing, and discloses an aeronautical text content cleaning and labeling method, system, device and medium wherein the method comprises: noise filtering: identifying and removing noise in an aeronautical text in combination with static cleaning and a general large model; format standardization: converting the aviation text after noise removal into a standardized format text; duplicate removal and error correction: detecting duplicate contents based on a hash algorithm, and correcting spelling errors and grammar errors based on a general large model to obtain an aviation text subjected to duplicate removal and error correction; entity identification: key entities are extracted based on the general large model, and the extracted key entities are labeled; active learning: screening high-value samples in the marked key entities based on an uncertainty query strategy; and dynamic optimization: performing verification and iterative optimization on the marking result in combination with the aviation knowledge base. According to the method, the automation level, the labeling accuracy and the system self-adaptive capability of aviation text processing can be remarkably improved.

The invention discloses a science and technology project document duplicate checking method and device based on multi-level abstract generation and a medium, belongs to the technical field of document duplicate checking, and solves the problem of how to improve the reliability and accuracy of science and technology project document duplicate checking. The method comprises the following steps: firstly, carrying out structured analysis on a project document to be subjected to duplicate checking, introducing a fine-tuned abstract generation model to obtain a multi-level abstract of a project to be subjected to duplicate checking, and extracting a feature vector of the multi-level abstract through a fine-tuned and trained text feature extraction model; secondly, performing similarity search in a vector database based on feature vectors corresponding to the dimensions of the main abstract, further calculating cosine similarities based on the dimensions of the subordinate abstract, and sequencing the weighted similarities to obtain a similarity rank; and finally, according to the structured analysis information and the similarity ranking result, LLM similarity analysis is performed by using a large language model, a document duplicate checking report is formed, the originality of the science and technology project is ensured, repeated subsidization is avoided, and the method has important practical application value for science and technology project management.

The embodiment of the invention provides an online man-machine conversation optimization method, system and device, electronic equipment, a storage medium and a program product. In the scheme provided by the embodiment, when a first dialogue generation model executes a streaming response based on user input data, a currently generated first dialogue text fragment is subjected to repeated detection, and when it is detected that repeated content exists in the first dialogue text fragment, the first dialogue generation model is controlled to interrupt execution of the streaming response; in addition, reply generation constraint information is determined according to the repeated content in the first reply text segment, and then reply generation cue words are generated based on the reply generation constraint information and the non-repeated content in the first reply text segment. And after the dialogue generation cue word is input into the first dialogue generation model, the first dialogue generation model is triggered to determine a dialogue generation starting point according to the non-repeated content in the first dialogue text fragment, and streaming response is continuously executed from the dialogue generation starting point.

The invention discloses a multi-document comparison method combining similarity calculation and accurate matching, and belongs to the technical field of information processing, and the method comprises the following steps: S1, text preprocessing, S2, blocking and positioning, S3, vector embedding, S4, similarity matrix construction, S5, Top-K pairing extraction, S6, similarity classification, S7, accurate matching, S8, visual highlighting, and S9, similarity evaluation. Through a sentence-level semantic vector and Top-K matching mechanism, the influence of document structure change is effectively overcome, the detection accuracy and robustness of content reconstruction type plagiarism are remarkably improved, matching dislocation caused by repeated content in a long document is avoided, the comparison stability is ensured, the problem of full-text vector semantic dilution is solved by embedding fine grit according to sentences, and the accuracy and robustness of content reconstruction type plagiarism detection are improved. According to the method, the recognition sensitivity of sentence pattern rewriting and local plagiarism is enhanced, and meanwhile, the abstract similarity is converted into traceable evidence by combining double-layer quantitative indexes with visual highlight positioning, so that the result interpretability and the practical value are greatly improved.

The application discloses a project budget generation method and device, terminal equipment and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: acquiring project construction content; performing repeated content processing on the project construction content to obtain de-duplicated project construction content; performing budget estimation on the de-duplicated project construction content to obtain a budget amount of the project; and performing budget examination on the budget amount of the project to obtain a final budget result of the project. The application can perform repeated content processing on large-scale and complex information system projects through an automatic means, and thus the processing efficiency can be improved.

The embodiment of the invention provides a task processing method, a text processing method, an automatic question answering method, a task processing model training method, an information processing method based on a task processing model and a cloud training platform, which are applied to the technical field of computers, and the task processing method comprises the following steps: obtaining task data of a target task; the task data are input into a task processing model, a task processing result of the target task is obtained, the task processing model is obtained through training on the basis of multiple pieces of target sample data and target sample results of the multiple pieces of target sample data, and the target sample data are obtained through screening on the basis of repeated content detection results of the multiple pieces of sample data; the target sample result content is not repeated. The target sample data is obtained by screening based on the repeated content detection result, and the content of the target sample result is not repeated, so that the repeated illusion of the model is relieved on the premise of not damaging the processing capability of the model and not depending on external knowledge, and the accuracy and integrity of the task processing result are improved.

To provide a method, computer device, and computer program for eliminating duplicate content in a large-scale recommendation system.SOLUTION: A method for eliminating duplicate content in a large-scale recommendation system includes steps of: generating, for each piece of video content, a video hash using multiple segmented images generated from the video content; and identifying, based on the video hash, duplicate video content and eliminating duplication.SELECTED DRAWING: Figure 3

Embodiments of the present disclosure relate to a test case processing method, device, equipment and medium, wherein the method comprises: obtaining a first test case file in table format; performing format conversion on first data corresponding to the first test case file to obtain second data in mind map data structure; performing deduplication and integration processing on element node data of multiple test cases in the second data to obtain third data; and converting the third data into a second test case file in mind map format through a mind map tool. With the above technical solution, in the conversion process of the test case file from the table format to the mind map format, through the deduplication and integration processing, the data redundancy problem existing in the converted mind map format is avoided, the user can quickly find the required data, the readability of the data is improved, and in subsequent maintenance, the repeated work of repeated content is avoided, and the maintenance efficiency is improved.

The invention relates to an electric power planning report three-level title structured analysis generation method and device and a medium, and the method calls a large language model to execute the following steps: obtaining a to-be-processed electric power planning report and carrying out report paragraph text extraction to obtain an original text; segmenting the original text into overlapped text blocks, wherein the overlapped part comprises a three-level title context; constructing a power planning professional cue word, for each text block, identifying a three-level title in a specific format, classifying sub-section contents into a corresponding three-level title text, and outputting an analysis result; and detecting repeated contents in analysis results of adjacent text blocks, eliminating repetition based on text tail fix matching, integrating the analysis results of all the text blocks, generating a structured result containing three-level titles, corresponding texts, analysis time and text statistical information, and performing structural optimization. Compared with the prior art, the method has the advantages of high three-level title recognition accuracy, high structured data format compliance rate and the like.

The present invention discloses a multi-document summary generation method for crisis help information; the method is divided into: an extractive summary stage and a generative summary stage. In the extractive summary stage, the sentence graph structure is used to fuse contextual information in a deep neural network, and the subgraph structure is used to optimize the information extraction to effectively extract key information from numerous documents; in the generative summary stage, the powerful sequence generation ability of the BART model and the characteristics of the pointer generation network in accurately copying key information and avoiding duplicate content generation are used to further streamline and summarize the extracted information, and generate an accurate and compact summary of the crisis help information. The method of the present invention is suitable for processing and summarizing massive help information in crisis scenarios. The two-stage method of the present invention can quickly and accurately provide clear information summaries for rescue and emergency management, greatly improving the efficiency and effectiveness of crisis response.

This application discloses an agent-based electronic medical record (EMR) data governance method and related apparatus, relating to the field of data processing technology. After acquiring the EMR to be governed, the method first processes each chapter of the EMR by removing garbled characters and duplicate content errors, obtaining the chapter governance result, thus obtaining the chapter-governed EMR. Then, for each chapter in the chapter-governed EMR, consecutive error-free EMR text in each field is copied, and EMR text with inconsistencies between field names and content is generated, obtaining the chapter's field governance result. The field governance results of all chapters are combined to obtain the field-governed EMR, which is the data-governed EMR. This solution can both handle errors in EMR data and improve the efficiency and quality of EMR data governance.

Provide a system. 【Solution means】 Means for collecting existing documents and converting them into text, Means for analyzing text data, classifying and tagging it, Means for detecting and integrating duplicate content, Means for generating a unified document, Means for collecting business result data and matching it with the document content, Means for automatically updating the document, Means for verifying user authentication and permissions, Means for allowing users to provide feedback, A system including means for revising the document based on the feedback.

The invention discloses a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of computers, character strings are divided in multiple dynamic division modes with different character lengths, palintext character strings in sub-character strings are found out, the division mode containing the most palintext character strings is used as a target division mode, and the target division mode is used for dividing the palintext character strings in the sub-character strings. According to the method, repeated contents in the data can be found to the greatest extent by maximizing the number of the palindromic character strings, so that repeated parts in the character strings can be accurately identified and removed, and the characteristics of the palindromic character strings are fully utilized to optimize a data storage structure. Therefore, the technical problem of high cost of data storage equipment caused by increase of the total data amount can be solved, and the technical effects of improving the storage utilization rate, reducing the number and capacity requirements of the storage equipment and further reducing the operation cost of enterprises are achieved.

The invention provides a science and technology document duplicate checking method and device, equipment and a medium, and the method comprises the steps: obtaining a fine tuning sample set according to an obtained sample science and technology document set, carrying out the fine tuning of a candidate large model, and obtaining a fine-tuned target large model; constructing a corresponding target science and technology knowledge graph, and obtaining a similar document set of the document object to be subjected to duplicate checking based on the target science and technology knowledge graph; and calling the model capability of the target large model, and obtaining a target duplicate checking result of the document object based on the similar document set through the model capability. According to the technical scheme, the document range for duplicate checking of the document object is narrowed through the target science and technology knowledge graph, the adaptation degree of the target large model and the industry to which the science and technology document belongs is improved through fine adjustment of the large model, the target duplicate checking result of the science and technology document is obtained based on the target large model, the duplicate checking efficiency and accuracy of the science and technology document are improved, and the practicability is high. And the duplicate checking method of the science and technology document is optimized.

The invention discloses a method for deleting repeated data based on fuzzy clustering in a distributed storage system. The method comprises a partitioning module, a superblock module, a clustering center module, a to-be-stored file S, a similarity Ri module, a threshold value delta module, a fingerprint module, an index table module and a fingerprint index updating module. According to the fuzzy clustering-based duplicated data deletion method in the distributed storage system, a file mapping table is constructed by applying file types and a division strategy to similar clustering, files are classified according to duplicated contents, blocking is carried out according to the file types, a similar tree structure is created, and the duplicated data deletion efficiency is improved. According to the method, by layering and gradually reducing the search range, the search efficiency is improved, and the metadata management overhead is reduced while the repeated data recognition rate is improved, so that the system performance is kept, and the data throughput and the system fault-tolerant capability can also be ensured without sacrificing the partial duplicate removal efficiency.

The systems and methods disclosed herein obtain (e.g., via a user interface) a collection of unstructured data, where each document includes a content set. Using a first AI model set, multiple summaries are generated by categorizing each document into clusters based on vector comparisons of content sets and summarizing the content for each cluster. A second AI model set (same as or different from the first AI model set) identifies duplicate content within the unstructured data by generating similarity values between pairs of summaries and determining if the similarity values meet a predefined threshold. A report is generated (e.g., on the user interface) indicating the duplicate content sets and / or the collection of unstructured data.

The present application discloses a method for identifying songs with duplicate content, including: performing song matching based on multiple audio clips of a song to be identified to obtain a target song; determining the offset time position of each audio clip in the target song among the multiple audio clips; wherein the offset time position is used to characterize the time point at which the audio clip matches the target song; calculating the difference between the distribution curve to be identified formed by all offset time positions and the standard time distribution curve; if the difference is greater than a threshold, the song to be identified is determined to be a song with duplicate content. The difference between multiple distribution curves to be identified and the standard time distribution curve is calculated. If the difference is greater, the song to be identified is determined to be a song with duplicate content, rather than using a model to identify songs with duplicate content, thereby improving the efficiency of identifying songs with duplicate content. The present application also discloses a server and a computer-readable storage medium, which have the above beneficial effects.

The present invention discloses an intelligent management system for water conservancy project construction plans, including a client installed in a local computer Windows system and a server installed in a server, wherein the client includes a desktop and a WEB client; a VSTO plug-in is used to embed the desktop client into Word or WPS; the desktop client has the functions of creating, opening, editing and saving new construction group documents; the server has a database and a template library, and the template library is used to store construction group template documents pre-made by users. The present invention also discloses a corresponding construction organization design method. The present invention can effectively reuse the content of historical construction group documents, and the results of one compilation work can be conveniently reused in subsequent projects, thereby improving the efficiency and consistency of compiling construction organization design documents, and solving problems such as inconsistent content formats, large compilation workload, confusing node levels, inconsistent expressions of repeated content, inconsistent node dates and difficulty in modification, and time-consuming and labor-intensive formula input.

The invention discloses a document retrieval method, device and equipment based on a large model and a storage medium, and relates to the technical field of natural language processing, and the method comprises the following steps: determining a target analysis method corresponding to an initial document according to a file type of the initial document, and converting the initial document into a target document by utilizing the target analysis method and a preset large model; splitting the target document based on a preset identifier in the target document to obtain document fragments; wherein the document information of the head part and the tail part of any document fragment is the same as the document information contained in the tail part and the head part of the adjacent document fragment; and vectorizing each document fragment to obtain vectorized fragments, storing each vectorized fragment into a vector database, and retrieving each vectorized fragment by using a retrieval enhancement generation system. By analyzing and storing the document data according to the file type, the application range of the RAG system is widened, and by setting repeated content for the adjacent fragments, the continuity of the data is ensured.

In one aspect, a method includes detecting a fingerprint match between query fingerprint data representing at least one audio segment within podcast content and reference fingerprint data representing known repetitive content within other podcast content, detecting a feature match between a set of audio features across multiple time-windows of the podcast content, and detecting a text match between at least one query text sentences from a transcript of the podcast content and reference text sentences, the reference text sentences comprising text sentences from the known repetitive content within the other podcast content. The method also includes responsive to the detections, generating sets of labels identifying potential repetitive content within the podcast content. The method also includes selecting, from the sets of labels, a consolidated set of labels identifying segments of repetitive content within the podcast content, and responsive to selecting the consolidated set of labels, performing an action.

The invention discloses a statistical image duplicate checking method and system based on a large language model, and the method comprises the steps: constructing a large language model architecture for converting a statistical image into a text, carrying out the analysis of the statistical image based on the large language model, converting the statistical image into a structured text, and carrying out the duplicate checking of the statistical image. And recognizing the text part and the numerical part of the statistical image structured text, and further calculating the similarity of the text and the numerical part to realize duplicate checking of the statistical image. According to the method, the image content can be efficiently retrieved and compared, the accuracy and efficiency of statistical image duplicate checking are improved, and the method has important application value in the aspects of image copyright protection, content auditing and the like.

The application discloses an optimization method and device for a machine learning model for text expansion, the method comprising: inputting a first text into a preset first machine learning model for expansion to obtain a second text; obtaining a sample set of repeated texts in the second text, and comparing the sample set with a first preset text to obtain a paired data set; training a reward model according to the paired data set, and optimizing the first machine learning model according to the reward model to obtain a second machine learning model; evaluating whether the repeated problem of the output text of the second machine learning model is alleviated, and determining the second machine learning model as a target machine learning model in the case that the repeated problem of the output text of the second machine learning model is alleviated. The method can significantly improve the quality of the expanded text, reduce the problem of low quality of the expanded text caused by repeated content, improve the user experience, and improve the efficiency.

In some embodiments, an electronic device detects that the user of the electronic device has a first subscription to a first content application and a second subscription to a second content application. In some embodiments, an electronic device displays a content user interface of the first content application. In some embodiments, after receiving a request (e.g., an input) to initiate a process to duplicate content items associated with a second user profile of the second application to a first user profile of the first application, the electronic device saves content items to the first user profile in the first application that meet one or more criteria. In some embodiments, the electronic device displays one or more visual indications of content items of the first content application that do not meet the criteria in a review user interface.

The invention relates to the field of data communication, in particular to an intelligent card data communication method and device, and the method comprises the steps: detecting bytes with the same and continuous content in an original data stream, and converting the content of a plurality of bytes with the same and continuous content into a compressed format content; wherein the original data stream comprises data to be written into the smart card; the compressed format content comprises sequentially arranged constant escape characters, repetition length and repetition content, the repetition length is the number of bytes with the same content and consecutive bytes, and the repetition content is the content of the bytes with the same content and consecutive bytes; and at the smart card end, converting the compressed format content into a plurality of same and continuous byte content based on the identification of the constant escape character. Therefore, by compressing the parts with the same and continuous content in the data to be written into the intelligent card and decompressing at the intelligent card end, the related communication time of the intelligent card can be greatly shortened, and the production and manufacturing efficiency is improved.

The invention provides an automatic migration control method for alignment of a containerized mirror image version and a database version. The method comprises the following sub-steps: S1, obtaining application mirror image version information; s2, obtaining a current database version number; s3, performing version consistency judgment; s4, loading a database migration script based on a migration mode, and generating a database migration script execution queue; s5, executing database migration and recording a result; s6, performing exception handling and start control; according to the method, migration scripts are sequenced according to a version identifier incremental rule, an execution queue is generated, and the script execution sequence is guaranteed to be compliant; repeated content is removed through difference comparison, and conflicts caused by independent maintenance of scripts of multiple projects are avoided; according to the method, the database migration process and the containerized application starting process are combined, triggering and execution of database migration are achieved, the manual intervention requirement is reduced, the risk of manual misoperation is reduced, and the automation efficiency of the system deployment and upgrading process is improved.