140 results about "Duplicate detection" patented technology

Techniques described herein provide for generation of structured output for documents identified using automated near-duplicate detection. In one example, a system can receive a set of documents including at least one pair of similar documents determined to be similar to one another based on similarity scores generated using a predefined similarity scoring technique. The system can generate document groups by merging together pairs of documents that share at least one document. The system can, for each of the document groups, identify a representative document for the document group. The system can generate an output for display including a section for each document group, in which each section includes the representative document for the document group and, for each document in the document group, the similarity score relative to the representative document for the document group.

At a machine learning service, a determination is made that an analysis to detect whether at least a portion of contents of one or more observation records of a first data set are duplicated in a second set of observation records is to be performed. A duplication metric is obtained, indicative of a non-zero probability that one or more observation records of the second set are duplicates of respective observation records of the first set. In response to determining that the duplication metric meets a threshold criterion, one or more responsive actions are initiated, such as the transmission of a notification to a client of the service.

The invention discloses a rapid marine ship target identification method based on YOLOv8, and is applied to the technical field of target identification. Comprising the following steps of collecting an image sample of a ship navigation water area in a digital vehicle, cleaning and marking the image sample, performing data enhancement on the image sample to obtain a data set which can be used for training and verification, constructing a target identification network model based on a YOLOv8 deep neural network, and obtaining a target identification result by setting hyper-parameters such as a learning rate and a batch processing size. And carrying out transfer learning by using the pre-training model, removing a repeated detection frame by using a non-maximum suppression algorithm and carrying out result filtering by using a confidence coefficient threshold to carry out post-processing on a reasoning result of the model so as to complete prediction of the ship target, and carrying out sparse pruning training on the trained model neural network so as to obtain the ship target. And carrying out calibration and quantification on the pruned model. According to the method, on the premise that the model detection precision is reserved to the maximum, memory consumption during equipment operation is reduced, and meanwhile the model detection speed is increased.

Techniques described herein provide for automated detection of near-duplicate documents. In one example, a system can cluster documents into a set of clusters based on character frequencies associated with the documents. For a given cluster, the system can generate first similarity scores associated with every pair of documents in the cluster. The system can then select a filtered group of documents associated with first similarity scores that meet or exceed a first predefined similarity threshold. Next, the system can convert the filtered group of documents into matrix representations. The system can generate second similarity scores for every pair of matrix representations. The system can then identify documents, from among the filtered group of documents, associated with second similarity scores that meet or exceed a second predefined similarity threshold. The identified documents can be duplicate or near-duplicate text documents.

The invention discloses an intelligent target detection method and device based on computer vision, and relates to the technical field of visual recognition. The method comprises the following specific implementation steps: step 1, detecting and positioning an area possibly containing a small target in an image by using saliency; step 2, carrying out saliency map fusion; step 3, carrying out super-separation treatment; step 4, adaptively cutting the image, and segmenting the large image by adopting a sliding window; step 5, combining repeated detection results; step 6, carrying out weighted fusion; according to the method, a set of complete processing flow from global saliency detection, candidate region screening, local super-resolution reconstruction, adaptive cutting to result fusion is formed through Step1 to Step6, and the accuracy and precision of small target detection are improved.

A duplication check system and method for paper generated by artificial intelligence includes steps: S1: the user uploading the academic paper to be detected to a system, and the system automatically extracting the title, the abstract, and the headline of each paragraph of the paper; S2: fusing the title, the abstract, and the headline of each paragraph of the paper with the contextual information of the paper and extracting theme features; S3: after the different themes of the paper are extracted, repeatedly using similar tones for each theme in all different AI tools to propose integrate text requirements, searching each theme for times of the number of repetitions of integration in each AI tool until no new content is obtained, matching all the obtained texts with the paper to be duplication checked, based on natural language understanding, for duplication check, and marking the matching repeated parts and indicating the sources.

This disclosure provides systems and methods for controlling a vehicle based on a combination of high precision detection and high recall detection. The disclosed systems and methods can efficiently generate trajectories by reducing duplicate detection or duplicate calculation of objects or obstacles of common and known object types and objects or obstacles without class identification.

Techniques described herein provide for text string comparison for documents identified using automated near-duplicate detection. In one example, a system can receive a pair of documents. The system can extract text strings from the documents. The system can normalize the extracted text strings using a predefined normalization scheme. The system can identify boilerplate text segments in the normalized text strings. The system can remove the boilerplate text segments from the normalized text strings to generate filtered text strings. The system can divide the filtered text strings by identifying section indicators. The system can, for each section, generate groupings of text strings and determine a similarity score between each pair of corresponding groupings to identify matching groupings of text strings. The system can generate an output for display showing the visual indications of the matched groupings of text strings.

The embodiment of the invention provides a file uploading optimization method and device based on parallel computing, and the method comprises the steps: receiving a file selected by a user, carrying out parallel file transmission operation on a data uploading module and a Hash algorithm module, receiving file data through the data uploading module, determining a corresponding data uploading result, and outputting the data uploading result. The method comprises the following steps: receiving file data through a Hash algorithm module, inputting the file data into a set Hash algorithm selection model to obtain a corresponding Hash algorithm, performing Hash repetition detection operation according to the Hash algorithm, the file data and a preset Hash value database, and determining a corresponding Hash repetition result, the data uploading result and the Hash repetition result are received in parallel through the data processing module, the corresponding uploading storage result is determined according to the preset data processing rule, the data uploading result and the Hash repetition result, and the file uploading efficiency and flexibility can be improved based on the intelligent Hash algorithm selection and parallel computing method.

Techniques described herein provide for automated near-duplicate detection for new text documents given text documents that were previously processed using automated near-duplicate detection for text documents. In one example, a system can receive new documents and documents that were previously processed using a predefined processing technique for automated near-duplicate detection. The system can process the new documents and cluster the new documents into multiple predefined clusters previously identified using the predefined processing technique. For each predefined cluster including at least one new document, the system can generate document groups by determining similarity scores using the predefined processing technique as applied to the documents in the predefined clusters. The system can identify a representative document for each document group and generate an output data structure including the document groups and the representative document for each group.

Provided in the embodiments of the present disclosure are a deadlock detection method, a device and a storage medium. In the embodiments of the present disclosure, a detection mechanism in which a triggering entity is also responsible for execution is provided, and each node in a distributed system can serve as an executor for deadlock detection, so that deadlock detection tasks in the distributed system can be executed in a distributed manner, rather than being concentrated on a single node, thereby effectively accommodating the scalability of the distributed system and preventing the detection efficiency from being affected as the system scales up. Moreover, a detection mechanism similar to optimistic locking is further provided, so as to prevent problems such as redundant detection that may occur between the deadlock detection tasks distributed on the nodes, thereby ensuring the accuracy of deadlock detection. In addition, a single deadlock detection task can detect all deadlocks in the system in one step, eliminating the need for serial detection and thus further improving the efficiency of deadlock detection in the distributed system.

The invention relates to the technical field of code analysis, and discloses a multi-terminal code repetition detection and component reconstruction method, device, equipment and medium, and the method comprises the steps: carrying out code analysis on a pre-acquired project code file, and identifying a code mapping relation to obtain an abstract syntax tree and a cross-platform code mapping relation, and according to the abstract syntax tree and the cross-platform code mapping relationship, carrying out repetition logic detection on the project code file to obtain a repetition detection result, according to the repetition detection result, confirming a repetition code in the project code file, carrying out cross-platform code optimization on the repetition code to obtain an optimized code, and carrying out cross-platform code optimization on the optimized code. The method comprises the steps of obtaining duplicated codes, extracting reusable components on the basis of the duplicated codes, generating a multi-end adaptive component template according to a preset condition compiling logic, and reconstructing the optimized codes according to the reusable components and the multi-end adaptive component template to obtain reconstructed codes, so that the code duplicating detection and reconstruction efficiency is improved.

The invention provides a solid state disk garbage collection and host write address repetition detection method, which comprises the following steps that: a host covers data written into a logic address, finds a first physical address which is written in a covered manner, and writes the first physical address into a hardware acceleration module; reading continuous physical address data in batches at the beginning of garbage collection, and writing a currently read second physical address into a hardware acceleration module; the hardware acceleration module compares the first physical address with the second physical address; after data are read through garbage collection, a comparison result of the hardware acceleration module is obtained, if the comparison result is consistent, the current second physical address is skipped, continuous physical address data are continuously read, and if the comparison result is inconsistent, the second physical address data are written into a new physical address; and the steps are repeated until garbage recycling is completed. According to the invention, the speed of garbage collection and data writing is accelerated, so that the purposes of improving the data transmission efficiency and reducing the use power consumption are achieved.

The invention discloses a method for solving missing detection and repeated detection of road diseases based on computer vision, and the method comprises the steps: S1, video collection and stream pulling: carrying out the real-time video stream pulling of an image collection device, and carrying out the real-time frame extraction of the video stream after stream pulling; s2, dynamic frame extraction: performing dynamic frame extraction on the video stream based on different driving speeds of the AI road recognition terminal to obtain a disease image; s3, caching: caching all disease images obtained by dynamic frame extraction in a sliding window mode; and S4, disease point duplicate removal: putting the disease images, which record a plurality of states, of the same disease point described in the disease images obtained in the step S3 into a buffer area, judging whether the disease images are the same disease or not based on the disease features, and if the disease images are the same disease, removing duplicate images and reserving one image. According to the method, the technical problems of missing detection and repeated detection of road disease detection are solved, it is guaranteed that the disease image uploaded to the platform from the inspection box is free of missing and repeated detection, and uniqueness is guaranteed.

The invention discloses a text duplicate checking method, a vehicle, a computer readable storage medium and a computer program product, and relates to the technical field of information processing. The method comprises the following steps: carrying out standardized preprocessing on an input text to be subjected to duplicate checking to obtain a current text fragment; performing semantic coding processing on the current text fragment by utilizing the target coding model to obtain a query vector; a similarity retrieval interface corresponding to the target search engine is called, a plurality of candidate vectors corresponding to the query vector are recalled from a semantic vector index, and the semantic vector index is constructed based on a historical text set and a graph index algorithm in the target search engine; sorting analysis is conducted on the multiple candidate vectors based on the multi-level duplicate judgment threshold values, a duplicate checking result is obtained, and the duplicate checking result is used for representing the duplicate situation between the historical texts corresponding to the multiple candidate vectors and the input text. The technical problems of low accuracy and low efficiency of a text duplicate checking method in related technologies are solved.

The invention relates to the technical field of board detection, in particular to wood board compression resistance detection equipment and method, and the wood board compression resistance detection equipment comprises a placement base, an abutting unit, a pressing mechanism and an adjusting unit. According to the invention, the acting force with a specified inclination angle is transmitted to the stress plate through the lower pressing plate and the hinge strip, and finally the acting force is transmitted to the rectangular bulge, so that the compression resistance of the vertical section of the rectangular bulge is detected, and the situation that the great wall plate is impacted by an external inclination angle during actual use is simulated; under the guidance of the V-shaped plate, the transverse moving plate and the downward pressing mechanism are driven to move to different positions for repeated detection operation, and single rectangular bulge is detected each time, so that the compression resistance data of each rectangular bulge are obtained, and the situation that the overall compression resistance data are taken as the compression resistance data of one or more rectangular bulges is avoided; the load applied to the local rectangular bulge of the great wall plate in actual use exceeds the actual bearing range of the great wall plate, so that the great wall plate is damaged.

The embodiment of the invention provides a duplicate checking small language model training method combined with multilevel knowledge distillation, and the method comprises the steps: obtaining duplicate checking sample pairs, and determining the complexity of the duplicate checking sample pairs according to the text features of the duplicate checking sample pairs; according to the complexity of the duplicate checking sample pair, determining distillation levels of the teacher model, and determining a weighting coefficient of each distillation level of the teacher model; determining the distillation loss between the teacher model and the student model according to the weighting coefficient of each network distillation of the teacher model, the first output result of each distillation level of the teacher model and the second output result of each distillation level of the student model; according to the distillation loss between the teacher model and the student model, updating parameters of the student model; and repeating the above steps until the updated student model meets the preset condition, and taking the updated student model as a duplicate checking small language model, thereby realizing a low-power-consumption and high-precision duplicate checking effect.

The invention discloses a cloud password service-oriented fuzzy test and compliance evaluation platform and method, and relates to the technical field of cloud computing and password security evaluation. The evaluation platform comprises a demand analysis module, a test case generation module, a fuzzy test execution module, a compliance detection module, a result fusion analysis module and a report generation module, and the result fusion analysis module is connected with the fuzzy test execution module and the compliance detection module. And the cloud password service detection module is used for carrying out association analysis on the response data, the abnormal log and the compliance detection result acquired by the fuzzy test, identifying the security vulnerability type and the compliance defect level of the cloud password service, and removing a repeated detection result. Fuzzy testing and compliance detection are organically fused, correlation analysis of the two results is achieved through the result fusion analysis module, the problems that in the prior art, the two tests are separated, and the results are difficult to integrate are solved, and the safety compliance condition of the cloud password service can be comprehensively reflected.

The invention discloses a duplicated data detection method and device, a storage medium and a computer program product, and relates to the technical field of data processing, a preset bit array is constructed through a plurality of preset functions and stored data blocks to serve as the basis of data duplicated detection of to-be-detected data blocks, and the data duplicated detection efficiency is improved. The to-be-detected data blocks which may be repeated are preliminarily screened out through the preset bit array, and then the to-be-detected data blocks which may be repeated are further subjected to refined detection through the preset hash table, so that layered detection of data is realized, and compared with hash table look-up traversal comparison of all the to-be-detected data blocks, the detection efficiency is effectively improved, and the detection time is shortened. And meanwhile, the detection accuracy is ensured.

The embodiment of the invention provides a deadlock detection method and device and a storage medium. In the embodiment of the invention, a detection mechanism that who triggers execution is provided, and each node in the distributed system can serve as an execution main body of deadlock detection, so that the deadlock detection tasks in the distributed system can be executed in a distributed manner and are not concentrated at a single node any more, the expandability of the distributed system can be effectively adapted, and the expandability of the distributed system is improved. And the detection efficiency is not influenced by the scale expansion of the system. Moreover, the invention also provides a detection mechanism similar to an optimistic lock, so as to avoid the problems of repeated detection and the like possibly occurring between deadlock detection tasks distributed on each node, and further ensure the accuracy of deadlock detection. Besides, a single deadlock detection task can detect all deadlocks in the system at one time, and serial detection is not needed any more, so that the deadlock detection efficiency in the distributed system can be further improved.

The invention discloses a duplicate checking method and system for products in all stages of software research and development, and belongs to the technical field of software engineering. In order to solve the problems of high efficiency and accuracy of code, document and function duplicate checking, the technical means of abstract syntax tree analysis, hash index construction, deep learning semantic analysis, image feature extraction, table structure comparison, semantic matching calculation and the like are mainly adopted. According to the method, code-level structured duplicate checking, document-level multi-modal comparison and functional-level semantic analysis can be realized, and the accuracy and efficiency of product duplicate checking in the software development process are improved.

The invention relates to the technical field of automatic safety management, in particular to an automatic safety facility review system which comprises an equipment dependence evaluation module, a risk weight calculation module, a fault trend analysis module, a review priority adjustment module and a fault diagnosis decision module. According to the method, key data such as steel wire rope tension, lifting hook bearing and winch frequency are synchronously collected, linkage frequency and response delay are accurately reflected, the dynamic sensing capacity of a system is improved, and after linkage strength between facilities is recognized, key nodes are clearly examined, resource allocation is optimized, and potential hazard recognition efficiency is improved based on abnormal fluctuation and dependence degree; by extracting the periodic trend of an alarm event, defining a hidden danger development path, adjusting the inspection period and improving the pertinence of inspection, through fault fluctuation homology analysis, optimizing a fault traceability path, improving the hidden danger treatment precision, constructing closed-loop logic, improving the inspection precision and reducing resource consumption and repeated detection burden.

This invention discloses a wireless monitoring method and system for communication data of charging piles. The monitoring method includes: receiving electromagnetic signals radiated from a CAN bus; performing analog-to-digital conversion on the electromagnetic signals to obtain a digitized signal to be processed; detecting the signal amplitude of sampled data in the signal to be processed in a time sequence; when the signal amplitude of any sampled data exceeds a threshold, inverting the level state of the sampled data to obtain feature data; filling a corresponding number of follower data between two adjacent feature data to obtain a monitoring signal including feature data and follower data; and performing frame parsing on the monitoring signal based on the CAN bus data frame transmission structure to obtain communication data. The above monitoring method for detecting communication data of charging piles has low detection cost, avoids repeated data detection, and offers fast detection speed and high accuracy.

This application provides a method and apparatus for deduplicating detected targets. The method includes: acquiring suspicious items from at least two human body images of the same scanned object from different directions; defining suspicious items located at the edge of the human body contour in the first human body image from the at least two different directions as targets to be deduplicated; deduplicating suspicious items in a second human body image from the at least two different directions, based on the targets to be deduplicated, to obtain the number of duplicate suspicious items in the second human body image; and determining the total number of suspicious items carried by the scanned object based on the number of duplicates and the number of suspicious items in the first human body image. This method can effectively identify and remove the same suspicious item repeatedly detected in images from different directions, avoiding duplicate counting and improving security inspection efficiency.

The present invention discloses a method for improving the accuracy of software system delivery quality detection based on traffic playback, including: S1) collecting software system program code, configuration files, log files and user operation records; S2) cleaning and processing the collected user operation data, removing irrelevant or redundant information and adopting a standardized data format; S3) using the cleaned data to train a machine learning model and construct a baseline standard set B; S4) constructing a retest task, repeatedly testing the updated software system, and obtaining a set of data sets X to be tested; S5) calculating the degree of difference between the data set X to be tested and the baseline standard set B; S6) calculating the quality fluctuation factor, and automatically judging whether the quality of the software system meets the requirements according to the set threshold. The present invention can avoid the leakage of customer sensitive information, and improve the test coverage, efficiency and accuracy of defect detection of the software system.

The embodiment of the invention provides an online man-machine conversation optimization method, system and device, electronic equipment, a storage medium and a program product. In the scheme provided by the embodiment, when a first dialogue generation model executes a streaming response based on user input data, a currently generated first dialogue text fragment is subjected to repeated detection, and when it is detected that repeated content exists in the first dialogue text fragment, the first dialogue generation model is controlled to interrupt execution of the streaming response; in addition, reply generation constraint information is determined according to the repeated content in the first reply text segment, and then reply generation cue words are generated based on the reply generation constraint information and the non-repeated content in the first reply text segment. And after the dialogue generation cue word is input into the first dialogue generation model, the first dialogue generation model is triggered to determine a dialogue generation starting point according to the non-repeated content in the first dialogue text fragment, and streaming response is continuously executed from the dialogue generation starting point.

Content streams may be segmented to provide automatic extraction and storage of content items without intervening commercials or other unrelated content. These content items may then be stored in a database and made accessible to subscribers through, for example, an on-demand service. Automatic segmentation may include the identification of program boundaries, segmentation of a content stream based on the boundaries and the subsequent classification of the segments into content types. For example, audio and video duplication detection may be used to identify commercials since commercials tend to repeat frequently over a relatively short amount of time. A system may further identify an end of program indicator in a video stream to determine when a program ends. Accordingly, if a program ends after a scheduled end time, a recording device (e.g., the program is being recorded) may automatically extend the recording time to capture the entire program.

Provided is a storage system capable of detecting a malicious program without executing compression processing of data. A storage system includes a processor that processes data input to and output from a storage device. The processor operates thereon a duplication detection program which deduplicates duplicated data, stores the deduplicated data in the storage device, calculates a duplication rate being a ratio of duplicated data in a predetermined unit of storage, and detects a change between a duplication rate before an update of the data and a duplication rate after the update of the data in units of the predetermined unit of storage. Moreover, the processor operates thereon a ransomware detection program which detects that the data is updated by ransomware, when a decrease amount of the duplication rate exceeds a duplication rate threshold value relating to the change in duplication rate detected by the duplication detection program.

The invention discloses an approximate repetition detection method and system fusing local retrieval and multi-dimensional decision. The method comprises the steps that a to-be-detected document is preprocessed, and text content and service meta-information are extracted; utilizing MinHash to generate a compact signature; quickly recalling a candidate set from a massive historical library through an LSH index; adopting a paragraph-level weighted Jaccard algorithm to accurately calculate the structural similarity between the candidate document and the document to be detected; when the similarity exceeds a threshold value, intelligent decision making is carried out according to a preset priority chain considering multi-dimensional business rules such as release mechanism levels and time, and finally reserved documents are determined; and recording the structured audit log in the whole process for tracing. The method realizes approximate repetition detection with high recall, high precision, low time consumption and interpretable and auditing decision process, and is especially suitable for massive de-duplication scenes of strong normative texts such as government affair official documents, news announcements and the like.

Techniques described herein provide for generation of structured output for documents identified using automated near-duplicate detection. In one example, a system can receive a set of documents including at least one pair of similar documents determined to be similar to one another based on similarity scores generated using a predefined similarity scoring technique. The system can generate document groups by merging together pairs of documents that share at least one document. The system can, for each of the document groups, identify a representative document for the document group. The system can generate an output for display including a section for each document group, in which each section includes the representative document for the document group and, for each document in the document group, the similarity score relative to the representative document for the document group.