37 results about "Data deduplication" patented technology

The application discloses a prognosis risk prediction model construction system and method based on chronic disease real world data, comprising a multi-source real world data acquisition and access module, a data standardization and quality control module, a multi-dimensional feature engineering and screening module, a comorbidity feature fusion and data set construction module, a prognosis risk prediction model training and optimization module, a model multi-dimensional verification and evaluation module, a model deployment and risk stratification output module; a special multi-source real world data acquisition and quality control system for chronic liver disease, diabetes and hypertension is constructed, data standardization processing is realized in combination with a chronic disease clinical diagnosis and treatment guideline, through multi-dimensional missing value, abnormal value processing and data deduplication integration, the quality and availability of the real world data are effectively improved, the core problem of disorderly and low-quality real world data in the prior art is solved, and a high-quality data source basis is provided for model construction.

This application provides a data deduplication method, apparatus, device, storage medium, and program. The method includes: segmenting a target data stream into multiple data blocks to be stored; determining candidate data blocks from the multiple data blocks based on a first-level fingerprint index table; determining a set of anchor point feature values ​​corresponding to the candidate data blocks, and performing duplicate verification on the candidate data blocks using the anchor point index table according to the set of anchor point feature values, obtaining a first verification result; if the first verification result indicates that the candidate data block is non-duplicate data, then performing duplicate verification on the candidate data block using a second-level fingerprint index table, obtaining a second verification result; and marking candidate data blocks where the first verification result indicates duplicate data or the second verification result indicates duplicate data for deduplication. The method of this application can improve the efficiency of online deduplication.

Example implementations relate to deduplication operations in a storage system. An example includes initiating a housekeeping job to delete a backup item stored in a deduplication storage system; identifying a set of container indexes affected by the housekeeping job, the set of container indexes including at least a first container index; in response to a determination that a housekeeping work map includes a first work entry associated with the first container index, determining an ingest count of new data units added to a first container during an ingest of the backup item into the deduplication storage system, the first container associated with the first container index; and adding the determined ingest count to the cumulative counter included in the first work entry associated with the first container index.

This invention relates to the field of data deduplication technology, and discloses a method and system for deduplicating redundant information based on data feature recognition. The method includes: acquiring terminal feature data of the target object; analyzing the correlation between features based on the preprocessed terminal feature data; generating a correlation contribution degree representing the correlation between features; this scheme can retain the nonlinear correlation between features through the correlation contribution degree; adaptively distinguishing the degree of redundancy through the generated feature correlation redundancy set; further fusing the same-direction locking value and the stripping binding coefficient to obtain the redundancy confidence degree; thereby eliminating common environmental interference; and finally obtaining the deduplicated feature set through natural segmentation points and reverse feature pairs. Compared with existing linear dimensionality reduction methods such as PCA, this scheme completely preserves the local structure and high-order correlation information in high-dimensional sparse data, which can reduce the missed and false judgment rates and improve the deduplication accuracy of smart home Wi-Fi router terminal feature data.

Deterministic encryption of data is provided that facilitates subsequent data deduplication. An input data block is read. A compressibility of the input data block is determined. The data block is compressed when the compressibility exceeds a predetermined threshold. The data block is encrypted based on a keyed hash. The encrypted data block is provided with associated metadata, the associated metadata comprising an indication of the compressibility and the keyed hash.

The application provides a vector data analysis method, system and electronic equipment. The method first acquires vector data to be analyzed, and determines a fixed hierarchical grid index and a shape parameter corresponding to the vector data. Then, the fixed hierarchical grid index and the shape parameter are used to calculate a scale coefficient and an optimal grid index level of the vector data. The grid index corresponding to the optimal grid index level is determined as a calculation grid index, and the vector data is calculated in association with the calculation grid index to obtain superimposed data pairs. Cross-grid data included in the superimposed data pairs is processed by a preset data deduplication strategy. Finally, the superimposed data pairs that have completed the deduplication processing are superimposed to obtain an analysis result of the vector data. The method can create a more balanced distributed calculation, thereby greatly improving the distributed calculation performance of the vector data under different scales, fully utilizing the calculation resources and shortening the calculation time.

This invention discloses an automatic medical coding normalization method, system, and storage medium for multimodal medical data, belonging to the field of medical information technology. The method includes: collecting original textual terms and subject structured information of adverse events and / or concomitant medications, and performing text denoising, format standardization, and data deduplication; generating an initial candidate coding set based on a pre-trained medical language model; performing context-aware semantic disambiguation and rearranging of candidate codes in conjunction with subject structured information; retrieving historically confirmed coding cases from other research centers during the coding confirmation stage, outputting consistency prompts based on the differences between the current context and historical contexts, and receiving reconfirmation, coding modification, or review submission operations; and storing the final confirmation results and retrieving them to an electronic data acquisition system. This invention can improve the accuracy, processing efficiency, traceability, and multi-center coding consistency of medical coding.

The application discloses a data deduplication method and system based on a learning index, and belongs to the technical field of computer storage. The application uses a learning index structure to replace a traditional hash table. The learning index structure is a tree structure, and comprises model nodes and data nodes. An ordered fingerprint array and a buffer are arranged in each data node. The application arranges the fingerprints compactly and uses a buffer with a fixed size, thereby reducing the memory overhead of the index. When the number of fingerprints stored in the buffer of any leaf node reaches a preset number, a retraining mechanism is triggered, the model parameters and the structure layout are continuously optimized, and the index can maintain high-precision prediction and stable performance in a long-term running process. In addition, the application adopts a lightweight asynchronous retraining mechanism, so that the foreground can continue to perform query when the model is retrained, and the performance bottleneck of the traditional hash table in the expansion and rehashing process is avoided.

This invention discloses a method, device, and medium for large-scale code change analysis based on multi-batch processing, relating to the field of artificial intelligence technology. The method includes: acquiring code change data; dividing the data into multiple analysis batches and generating batch metadata for each batch, ensuring that all changes to a single file are completely within the same batch and that the data volume of each batch does not exceed a processing capacity threshold; for each analysis batch, calling a large language model for analysis based on its change data and metadata, and performing source code-based verification on the output preliminary analysis results to generate verified analysis results; integrating the verified analysis results from all batches, performing cross-batch data deduplication, correlation analysis, and structured summarization to generate a global code change analysis report. This invention achieves automated and in-depth correlation analysis of ultra-large-scale code changes.

The embodiment of the application discloses a data deduplication method and device, a computing device cluster and related products, and belongs to the technical field of big data. In the embodiment of the application, a similarity relationship between a plurality of data in a first data set is generated according to the similarity between the plurality of data; a first deduplicated data is determined according to the similarity relationship, the first deduplicated data being the data with the largest number of similarity relationships with other data in the first data set; and the first deduplicated data is deleted from the first data set. On the one hand, data deduplication is achieved, and on the other hand, compared with a deduplication scheme that only retains the data with the largest number of similarity relationships, the data deduplication scheme provided by the embodiment of the application can also retain a larger number of representative data. Therefore, compared with the deduplication scheme that only retains the data with the largest number of similarity relationships, the method provided by the embodiment of the application not only achieves data deduplication but also achieves data expansion, thereby improving the quality of the deduplicated data set.

The application discloses a data storage method and device, a storage medium and a program product. All attribute data of a document set is subjected to data deduplication to obtain a plurality of target attribute data, and all target attribute data in the document set is continuously stored in a first storage space. According to the storage position of each attribute data in the first storage space, a first index value corresponding to each attribute data is determined, and all first index values are continuously stored in a second storage space. According to the storage position of the first index value in the second storage space, a second index value corresponding to each non-empty attribute in each document is determined, and all second index values are stored in a third storage space. The application can effectively reduce the data storage amount and improve the data search performance, and can avoid any memory fragmentation. The application can be widely applied to storage technologies in scenes such as cloud technology, artificial intelligence, intelligent transportation, and auxiliary driving.

The application discloses a kind of big model training data deduplication method based on semantic clustering.The method is first filtered by Hash matching and MinHash structure to completely repeated and locally repeated text;Subsequently, a pre-training semantic encoding model is used to generate a deep semantic vector, and a density clustering method is used to construct a semantic cluster, converting global high-complexity comparison into local retrieval within the cluster;Then, combined with the efficient neighbor search based on FAISS, the semantic similar samples within the cluster are grouped and deduplicated;Finally, an information entropy sensing mechanism is introduced, and representative samples are selected according to the content complexity of the samples, to significantly reduce redundancy while maintaining corpus diversity.The application can effectively reduce the risk of model memory caused by data duplication, improve training efficiency, and reduce data processing cost, suitable for various large-scale corpus construction and privacy-sensitive scenarios.

The application relates to the field of event propagation path analysis methods, discloses an event propagation path analysis method based on RAG retrieval and information extraction technology, and discloses an event propagation path analysis method and system based on RAG retrieval and information extraction technology.The method first constructs a standardized data pool containing unique content fingerprints, realizes data deduplication and tracing by using SimHash; then constructs an exclusive RAG vector knowledge base, converts the seed context into an initial query vector; performs iterative retrieval, extracts propagation triples by using a model, and converts new entities into query vectors to trigger multi-hop retrieval to mine implicit correlations; merges nodes by using co-reference resolution and constructs a time-series directed graph; and finally calculates node centrality and vector semantic offset.The application solves the problems of propagation path breakage and content alienation recognition by iterative retrieval and semantic quantitative analysis, and realizes accurate analysis of event evolution.

This application discloses a data deduplication method, apparatus, and computer-readable storage medium, relating to the field of big data. It involves acquiring a target window from a ZSet structure at preset intervals; the target window is the first window in the ZSet structure arranged according to preset requirements; it determines whether the target window meets preset conditions; if not, it reads all key strings from the Set structure within the target window for message deduplication using the Set structure; wherein the key strings are multiple key strings containing messages generated by a message queue. Therefore, the above scheme generates key strings from messages in the message queue and utilizes the deduplication mechanism of the Set structure of windows to deduplicate key strings containing messages, achieving aggregated deduplication of data. This solves the problem of resource waste caused by data processing resource constraints and redundant data processing when dealing with large amounts of data, thus improving resource utilization.

This invention discloses a lightweight deep neural network array direction finding method based on data redundancy removal, which solves the problem that the traditional RDNN direction finding method has a large number of deep neural network nodes and cannot achieve network lightweighting. The method performs redundancy removal operation on the upper triangular elements of the covariance matrix, simplifies the number of subsequent hidden layer nodes, reduces the computational complexity of the model, and ensures that the direction finding accuracy of the model is not lost, thus realizing high-precision direction finding of lightweight deep neural networks.

The application discloses a unified storage conversion method and system for finite element analysis result data, relates to the field of data conversion, and solves the problem that current finite element result data cannot be effectively stored and converted. The method comprises the following steps: converting different original files into identifiable files through a converter plug-in, and extracting finite element result data in the identifiable files; processing three-dimensional result arrays in the finite element result data to obtain reorganized result arrays; performing data deduplication on the finite element result data, and intelligently grouping different standard physical quantities in the reorganized result arrays according to byte bits; storing node coordinate arrays, element connectivity arrays and different byte bit arrays into different subfolders respectively, and querying the byte bit arrays according to different query conditions. The application realizes effective storage and conversion of finite element result data.

This application discloses a method, apparatus, device, and medium for blocking outbound calls based on a table partitioning strategy, relating to the field of data deduplication. The method includes: receiving an outbound call number file sent by a client and verifying the file based on preset verification rules to generate corresponding verification results; parsing the file based on the verification results and preprocessing the parsed batch of numbers to obtain preprocessed batch numbers; grouping the preprocessed batch numbers based on preset table partitioning rules and performing deduplication verification on the grouped batch numbers to identify duplicate numbers; blocking outbound calls for duplicate numbers, generating an error report corresponding to the duplicate numbers, and sending the error report to the client. This allows for the import of a large number of numbers for deduplication detection, efficiently and accurately determining whether imported numbers are duplicates and blocking duplicate numbers to prevent user harassment and avoid complaint risks.

The present application relates to a data deduplication method and apparatus, a computer device, and a storage medium. The method comprises: dividing multiple pieces of data to be deduplicated into at least one data subset (S102); for each data subset, determining, on the basis of a probability filter corresponding to the data subset, a mapping value of each piece of data to be deduplicated in the data subset (S104); on the basis of the mapping value of each piece of data to be deduplicated in the data subset, correspondingly determining a data filtering result of each piece of said data in the data subset (S106); and, on the basis of the data filtering result of each piece of data to be deduplicated in the data subset, performing deduplication on the data subset (S108).

A system and a method for client-side deduplication system for a plurality of backup streams, generated by a backup and recovery client from a client database, is presented. The system includes a stream handler configured to generate a unique file name for an underlying file of each backup stream of the plurality of backup streams based on one or more data blocks in each backup stream. The system further includes a file creator configured to create a data file corresponding to each backup stream of the plurality of backup streams in a local cache of a client database server, wherein each data file has a file name corresponding to the unique file name generated by the stream handler. The client-system furthermore includes a dedupe module configured to dedupe subsequent backup streams based on the data files in the local cache.

An apparatus includes a processing means configured to maintain a deduplicated data structure including sub-sections of data block identifiers associated with different compression ratio ranges and having different numbers of data block identifiers. The processing means is further configured to identify a given data block identifier and a given compression ratio for a given data block to be stored, and to determine whether the given data block identifier is in a given sub-section having a given compression ratio range including the given compression ratio. The processing means is further configured to write the given data block to a physical space block of a storage system in response to determining that the given data block identifier is not in the given sub-section, and to increment a deduplication reference count of the given data block identifier in response to determining that the given data block identifier is in the given sub-section.

This application provides a data deduplication method, apparatus, electronic device, and machine-readable storage medium. The method includes: determining a target event for which a deduplication strategy is configured based on a detected event selection instruction; determining a target deduplication strategy for the target event based on a detected candidate deduplication strategy selection instruction; and performing deduplication processing on the received alarm data of the target event according to the target deduplication strategy. This method can flexibly configure deduplication strategies for different specific events, optimize the deduplication effect of alarm data, and effectively reduce the processing of duplicate alarm data.

Systems, methods, and software are disclosed herein for identifying duplicate blocks of a storage system and deduplicating the storage system. In one example, a method of operating a computing device includes scanning first metadata of blocks of a container file of a virtual volume to generate a first log file including records of virtual volume block numbers (VVBNs) and fingerprints of the blocks; scanning second metadata of blocks of an active file system of the virtual volume to generate a second log file including records of VVBNs and file block numbers (FBNs) of the blocks; generating tuples based on merging the records of the first log file and the second records of the second log file according to the VVBNs; identifying duplications among the blocks based on the tuples; and deduplicating the blocks based on the duplications in the active file system identified based on the tuples.

A computer-implemented method comprising: detecting (510) a cloning operation of a manifest area by a memory controller (110) of a deduplication storage system; loading a journal (130, 320) from persistent storage (140) into memory in response to the detected cloning operation, wherein the journal (130, 320) serves to store changes to a container index (220) associated with the manifest area, and wherein the container index (220) is not loaded into memory in response to the detected cloning operation; and updating the journal (220) in memory to include a display of changes to metadata of the container index (220) that is not loaded into memory, wherein the changes to the metadata are associated with the detected cloning operation.

This invention discloses a concept graph-based method for enhancing astronomical computational reasoning instructions, comprising the following steps: obtaining seed problems labeled with subdomains and topics as basic input; extracting knowledge points from the seed problems using a large language model to establish a knowledge framework covering subdomains, topics, and knowledge points; and transforming problem instances into conceptual representations; constructing a concept graph based on the knowledge framework; executing a dynamic random walk algorithm on the concept graph to output a complete concept combination; and using the complete concept combination to guide the large language model in generating astronomical computational problems and corresponding reasoning processes through example matching, data deduplication, and thought chain distillation. This invention, by constructing a concept graph and employing a random walk strategy, improves the diversity and logical rigor of the generated data, providing a new path for optimizing astronomical computational reasoning instructions.

A data deduplication method, including determining, in response to a data deduplication request, a plurality of duplicate blocks from data blocks stored in a persistent medium, determining a to-be-deduplicated block from the plurality of duplicate blocks based on deduplication priorities of the plurality of duplicate blocks, where the deduplication priority of the duplicate block is determined based on an attribute of the duplicate block, and performing deduplication on the to-be-deduplicated block, where a fragmentation degree of data in the persistent medium after the to-be-deduplicated block is deduplicated is less than a fragmentation degree of data in the persistent medium after a data block other than the to-be-deduplicated block in the plurality of duplicate blocks is deduplicated.

Methods, apparatuses, and computer readable media are configured to perform operations comprising: obtaining, by a data management system (DMS), a first snapshot of a computing system; generating, by the DMS, one or more vectors based at least in part on data from the first snapshot, the one or more vectors representative of one or more respective portions of text within one or more files represented by the first snapshot; adding, by the DMS, the one or more vectors to a vector database along with metadata or a pointer to the metadata; storing, by the DMS, the one or more respective portions of text in a secondary storage environment; and adding, by the DMS to a mapping log, respective indications of mappings between the one or more vectors and the one or more respective portions of text.

The present application relates to a method for deduplicating a file, a computer readable storage medium and a file system, the file having been divided into a plurality of data blocks and the plurality of data blocks having been deduplicated, the method comprising: defining a calculation range in the file according to a type of data block in the file, wherein the calculation range comprises a plurality of continuous data blocks in the file; generating an evaluation value according to the type of data block in the calculation range to determine whether to mark the data block in the calculation range; and re-dividing and deduplicating the marked data block in the file.

The embodiment of the application provides a data deduplication method, device, equipment, storage medium and program product, and relates to the field of big data. The method comprises the following steps: obtaining a preset marked historical data set, a preset unmarked historical sample pool and to-be-deduplicated data records, generating training candidate pairs and obtaining text sequences through serialization, inputting the text sequences into a model to obtain a repetition probability, performing uncertainty sampling based on the probability and supplementing the marked samples to the marked historical data set, iteratively updating the model parameters to obtain a target model, inputting the to-be-deduplicated data into the target model to obtain a target probability, and performing retention or merging processing on the data. The embodiment increases the mechanism of iteratively expanding the marked samples based on the unmarked sample pool and continuously updating the model parameters, improves the learning ability and generalization ability of the model for the repetition characteristics of multi-source heterogeneous data, improves the accuracy and processing efficiency of multi-source data deduplication, and reduces the dependence on initial artificial marking samples.

The application discloses a data processing method, device and equipment and a computer readable storage medium, and is applied to the field of data processing and comprises the following steps: obtaining a deduplication granularity interval; calculating a deduplication ratio corresponding to the deduplication granularity in the deduplication granularity interval, and adjusting the deduplication granularity interval according to the deduplication ratio to obtain an optimal deduplication granularity interval; determining an optimal deduplication granularity from the optimal deduplication granularity interval, and deleting repeated data in a storage space according to the optimal deduplication granularity. The application adjusts the deduplication granularity interval adaptively through the deduplication ratio to obtain the optimal deduplication granularity interval, and then further determines the optimal deduplication granularity from the optimal deduplication granularity interval, and performs data deduplication by using the optimal deduplication granularity. The flexible and accurate deduplication granularity determination method can not only ensure the normal operation of data deduplication, but also significantly improve the deduplication ratio, save the storage space and further optimize the deduplication performance.

This invention discloses a method for defending against ownership spoofing attacks in a multimedia data deduplication system. The method includes: a client using a user password as a private key to generate a mask and hash value for file data blocks, constructing a Merkle tree, and sending a commitment value to a server; the server receiving the commitment value, generating challenge information, and sending it to the client; the client generating a random index based on the challenge information, extracting sibling paths from the Merkle tree as proof, and sending it to the server; the server recalculating the hash value based on the challenge information and the client's private key, reconstructing the Merkle tree using the sibling paths, and comparing it with the client's commitment value for verification. This invention not only considers more complex attack scenarios but also demonstrates excellent performance and effectiveness in practical applications. It provides strong protection for user data while ensuring efficient system operation.