430 results about "Data deduplication" patented technology

The invention relates to the technical field of data processing, in particular to a cloud backup platform data deduplication method based on distributed storage. Comprising the following steps of: performing self-adaptive blocking on data by utilizing a dynamic blocking algorithm, generating a unique identification code and extracting multi-dimensional feature information; establishing a global index through a distributed hash table, mapping a data position and maintaining index consistency; performing redundancy analysis and similarity matching based on a deep learning algorithm, and executing a duplicate removal strategy; according to the access frequency and the load condition, distributed storage of data blocks is realized by adopting a hierarchical graph division algorithm; and a partitioning strategy, index frequency and storage layout are dynamically optimized by using a reinforcement learning algorithm, and global optimization of performance and resources is realized. According to the invention, high efficiency, accuracy and dynamic optimization of data deduplication are realized, and the data storage and management efficiency of the cloud backup platform is improved.

The invention discloses a safe and rapid storage method and system for image data of an imaging department, and relates to the technical field of medical image data processing. The method comprises the following steps: 1, designing a distributed storage system, and selecting and configuring storage nodes; 2, selecting and updating an encryption algorithm, and implementing file-level and block-level encryption; 3, realizing accurate data deduplication by utilizing a machine learning technology, and selecting a proper compression algorithm; 4, constructing an index system, and establishing a data warehouse and a data lake by using big data analysis and machine learning technologies; 5, establishing an image data backup and recovery mechanism; and 6, constructing a data integrity checking and monitoring system. According to the invention, by adopting the technologies of distributed storage, encryption, data deduplication, compression, big data analysis, machine learning, backup and recovery, data integrity verification, monitoring and early warning and the like, efficient storage and safety protection of image data are realized.

The invention provides a data deduplication method, which comprises the following steps: in response to a data deduplication request, determining a plurality of duplicate blocks from data blocks stored in a persistent medium, and determining a to-be-deduplicated block from the plurality of duplicate blocks according to the deduplication priority of the plurality of duplicate blocks, the deduplication priority of the duplicate blocks being determined based on the attribute of the duplicate blocks, and the to-be-deduplicated block being determined based on the attribute of the duplicate blocks. And then performing deduplication on the to-be-deduplicated block, wherein the fragmentation degree of data in the persistent medium after the to-be-deduplicated block is deduplicated is smaller than the fragmentation degree of data in the persistent medium after data blocks except the to-be-deduplicated block in the plurality of duplicate blocks are deduplicated. According to the method, the deduplication priorities are set for the data blocks by analyzing different fragmentation degrees after the data blocks are deduplicated, and the blocks to be deduplicated are selected from the multiple duplicated blocks for deduplication according to the deduplication priorities, so that the fragmentation degree of reading is reduced, and the reading efficiency is improved.

The invention discloses a data deduplication method and device, a storage medium, electronic equipment and a program product, and relates to the technical field of data storage, the method comprises the steps that a memory layer of a tree-shaped storage architecture is arranged in a memory, only memory read operation is needed for online deduplication, hard disk access is not needed, delay and overhead of hard disk read operation in related technologies are avoided, and the data deduplication efficiency is improved. Data writing can be quickly responded, and the data processing efficiency is remarkably improved; background deduplication is adopted in a plurality of hard disk layers, and a corresponding deduplication strategy is automatically triggered after a corresponding triggering strategy is met, so that frequent online hard disk reading and writing are reduced, the disk reading frequency is reduced, and the overall performance of the all-flash memory array is improved; wherein the memory layer, the first hard disk layer and other hard disk layers adopt different deduplication strategies, and the differentiated deduplication strategies are formulated according to the characteristics of different levels, so that the deduplication operation is more targeted and efficient, the efficiency and accuracy can be improved, and the problem of high disk reading overhead in online data deduplication in related technologies is solved.

The invention belongs to the technical field of wind power prediction, and discloses an ultra-short-term wind power prediction method based on Bayesian optimization XGboost-LSTM, and the method comprises the following specific steps: 1, inputting wind power data collected by a wind power plant and numerical weather prediction data of a corresponding time sequence; 2, data preprocessing, wherein missing value interpolation and data deduplication are carried out on input data; according to the method, an XGBoost feature optimization method is used for selecting key features influencing the wind power, the influence of irrelevant feature noise on the model prediction precision and accuracy is eliminated, then a Bayesian optimization algorithm is used for carrying out hyper-parameter tuning on an LSTM model, and finally an XGBoost-LSTM-BO model is constructed. XGboost feature optimization and Bayesian hyper-parameter optimization can obviously improve the prediction effect of the LSTM on future data and improve the model prediction precision, and compared with a traditional prediction model, the wind power data generalization ability of the model can be improved while the high prediction precision is kept, and higher prediction performance is achieved.

A data deduplication method includes: receiving a write request with first data block; writing the first data block into a storage device; writing metadata of the first data block into a first partition that is in a plurality of partitions of a metadata management structure and that is determined based on a feature of the first data block; and deleting the metadata of the first data block in the first partition and deleting the first data block from the storage device based on address information of the first data block when a fingerprint that is the same as the fingerprint of the first data block exists in the first partition. The method can prevent infrequently updated data from being evicted because resources are occupied by frequently updated data, thereby improving a deduplication ratio.

The invention discloses a data deduplication method and device, electronic equipment, a storage medium and a product. The method comprises the following steps: under the condition that data to be subjected to duplicate removal is obtained, inserting the data to be subjected to duplicate removal into a first hash table for performing hash duplicate removal until the data in the first hash table reaches a target number of rows, and outputting the data in the first target number of rows in the first hash table; under the condition that the first hash table is not full, continuing to insert the data to be subjected to duplicate removal into the first hash table for performing hash duplicate removal until the first hash table is full; under the condition that the first hash table is full, repeated detection is carried out on the data to be subjected to duplicate removal and existing data in the first hash table, and the data to be subjected to duplicate removal, which are judged to be non-repeated, are subjected to disk brushing; and under the condition that the processing of the data to be subjected to duplicate removal is completed, performing duplicate removal processing on all the data in the disk, outputting the data, and outputting the data except the data in the previous target number of rows in the first hash table. According to the scheme, the previous target number of rows of data obtained through duplicate removal can be returned in advance, and the user experience can be improved.

The invention discloses an intelligent cold data migration method and system, and relates to the technical field of data migration, and the method comprises the steps: predicting the state space of a data block through a time sequence analysis algorithm, automatically classifying the cold and hot degrees of the data block based on a clustering algorithm, and optimizing the migration data size in combination with a data deduplication technology after recognizing repeated data blocks in clustering. And carrying out association analysis on the data blocks needing to be migrated, carrying out secondary optimization on the migrated data volume, and carrying out multi-level migration processing on the data blocks needing to be migrated by utilizing an edge-cloud collaborative grading method in combination with a data block optimization result and a cold and hot degree clustering result. After the migration system automatically classifies the cold and hot degrees of the data blocks, repeated data blocks can be further identified in the clustering process, and by combining a data deduplication technology, invalid data migration is reduced, the storage utilization rate is improved, the distribution mode of storage nodes is dynamically adjusted, so that data storage resources are more balanced, and the problem that single-point storage pressure is too high is avoided; and the overall storage efficiency of the system is improved.

The invention discloses a knowledge data de-duplication method and device, a storage medium and computer equipment, relates to the technical field of data processing, is suitable for businesses such as financial science and technology and smart medical treatment, and mainly aims to solve the problem of poor de-duplication effect when large-scale knowledge data is processed in existing knowledge data de-duplication. Comprising the steps of obtaining large-scale knowledge data of related businesses; performing clustering processing on the large-scale knowledge data by adopting a MinHash LSH model to obtain a repeated text clustering result; the repeated text clustering result comprises a plurality of groups of similar data sets; performing word embedding calculation on each group of similar data sets by adopting a bge-m3 model to obtain word embedding corresponding to each group of similar data sets; and respectively carrying out semantic similarity de-duplication processing on the word embedding in each group of similar data sets to obtain a de-duplication result of the knowledge data.

Disclosed is a data deduplication method capable of reducing overhead of a CPU due to a data deduplication operation. The disclosed data deduplication method includes the steps in which: a GPU performs a deduplication operation on write-requested data when a write request is generated; and a CPU stores the write-requested data in a storage according to a result of the deduplication operation of the GPU.

A data deduplication process is performed in a storage system which includes storage nodes, and storage control nodes which can access data directly from each storage node. A first storage control node sends a message to a second storage control node to initiate a deduplication process with respect to a given data block and an original data block owned by the second storage control node. The second storage control node increments a reference counter associated with the original data block, and sends a message to the first storage control which includes metadata. The first storage control node uses the metadata to read the original data block from a given storage node, performs a data compare process to determine whether the given data block matches the original data block, and creates a reference to the original data block, if the given data block matches the original data block.

The invention discloses a data deduplication method and system based on incremental calculation and feature clustering and electronic equipment, and the method comprises the steps: receiving multi-source heterogeneous text, image, audio and video data streams in real time, adding a timestamp to each data unit, and generating an input data set with a time attribute; performing multi-modal feature extraction on the input data set, and dynamically adjusting feature weights of the extracted multi-modal features through a sliding window model and a time decay factor to obtain an increment feature vector set with the weights; executing two-stage clustering based on the feature vector set to generate a plurality of fine-grained data clusters; and comparing the intra-cluster data of each fine-grained data cluster in pairs by adopting a composite similarity model to determine intra-cluster repeated data, and performing data deduplication on the input data set according to the intra-cluster repeated data to obtain a deduplicated input data set. According to the method, the processing efficiency of the real-time data is improved, and the storage and calculation cost is remarkably reduced.

The invention discloses a data processing method and device, computer equipment and a cluster, and relates to the field of computers. A plurality of sliding windows can obtain a plurality of tangency points by processing the to-be-stored data at a time, and the calculation time consumed by the calculation device for obtaining the tangency points is shortened. If the number of the sliding windows is too large, the multiple sliding windows slide in parallel to search for the tangency points of the to-be-stored data, the storage space of the cache may be insufficient, and the cache occupancy rate is high, or the multiple sliding windows slide in parallel to determine multiple tangency points, and the calculation time consumption of the tangency points may be long; if yes, the computing device adjusts the degree of parallelism according to at least one of the computing consumed time of the first tangent point set and the cache occupancy rate so as to adjust the number of sliding windows, so that the computing consumed time of tangent points is reduced, cache occupancy is reduced, reasonable tangent points of the to-be-stored data are obtained as soon as possible, and the length of data blocks selected by the multiple sliding windows is adjusted; the accuracy of the computing device for acquiring the tangent point is improved, the acquisition rate of data repeated with the stored data in the to-be-stored data is improved, and the data deduplication rate is improved.

The embodiment of the invention discloses a chemical early warning interception method and device, electronic equipment and a medium. A specific embodiment of the method comprises the following steps: carrying out identity recognition on a target chemical detection terminal set to obtain a detection terminal identity recognition information set; performing state information extraction on the multi-modal target chemical information set to obtain a chemical multi-modal state information set; performing data deduplication on the multi-modal state information set of the chemicals and then performing association identification to obtain a multi-modal association information set; performing chemical risk identification on the de-duplicated multi-modal chemical information set to obtain a to-be-tracked chemical information set; and performing trajectory tracking early warning on the to-be-tracked chemical article information set to obtain a chemical article trajectory early warning information set, and performing early warning interception on the to-be-tracked chemical article information set. According to the embodiment, a data barrier can be broken, data sharing and safety are improved, and the accuracy of accurate interception and risk early warning of target chemicals is improved.

The invention discloses a repeated data deletion method and system based on ZNS SSD, and belongs to the field of repeated data deletion.The method comprises the steps that a fingerprint index table and a fingerprint updating cache are maintained, three-time opportunity repeated data deletion is achieved, and repeated data search is conducted in the fingerprint index table, the fingerprint updating cache and an address mapping table respectively; an address mapping table, an address reverse mapping table and a reference counting table are maintained, physical space garbage collection / defragmentation and write request merging are achieved, the address mapping table, the address reverse mapping table and the reference counting table conduct garbage collection with partition as granularity when free space is insufficient, and a fingerprint index table is not updated after garbage collection data migration; and instead, the mapping of fingerprints to migration-in physical pages is stored in a fingerprint updating cache, the physical pages are sequentially distributed in partitions by the fingerprint updating cache, normal data write requests and garbage collection data migration write requests are respectively cached, and the write requests which are continuous in sequence are merged and then issued. According to the method, the high-performance and low-cost efficient storage system can be constructed on the basis of the ZNS SSD at relatively low overhead.

The invention provides a data de-duplication method and device and electronic equipment, and the method comprises the steps: carrying out the local de-duplication processing of each text data set (global data) for training a large language model through employing a multi-round distribution local de-duplication strategy, and obtaining a first target de-duplication text set containing a plurality of pieces of target sample data; according to the data deduplication processing method and device, even under the actual limitation condition that the single-machine memory is not enough to support global data deduplication processing, the first processing device with the small memory is used for quickly carrying out multi-round local deduplication processing on the global data, the utilization rate of the memory is fully improved, the data volume of data deduplication is reduced, and the processing efficiency of data deduplication is improved. And then determining the plurality of pieces of target sample data as target training sample data for training the large language model. In this way, the data size of useless repeated data during large language model training can be effectively reduced, and the training effect and training efficiency of the large language model can be guaranteed.

The invention discloses a data deduplication method and device, a storage medium and electronic equipment, and relates to the technical field of computers. The method comprises the steps that a to-be-deduplicated data set is obtained, the to-be-deduplicated data set comprises first data and second data, the generation time of the first data is earlier than the generation time of the second data, and the first data and the second data have the same user identification; calculating a difference value between the generation time of the second data and the generation time of the first data; according to the difference value and a threshold value, determining unduplicated data in the first data and the second data; according to the generation time of the non-duplicated data, under the condition that it is determined that duplicated data exist in the third data, the duplicated data are deleted, the generation time of the third data is later than the generation time of the second data, and the third data and the first data have the same user identifier. According to the method and the device, the problems of data redundancy and storage space waste caused by excessive repeated or similar data stored by a big data platform can be solved.

A data storage management system is enhanced to accommodate, and moreover to optimize, the storing and retention of deduplicated secondary copies at write-once read-many (WORM) enabled storage platforms. Enhancements include without limitation: user interface (UI) options to enable WORM functionality for secondary storage, whether used for deduplicated or non-deduplicated secondary copies; enhancements to secondary copy (e.g., deduplication copy, backup) operations; and pruning changes. The storage manager is generally responsible for managing the creation, tracking, and deletion of secondary copies, with and without deduplication. Media agents that store secondary copies to and prune them from the WORM-enabled storage platforms also are enhanced for communicating and interoperating with both bucket-level and object-level WORM-enabled storage platforms to implement the features disclosed herein.

The invention relates to the technical field of image processing and data management, and discloses an infrared image data classification filing method and platform, and the method comprises the following steps: 1, collecting infrared image data from a plurality of different types of infrared sensors, the sensors including an infrared thermal imaging camera, a visible light camera and a radar; 2, the collected infrared image data are preprocessed, noise removal, temperature correction and image enhancement are included, and wavelet transform and a deep convolutional neural network are adopted for noise removal; through intelligent classification and management, the infrared image data processing efficiency is improved, manual intervention is reduced, the error rate is reduced, the data storage and retrieval speed is optimized by adopting multi-level storage and distributed management, the storage cost is reduced, the data safety and reliability are guaranteed through redundant backup and data de-duplication, and the data storage and retrieval efficiency is improved. And long-time reliable storage and efficient recovery of the data are ensured.

A system and method for flow-level deduplication of network traffic are disclosed. A network node receives a first plurality of packets from a first network endpoint. The first plurality of packets represent a flow of data being communicated between the first network endpoint and a second network endpoint. The network node further receives a second plurality of packets from the second network endpoint. The network node identifies a sequence identifier of each packet of the first and second pluralities of packets. The network node determines that the first and second pluralities of packets are all associated with the same flow, based on the sequence identifiers of the first and second pluralities of packets. In response to that determination, the network node deduplicates the flow by discarding the first plurality of packets or the second plurality of packets. The network node may be a traffic visibility node.

The invention discloses an intelligent examination and approval management method for the housing and construction industry, and the method comprises the steps: collecting examination and approval application data, carrying out the data cleaning, data deduplication and data standardization operation of the examination and approval application data, and obtaining preprocessed data; constructing a data classification model based on a machine learning algorithm, and performing material classification on the preprocessed data by using the data classification model to obtain a material classification result; according to a material classification result, performing primary processing and risk assessment on the preprocessed data by using an intelligent auditing system to obtain a risk score; performing secondary processing on the pre-processed data by using an expert system to obtain a personalized approval scheme; and examining and approving the preprocessed data according to the personalized examination and approval scheme, and storing the risk score and the examination and approval result to an intelligent examination and approval system. The invention relates to the technical field of examination and approval management, and solves the technical problems of low data quality, lack of flexibility and intelligence and the like of an existing residence construction examination and approval system.

A method for configurable caching, deduplication, and rate limit handling of user interface data requests according to an embodiment includes receiving a data request initiated by a user interface element of a user interface, determining whether a deduplication cache includes a deduplication entry associated with the data request, determining whether a main cache includes a cache entry associated with the data request in response to determining that the deduplication cache does not include the deduplication entry, determining whether the data request is associated with a rate-limited group of application programming interface endpoints in response to determining that the main cache does not include the cache entry or determining that the cache entry has expired, and making the data request to a backend system after a predefined server retry period associated with the rate-limited group has elapsed in response to determining that the data request is associated with the rate-limited group.

Systems and method for implementing deduplication process based on performance analyses. The system may include a processing device to determine a first performance metric associated with retrieving a second stored data block that is within a specified range of a duplicate of the first data block and a second performance metric associated with retrieving a hash value corresponding to the second stored data block. The processing device further to retrieve the second stored data block within a specified range of the duplicate of the first data block in response to the first performance metric not exceeding the second performance metric.

A computer-implemented method is disclosed for data processing. The method includes receiving real-time streaming data that includes data object identifiers, arranged in a sequenced source data array, for source data objects. The method also includes determining an integer number N of data object identifiers in the source data array and selecting three mutually prime integers N1, N2, N3 such that N1 is greater than N2, N2 is greater than N3, and an arithmetic product of N1, N2, N3 is greater than N. The method further includes generating a first, second and third modulus reduction arrays of lengths, respectively, N1, N2, and N3. The method also includes initializing the first, second and third modulus reduction arrays with dummy values, storing the source data objects in a source data object array, and storing the first, second and third modulus reduction arrays in a short term memory.

The embodiment of the invention provides a distributed data deduplication method and product. The method comprises the steps of obtaining a data key of data to be subjected to deduplication; distributing the data to be subjected to duplicate removal to the ith sub-bucket according to the mixed hash value of the data key; mapping the mixed hash value according to a hash function attribute value configured for the ith sub-bucket to obtain k position numbers; reading numerical values of positions corresponding to the k position numbers from a bit array which is stored at the current moment and corresponds to the ith sub-bucket to obtain k numbers; if it is confirmed that the k numbers are all first numerical values, it is confirmed that the data to be subjected to duplicate removal belong to duplicate data; and if at least one of the k digits is confirmed to be a second numerical value, confirming that the data to be subjected to duplicate removal belongs to new data. According to the embodiment of the invention, the speed and accuracy of big data deduplication processing are remarkably improved, and the problem of single-node calculation overload (different buckets can be distributed to different calculation nodes) is effectively solved.

The embodiment of the invention provides a big data storage optimization method based on a zero-copy collaboration technology and related equipment, and belongs to the technical field of big data storage. The method comprises a zero-copy data transmission module used for directly writing a data stream into a front-end buffer area of an annular structure through a direct memory access technology, and adopting a zero-copy algorithm based on pointer offset to parallelly separate data of each channel in a multi-thread environment; the intelligent data reduction module is used for deleting duplicated data from the data and then compressing the duplicated data; and the hybrid storage management module is used for managing the hybrid partition storage architecture and dynamically adjusting the distribution of the data in the hybrid partition storage architecture according to the data access mode. The CPU copy frequency is reduced through the zero copy technology, the data transmission efficiency is remarkably improved by combining the data reduction and intelligent layering strategies, the occupied storage space is reduced, and meanwhile the data safety and compliance are guaranteed.

Embodiments herein relate to data management and, more particularly, to collecting data from a plurality of sources, and linking the collected data to derive information and knowledge. The method includes defining at least one data model and asset by including data models, vocabulary, data quality rules, data mapping rules for at least one of, a particular data industry, a data domain, or a data subject area, importing data from a plurality of data sources, performing de-duplication of the imported data and data profiling of the imported data, and creating linked data either by semantic mapping, or by curating the data.

Techniques coordinate deduplication among nodes. The techniques involve, in response to a lookup query to search a deduplication index for an entry that maps a fingerprint which is based on incoming data, detecting an invalid result. The techniques further involve, in response to detecting the invalid result, updating the deduplication index to include an entry that maps the fingerprint, the entry initially storing an “in-progress” flag. The techniques further involve, after the deduplication index is updated to include the entry that maps the fingerprint and after the incoming data is stored in a storage location, removing the “in-progress” flag from the entry.

The invention provides a data duplicate removal method, electronic equipment and a storage medium. According to the method, smooth data transition is achieved through a double-bloom filter alternating mechanism. In first time period processing, query and addition operations are only executed on a first bloom filter of a current time window. And second time period initialization: creating a new second bloom filter at the first unit time of the second time period as a next window preparation filter. According to the double-writing preheating mechanism, data are written into the first Bloom filter and the second Bloom filter at the same time in the second time period, and the missed judgment problem during window switching is avoided. According to window switching logic, old filter resources are released, the preliminary filter is marked as a new main filter, and monthly / periodic seamless connection is achieved. According to the method, the misjudgment rate is controlled through a double-bloom filter time window switching mechanism, the data duplicate removal accuracy and the system resource utilization rate can be improved, and the method is suitable for the efficient duplicate removal requirement in a mass data collection scene.

The invention discloses a vehicle end data processing method and device, equipment and a storage medium, and relates to the technical field of vehicle end data processing, and the method comprises the steps: carrying out the data deduplication of an original CAN message of a target vehicle, and obtaining a to-be-analyzed CAN message; determining a current starting task based on the priority of the signal acquisition task; based on the rule information of the current starting task, determining a target vehicle signal in the analyzed vehicle signals corresponding to the CAN message to be analyzed, and collecting the target vehicle signal; and uploading the signal data acquired in the de-duplication acquisition time period to a cloud. Through the above mode, the received original CAN message is subjected to de-duplication, and the data acquisition time period is subjected to de-duplication before data uploading, so that repeated data uploading is avoided, repeated redundant data is reduced, vehicle end resource consumption is reduced, the priority is set between tasks, signal acquisition is performed according to the priority, the vehicle end is prevented from entering a high-load state, and the reliability of the vehicle end is improved. The treatment efficiency is improved.