736 results about "Data deduplication" patented technology

The invention relates to the technical field of data processing, in particular to a cloud backup platform data deduplication method based on distributed storage. Comprising the following steps of: performing self-adaptive blocking on data by utilizing a dynamic blocking algorithm, generating a unique identification code and extracting multi-dimensional feature information; establishing a global index through a distributed hash table, mapping a data position and maintaining index consistency; performing redundancy analysis and similarity matching based on a deep learning algorithm, and executing a duplicate removal strategy; according to the access frequency and the load condition, distributed storage of data blocks is realized by adopting a hierarchical graph division algorithm; and a partitioning strategy, index frequency and storage layout are dynamically optimized by using a reinforcement learning algorithm, and global optimization of performance and resources is realized. According to the invention, high efficiency, accuracy and dynamic optimization of data deduplication are realized, and the data storage and management efficiency of the cloud backup platform is improved.

The invention discloses a safe and rapid storage method and system for image data of an imaging department, and relates to the technical field of medical image data processing. The method comprises the following steps: 1, designing a distributed storage system, and selecting and configuring storage nodes; 2, selecting and updating an encryption algorithm, and implementing file-level and block-level encryption; 3, realizing accurate data deduplication by utilizing a machine learning technology, and selecting a proper compression algorithm; 4, constructing an index system, and establishing a data warehouse and a data lake by using big data analysis and machine learning technologies; 5, establishing an image data backup and recovery mechanism; and 6, constructing a data integrity checking and monitoring system. According to the invention, by adopting the technologies of distributed storage, encryption, data deduplication, compression, big data analysis, machine learning, backup and recovery, data integrity verification, monitoring and early warning and the like, efficient storage and safety protection of image data are realized.

Cloud units in cloud storage include containers including data containers storing file segments, segment tree containers storing upper-level segments of segment trees representing the files, and cloud containers storing headers from the data and segment tree containers. A header for a data container includes fingerprints identifying the file segments. A header for a segment tree container includes fingerprints identifying the upper-level segments. A storage appliance stores a first level of metadata for each cloud unit in a read-write state, a second level of metadata for each cloud unit in a read-only state, and a third level of metadata for each cloud unit in an offline state. The third level requires less storage than the first and second levels. The second level requires less storage than the first level. The limited metadata space of the appliance is managed by maintaining at least a subset of cloud units in the read-only state.

The invention provides a data deduplication method, which comprises the following steps: in response to a data deduplication request, determining a plurality of duplicate blocks from data blocks stored in a persistent medium, and determining a to-be-deduplicated block from the plurality of duplicate blocks according to the deduplication priority of the plurality of duplicate blocks, the deduplication priority of the duplicate blocks being determined based on the attribute of the duplicate blocks, and the to-be-deduplicated block being determined based on the attribute of the duplicate blocks. And then performing deduplication on the to-be-deduplicated block, wherein the fragmentation degree of data in the persistent medium after the to-be-deduplicated block is deduplicated is smaller than the fragmentation degree of data in the persistent medium after data blocks except the to-be-deduplicated block in the plurality of duplicate blocks are deduplicated. According to the method, the deduplication priorities are set for the data blocks by analyzing different fragmentation degrees after the data blocks are deduplicated, and the blocks to be deduplicated are selected from the multiple duplicated blocks for deduplication according to the deduplication priorities, so that the fragmentation degree of reading is reduced, and the reading efficiency is improved.

Methods, computer systems, and computer readable medium are described for low latency reads using cached deduplicated data, including: receiving a request to read data from a storage system; query, using a generated hash value associated with the request to read data, one or more deduplication tables that corresponds to the hash value; and responsive to determining that the one or more deduplication tables includes an entry that corresponds to the hash value, using a mapping contained in the entry to perform the requested to read data, wherein the mapping includes a pointer to a physical location where at least a portion of the data is stored.

The disclosure herein describes deduplicating data chunks using chunk objects. A batch of data chunks is obtained from an original data object and a hash value is calculated for each data chunk. A first duplicate data chunk is identified using the hash value and a hash map. A chunk logical block address (LBA) of a chunk object is assigned to the duplicate data chunk. Payload data of the duplicate data chunk is migrated from the original data object to the chunk object, and a chunk map is updated to map the chunk LBA to a physical sector address (PSA) of the migrated payload data on the chunk object. A hash entry is updated to map to the chunk object and the chunk LBA. An address map of the original data object is updated to map an LBA of the duplicate data chunk to the chunk object and the chunk LBA.

The invention discloses a data deduplication method and device, a storage medium, electronic equipment and a program product, and relates to the technical field of data storage, the method comprises the steps that a memory layer of a tree-shaped storage architecture is arranged in a memory, only memory read operation is needed for online deduplication, hard disk access is not needed, delay and overhead of hard disk read operation in related technologies are avoided, and the data deduplication efficiency is improved. Data writing can be quickly responded, and the data processing efficiency is remarkably improved; background deduplication is adopted in a plurality of hard disk layers, and a corresponding deduplication strategy is automatically triggered after a corresponding triggering strategy is met, so that frequent online hard disk reading and writing are reduced, the disk reading frequency is reduced, and the overall performance of the all-flash memory array is improved; wherein the memory layer, the first hard disk layer and other hard disk layers adopt different deduplication strategies, and the differentiated deduplication strategies are formulated according to the characteristics of different levels, so that the deduplication operation is more targeted and efficient, the efficiency and accuracy can be improved, and the problem of high disk reading overhead in online data deduplication in related technologies is solved.

The invention belongs to the technical field of wind power prediction, and discloses an ultra-short-term wind power prediction method based on Bayesian optimization XGboost-LSTM, and the method comprises the following specific steps: 1, inputting wind power data collected by a wind power plant and numerical weather prediction data of a corresponding time sequence; 2, data preprocessing, wherein missing value interpolation and data deduplication are carried out on input data; according to the method, an XGBoost feature optimization method is used for selecting key features influencing the wind power, the influence of irrelevant feature noise on the model prediction precision and accuracy is eliminated, then a Bayesian optimization algorithm is used for carrying out hyper-parameter tuning on an LSTM model, and finally an XGBoost-LSTM-BO model is constructed. XGboost feature optimization and Bayesian hyper-parameter optimization can obviously improve the prediction effect of the LSTM on future data and improve the model prediction precision, and compared with a traditional prediction model, the wind power data generalization ability of the model can be improved while the high prediction precision is kept, and higher prediction performance is achieved.

The invention discloses a distributed traffic mirror image acquisition method based on an OVS flow table and a VXLAN protocol, and the method comprises the following steps: S1, initializing a working environment, and configuring an OVS service; s2, creating a virtual machine and acquiring network bridge information of the virtual machine; s3, binding the VXLAN tunnel interface to a network bridge of the virtual machine; s4, adding a flow table rule in the network bridge to realize flow matching and mirror image output; s5, the rules are dynamically adjusted through an adaptive flow table optimization algorithm, and the flow processing efficiency is improved; s6, detecting and removing repeated data packets by using an in-segment data deduplication algorithm; s7, receiving the mirror image data and carrying out traffic summarization and classification processing; and S8, completing distributed flow collection, and obtaining high-quality flow data for analysis and processing. According to the method, additional services are not needed, the deployment process is simplified, the flow collection efficiency and the data integrity are improved through dynamic adjustment and de-duplication optimization, and the method is suitable for network flow management and analysis of a large-scale cloud computing environment.

The invention discloses a real-time data deduplication method and device based on Flink, electronic equipment and a medium, and the method comprises the steps: determining a deduplication rule and a control strategy, and submitting an Flink task; acquiring real-time data in the message queue as an input data stream, determining a key value of the input data stream according to a deduplication rule, and further grouping the input data stream according to the key value through a grouping operator to obtain a grouped data stream; performing data deduplication on the grouped data stream according to the key value through a deduplication operator to obtain a main data stream and a side stream; and outputting the main data stream, and judging whether to store the side stream or not according to the control strategy. Real-time data deduplication is achieved based on the Flink, the data deduplication efficiency and reliability are improved, and the method can be widely applied to the technical field of data processing.

The invention discloses an official document data processing method in a large model scene, and relates to the technical field of natural language processing, and the method comprises the steps: automatically collecting official document data from a multi-source channel, and carrying out format conversion and preprocessing optimization to obtain a data set; performing intelligent sampling on the data set by adopting a dynamic sampling algorithm, performing intelligent review on the sampled data, and making a cleaning strategy template library according to a review result; calling a cleaning strategy template library through a strategy configuration and execution engine to perform data cleaning, and executing data deduplication and similarity management; evaluating and reviewing the cleaned data; executing a set cleaning strategy and de-duplication on a data set in a large scale by utilizing a distributed computing framework; scientifically dividing the data set subjected to cleaning and duplicate removal, and converting the data set into an efficient storage format; the whole cleaning process is monitored in real time, and key steps and performance indexes in the data cleaning process are comprehensively recorded. The document data processing efficiency and response speed can be improved, and the labor cost is reduced.

A data deduplication method includes: receiving a write request with first data block; writing the first data block into a storage device; writing metadata of the first data block into a first partition that is in a plurality of partitions of a metadata management structure and that is determined based on a feature of the first data block; and deleting the metadata of the first data block in the first partition and deleting the first data block from the storage device based on address information of the first data block when a fingerprint that is the same as the fingerprint of the first data block exists in the first partition. The method can prevent infrequently updated data from being evicted because resources are occupied by frequently updated data, thereby improving a deduplication ratio.

There are provided systems and methods for automated alert deduplication or suppression in data processing systems based on recurring data identifiers. An entity, such as company or business, may utilize computing services provided by a service provider. When providing these services, one or more computing services, processors, or the like of the service provider's computing architecture may be used. Use of computing services may generate security alerts when computing events are flagged as risky, fraudulent, malicious, computing attacks, or the like. To automate security alert management, the service provider may utilize an alert management system that may parse and extract data from incoming security alerts and calculate identifiers from such data, such as by transforming or converting using identifier functions. Recurring identifiers may be automatically organized for suppression or deduplication based on past occurrence of such identifiers with other security alerts.

The invention discloses a data deduplication method and device, electronic equipment, a storage medium and a product. The method comprises the following steps: under the condition that data to be subjected to duplicate removal is obtained, inserting the data to be subjected to duplicate removal into a first hash table for performing hash duplicate removal until the data in the first hash table reaches a target number of rows, and outputting the data in the first target number of rows in the first hash table; under the condition that the first hash table is not full, continuing to insert the data to be subjected to duplicate removal into the first hash table for performing hash duplicate removal until the first hash table is full; under the condition that the first hash table is full, repeated detection is carried out on the data to be subjected to duplicate removal and existing data in the first hash table, and the data to be subjected to duplicate removal, which are judged to be non-repeated, are subjected to disk brushing; and under the condition that the processing of the data to be subjected to duplicate removal is completed, performing duplicate removal processing on all the data in the disk, outputting the data, and outputting the data except the data in the previous target number of rows in the first hash table. According to the scheme, the previous target number of rows of data obtained through duplicate removal can be returned in advance, and the user experience can be improved.

The invention discloses an intelligent cold data migration method and system, and relates to the technical field of data migration, and the method comprises the steps: predicting the state space of a data block through a time sequence analysis algorithm, automatically classifying the cold and hot degrees of the data block based on a clustering algorithm, and optimizing the migration data size in combination with a data deduplication technology after recognizing repeated data blocks in clustering. And carrying out association analysis on the data blocks needing to be migrated, carrying out secondary optimization on the migrated data volume, and carrying out multi-level migration processing on the data blocks needing to be migrated by utilizing an edge-cloud collaborative grading method in combination with a data block optimization result and a cold and hot degree clustering result. After the migration system automatically classifies the cold and hot degrees of the data blocks, repeated data blocks can be further identified in the clustering process, and by combining a data deduplication technology, invalid data migration is reduced, the storage utilization rate is improved, the distribution mode of storage nodes is dynamically adjusted, so that data storage resources are more balanced, and the problem that single-point storage pressure is too high is avoided; and the overall storage efficiency of the system is improved.

The invention discloses an informatization asset identification method based on machine learning. The method comprises the following steps: step 1, data acquisition and preprocessing; step 2, cross-platform data integration; 3, designing and training a context awareness model; and 4, dynamic asset identification and decision making. By designing the cross-platform data integration module and combining the machine learning model to perform data deduplication and consistency check, standardization and efficient integration of multi-source data are realized. Compared with a traditional manual data arrangement mode, the method has the advantages that structured and unstructured data can be automatically processed, and the accuracy and integrity of the integrated data are ensured, so that the condition of information redundancy or data conflict is avoided.

The invention discloses a knowledge data de-duplication method and device, a storage medium and computer equipment, relates to the technical field of data processing, is suitable for businesses such as financial science and technology and smart medical treatment, and mainly aims to solve the problem of poor de-duplication effect when large-scale knowledge data is processed in existing knowledge data de-duplication. Comprising the steps of obtaining large-scale knowledge data of related businesses; performing clustering processing on the large-scale knowledge data by adopting a MinHash LSH model to obtain a repeated text clustering result; the repeated text clustering result comprises a plurality of groups of similar data sets; performing word embedding calculation on each group of similar data sets by adopting a bge-m3 model to obtain word embedding corresponding to each group of similar data sets; and respectively carrying out semantic similarity de-duplication processing on the word embedding in each group of similar data sets to obtain a de-duplication result of the knowledge data.

An information handling system executing a predictive hash table and carbon dioxide (CO2) minimizing system may comprise a network interface device to receive from a remote data center deduplication process workload metrics, fingerprint hash table size metrics, and a current reserved memory size allocated for storage of a fingerprint hash table, a processor executing code instructions to predict a future workload of deduplication processes and a future fingerprint hash table size to determine a predicted amount of memory to reserve for storage of a future fingerprint hash table based on the predicted future workload of deduplication processes and fingerprint hash table size, and to determine an over-allocated portion of the current reserved memory size where the network interface device transmits an instruction to the data center to limit power provided to the over-allocated portion of the current reserved memory size.

Disclosed is a data deduplication method capable of reducing overhead of a CPU due to a data deduplication operation. The disclosed data deduplication method includes the steps in which: a GPU performs a deduplication operation on write-requested data when a write request is generated; and a CPU stores the write-requested data in a storage according to a result of the deduplication operation of the GPU.

Described are systems and methods for deduplicating shared dependencies for micro-frontend applications. Embodiments of the present disclosure can provide hosting of shared dependencies on a networked resource, such as a content delivery network (CDN), and import statements of the shared dependencies can be replaced with calls to a network identifier specifying a network location where the shared dependencies are hosted. Accordingly, the shared dependencies can be natively imported from the network location and shared among the micro-frontend bundles.

The invention discloses an automobile open source boundary research method based on a large language model, a medium and terminal equipment, and the method comprises the steps: carrying out the static code analysis of an extracted program file, and obtaining first open source component information; performing large language model analysis on the extracted engineering construction file and configuration file to obtain second open source component information; the firmware of the automatic driving system is analyzed, and third open source component information is obtained by jointly using an SCA software component analysis technology; analyzing the configuration file of the Apk through a large language model to obtain fourth open source component information; and performing data merging and data deduplication on the first open source component information, the second open source component information, the third open source component information and the fourth open source component information. According to the method, the construction information of the source code project and the configuration information in the firmware are analyzed through the large model technology, and then the open source component information is obtained, so that the obtaining efficiency and effectiveness of the open source information are improved.

A data deduplication process is performed in a storage system which includes storage nodes, and storage control nodes which can access data directly from each storage node. A first storage control node sends a message to a second storage control node to initiate a deduplication process with respect to a given data block and an original data block owned by the second storage control node. The second storage control node increments a reference counter associated with the original data block, and sends a message to the first storage control which includes metadata. The first storage control node uses the metadata to read the original data block from a given storage node, performs a data compare process to determine whether the given data block matches the original data block, and creates a reference to the original data block, if the given data block matches the original data block.

The invention discloses a data deduplication method and system based on incremental calculation and feature clustering and electronic equipment, and the method comprises the steps: receiving multi-source heterogeneous text, image, audio and video data streams in real time, adding a timestamp to each data unit, and generating an input data set with a time attribute; performing multi-modal feature extraction on the input data set, and dynamically adjusting feature weights of the extracted multi-modal features through a sliding window model and a time decay factor to obtain an increment feature vector set with the weights; executing two-stage clustering based on the feature vector set to generate a plurality of fine-grained data clusters; and comparing the intra-cluster data of each fine-grained data cluster in pairs by adopting a composite similarity model to determine intra-cluster repeated data, and performing data deduplication on the input data set according to the intra-cluster repeated data to obtain a deduplicated input data set. According to the method, the processing efficiency of the real-time data is improved, and the storage and calculation cost is remarkably reduced.

The invention discloses a data processing method and device, computer equipment and a cluster, and relates to the field of computers. A plurality of sliding windows can obtain a plurality of tangency points by processing the to-be-stored data at a time, and the calculation time consumed by the calculation device for obtaining the tangency points is shortened. If the number of the sliding windows is too large, the multiple sliding windows slide in parallel to search for the tangency points of the to-be-stored data, the storage space of the cache may be insufficient, and the cache occupancy rate is high, or the multiple sliding windows slide in parallel to determine multiple tangency points, and the calculation time consumption of the tangency points may be long; if yes, the computing device adjusts the degree of parallelism according to at least one of the computing consumed time of the first tangent point set and the cache occupancy rate so as to adjust the number of sliding windows, so that the computing consumed time of tangent points is reduced, cache occupancy is reduced, reasonable tangent points of the to-be-stored data are obtained as soon as possible, and the length of data blocks selected by the multiple sliding windows is adjusted; the accuracy of the computing device for acquiring the tangent point is improved, the acquisition rate of data repeated with the stored data in the to-be-stored data is improved, and the data deduplication rate is improved.

The embodiment of the invention discloses a chemical early warning interception method and device, electronic equipment and a medium. A specific embodiment of the method comprises the following steps: carrying out identity recognition on a target chemical detection terminal set to obtain a detection terminal identity recognition information set; performing state information extraction on the multi-modal target chemical information set to obtain a chemical multi-modal state information set; performing data deduplication on the multi-modal state information set of the chemicals and then performing association identification to obtain a multi-modal association information set; performing chemical risk identification on the de-duplicated multi-modal chemical information set to obtain a to-be-tracked chemical information set; and performing trajectory tracking early warning on the to-be-tracked chemical article information set to obtain a chemical article trajectory early warning information set, and performing early warning interception on the to-be-tracked chemical article information set. According to the embodiment, a data barrier can be broken, data sharing and safety are improved, and the accuracy of accurate interception and risk early warning of target chemicals is improved.

The invention discloses a cross-port data migration management system and method, and relates to the technical field of data migration. According to the method, the representative fingerprints BL are selected after the similar data are clustered, the representative fingerprints BL are fingerprints b with the minimum fingerprint value, and the representative fingerprints BL form the representative fingerprint set, so that the fingerprint comparison range is expanded, the probability of finding repeated fingerprints is improved, similarity calculation does not need to be carried out on each piece of data once, and the accuracy of fingerprint comparison is improved. The cluster similar to the to-be-input data in mouth can be found in the buffer database for de-duplication, so that the cluster most similar to the to-be-processed data can be efficiently positioned in each de-duplication operation; after the buffered data fingerprints are clustered, the fingerprints are classified in a layered manner through the classification labels, so that the fingerprint comparison amount in the similarity calculation process is reduced, the hit rate of the fingerprints accessing data in the future is increased, the query efficiency of fingerprint indexes in a buffer database is ensured while the maximum utilization of the memory space is ensured, and the user experience is improved. And the performance of deleting the repeated data is ensured.

The invention discloses a repeated data deletion method and system based on ZNS SSD, and belongs to the field of repeated data deletion.The method comprises the steps that a fingerprint index table and a fingerprint updating cache are maintained, three-time opportunity repeated data deletion is achieved, and repeated data search is conducted in the fingerprint index table, the fingerprint updating cache and an address mapping table respectively; an address mapping table, an address reverse mapping table and a reference counting table are maintained, physical space garbage collection / defragmentation and write request merging are achieved, the address mapping table, the address reverse mapping table and the reference counting table conduct garbage collection with partition as granularity when free space is insufficient, and a fingerprint index table is not updated after garbage collection data migration; and instead, the mapping of fingerprints to migration-in physical pages is stored in a fingerprint updating cache, the physical pages are sequentially distributed in partitions by the fingerprint updating cache, normal data write requests and garbage collection data migration write requests are respectively cached, and the write requests which are continuous in sequence are merged and then issued. According to the method, the high-performance and low-cost efficient storage system can be constructed on the basis of the ZNS SSD at relatively low overhead.

The invention belongs to the technical field of ship engineering and information, and particularly relates to a BOM data integrated storage and management method applied to ship overall design software, which comprises the following steps: defining a metadata model of BOM data in the ship overall design software, including part types, attributes, relationships and constraint information; the management process of the BOM data is simplified through a unified data storage library and standardized processing, and the working efficiency of a design team is improved. And the data error rate is reduced: through data deduplication and redundancy processing, the possibility of data errors is reduced, and the design accuracy is improved. And through data integration and establishment of an association relationship, data exchange and sharing among different majors are promoted, and collaborative design of multiple majors is facilitated. Through data standardization and naming specification, the query and manageability of the BOM data are improved, and a design team can conveniently search and use the data.

According to the method and system for constructing the corpus based on the webpage data set, the large-scale and high-quality corpus can be constructed based on the massive webpage data set through a series of operations of mass filtering, weight removal, hatred speech detection and confusion degree calculation; the steps of quality filtering and data deduplication greatly reduce repeated and low-quality contents in a corpus, and improve the generalization ability of the model and the reliability of output; in addition, high-standard and multi-level quality of the corpus is further ensured through hatred speech detection and puzzle calculation, potential negative effects are reduced, and the model is more accurate and efficient when processing various natural language tasks.