186 results about "Data deduplication" patented technology

The embodiment of the invention discloses a chemical early warning interception method and device, electronic equipment and a medium. A specific embodiment of the method comprises the following steps: carrying out identity recognition on a target chemical detection terminal set to obtain a detection terminal identity recognition information set; performing state information extraction on the multi-modal target chemical information set to obtain a chemical multi-modal state information set; performing data deduplication on the multi-modal state information set of the chemicals and then performing association identification to obtain a multi-modal association information set; performing chemical risk identification on the de-duplicated multi-modal chemical information set to obtain a to-be-tracked chemical information set; and performing trajectory tracking early warning on the to-be-tracked chemical article information set to obtain a chemical article trajectory early warning information set, and performing early warning interception on the to-be-tracked chemical article information set. According to the embodiment, a data barrier can be broken, data sharing and safety are improved, and the accuracy of accurate interception and risk early warning of target chemicals is improved.

The embodiment of the invention provides a distributed data deduplication method and product. The method comprises the steps of obtaining a data key of data to be subjected to deduplication; distributing the data to be subjected to duplicate removal to the ith sub-bucket according to the mixed hash value of the data key; mapping the mixed hash value according to a hash function attribute value configured for the ith sub-bucket to obtain k position numbers; reading numerical values of positions corresponding to the k position numbers from a bit array which is stored at the current moment and corresponds to the ith sub-bucket to obtain k numbers; if it is confirmed that the k numbers are all first numerical values, it is confirmed that the data to be subjected to duplicate removal belong to duplicate data; and if at least one of the k digits is confirmed to be a second numerical value, confirming that the data to be subjected to duplicate removal belongs to new data. According to the embodiment of the invention, the speed and accuracy of big data deduplication processing are remarkably improved, and the problem of single-node calculation overload (different buckets can be distributed to different calculation nodes) is effectively solved.

The embodiment of the invention provides a big data storage optimization method based on a zero-copy collaboration technology and related equipment, and belongs to the technical field of big data storage. The method comprises a zero-copy data transmission module used for directly writing a data stream into a front-end buffer area of an annular structure through a direct memory access technology, and adopting a zero-copy algorithm based on pointer offset to parallelly separate data of each channel in a multi-thread environment; the intelligent data reduction module is used for deleting duplicated data from the data and then compressing the duplicated data; and the hybrid storage management module is used for managing the hybrid partition storage architecture and dynamically adjusting the distribution of the data in the hybrid partition storage architecture according to the data access mode. The CPU copy frequency is reduced through the zero copy technology, the data transmission efficiency is remarkably improved by combining the data reduction and intelligent layering strategies, the occupied storage space is reduced, and meanwhile the data safety and compliance are guaranteed.

The invention relates to the technical field of language generation algorithms, and discloses a natural language generation SQL method based on ES, knowledge base and interaction enhancement, and the method comprises the following four sequentially executed core steps: data feature preprocessing: carrying out deep analysis (including field type, constraint relationship, data distribution and the like) on a database table structure and data features, and carrying out data feature preprocessing; and after data de-duplication is executed, the ES is imported, and an independent structured index is established according to a table-level isolation principle to provide basic data for subsequent retrieval. According to the method, the keyword matching efficiency and the semantic understanding depth are both considered through collaborative retrieval of the double-storage-layer knowledge base in combination with Elasticsearch keyword inverted index and Milvus vector library semantic retrieval. And the results are fused through the RRF algorithm, so that the retrieval precision is remarkably improved, the limitation of a single retrieval mode is solved, and the generated SQL better fits the database structure and the user intention.

The invention provides a duplicated data deletion method based on update perception. The method comprises the following steps: acquiring a to-be-processed file; the version type of the to-be-processed file is determined and correspondingly marked, and the version type comprises a basic version and an incremental version; performing block processing on the to-be-processed file, and calculating a fingerprint corresponding to each data block; if the version type of the to-be-processed file is a basic version, deduplication is conducted on all data blocks and an existing data fingerprint set in a hash table in sequence according to fingerprints, unique data blocks are stored in a container, the hash table of the basic version is updated, and the fingerprint, the storage address and the reference frequency of each data block are recorded in the hash table; if the version type of the to-be-processed file is an incremental version, the corresponding basic version hash table needs to be loaded firstly, and then duplicate removal processing is carried out; the data index range is narrowed by distinguishing the basic version from the incremental version, so that the data processing efficiency is improved.

The invention discloses a data partitioning method and device, electronic equipment and a storage medium, and relates to the technical field of computers, and the method comprises the following steps: dividing a sliding window of to-be-processed data into a plurality of sub-regions, comparing adjacent byte pairs in the sub-regions in parallel by utilizing a vector instruction to generate a local maximum set, vector maximum comparison is carried out on a local maximum set through a tree structure so as to reduce the scale of the set layer by layer, finally, a global extreme value is determined to serve as a data block boundary, efficient data block division based on vector parallel calculation is achieved, byte-by-byte or block-by-block serial calculation logic is not adopted, and therefore efficient data block division is achieved. The technical problem that in the prior art, due to the fact that serial computing logic is adopted, computing efficiency is limited, performance bottleneck occurs, and then the processing speed and the resource utilization rate of applications such as data deduplication, compression and encryption are affected can be solved. The technical effects of improving the calculation efficiency, reducing storage fragmentation, relieving the performance bottleneck of a high-concurrency scene and improving the data processing speed and the resource utilization rate are achieved.

The invention provides a bitmap index container optimization method and system based on a multi-dimensional dynamic decision, and relates to the technical field of databases, and the method comprises the steps: constructing a feature fusion vector by calculating a skewness coefficient value and fusing a data distribution feature vector, a data access rule vector and a change trend vector; a self-adaptive threshold value is obtained through iterative optimization of a double-strategy gradient algorithm, whether the container type is switched or not is judged, data deduplication, priority ranking and batch data migration are conducted through a temporary container, and a bidirectional index table is established to redirect an access request.

The present disclosure relates to a data deduplication method, device, equipment and medium. The data deduplication method comprises: obtaining to-be-deduplicated data and identification information corresponding to the to-be-deduplicated data, the identification information comprising a first fingerprint key corresponding to the to-be-deduplicated data; determining a target storage area corresponding to the to-be-deduplicated data based on the identification information, the target storage area corresponding to a preset instance, the preset instance being used to perform a data write operation and / or a data read operation on the target storage area; performing a data read operation based on the preset instance to obtain target storage data in the target storage area, and determining whether the to-be-deduplicated data exists based on the target storage data. According to the embodiments of the present disclosure, the efficiency and accuracy of data deduplication can be improved.

The embodiment of the invention provides a data deduplication method, device and equipment. The method comprises the steps that a data storage request is received, the data storage request is used for requesting to store target data in a storage area of a server, and the data storage request comprises the data type and attribute information of the target data; determining a data state of the target data according to the data type; the data state is a deduplication state or a non-deduplication state; under the condition that the data state is a deduplication state, determining a target deduplication queue of the target data according to the attribute information; and performing data deduplication processing on the target data according to the priority of the target deduplication queue to obtain deduplication data corresponding to the target data. According to the data deduplication method provided by the invention, the data deduplication processing effect on the target data is improved.

This invention relates to the field of network connectivity control, and particularly to an intelligent network switching method, medium, and device based on a dual-connectivity architecture. When both Wi-Fi and cellular networks are enabled simultaneously, this method establishes and maintains two connections at the application layer pointing to the same server, binding the Wi-Fi and cellular network interfaces respectively. By monitoring the network quality parameters corresponding to the two connections in real time, when the Wi-Fi network quality parameter consistently falls below a preset threshold, and the cellular network consistently outperforms Wi-Fi, the system switches from Wi-Fi to cellular; when the Wi-Fi quality parameter consistently exceeds the threshold, it switches back to Wi-Fi, thus avoiding frequent switching. A unique sequence number is assigned to each data packet, and the server deduplicates data to ensure consistency. This method, through dual connectivity, real-time monitoring, differentiated switching, and data deduplication, solves problems such as untimely network switching, high latency, and poor stability, improving the communication quality and user experience of mobile terminals.

The invention relates to a big data information collection method. The method comprises the following steps: collecting retrieval data, browsing data, consulting data and auxiliary data of multiple users in an application system; invalid data removal and noise data filtering are carried out on the collected data, invalid data removal comprises null value and meaningless data removal, duplicate data de-duplication and format standardization processing, and noise data filtering comprises regular filtering and statistical filtering; and constructing an invalid and noise data judgment model based on the processed valid data and invalid and noise data. According to the method, through data preprocessing and double screening of invalid and noise data judgment models, invalid data and noise data can be effectively eliminated, so that the accuracy of data information collection is improved; through automatic judgment of invalid and noise data judgment models and an incremental updating mechanism, the models are continuously optimized along with data accumulation, the manual screening workload of data information collection can be reduced, and the misjudgment rate of automatic judgment is reduced.

The application discloses a prognosis risk prediction model construction system and method based on chronic disease real world data, comprising a multi-source real world data acquisition and access module, a data standardization and quality control module, a multi-dimensional feature engineering and screening module, a comorbidity feature fusion and data set construction module, a prognosis risk prediction model training and optimization module, a model multi-dimensional verification and evaluation module, a model deployment and risk stratification output module; a special multi-source real world data acquisition and quality control system for chronic liver disease, diabetes and hypertension is constructed, data standardization processing is realized in combination with a chronic disease clinical diagnosis and treatment guideline, through multi-dimensional missing value, abnormal value processing and data deduplication integration, the quality and availability of the real world data are effectively improved, the core problem of disorderly and low-quality real world data in the prior art is solved, and a high-quality data source basis is provided for model construction.

To reduce wasteful consumption of resources in managing deduplication in storage. [Solution] The storage control device 20 comprises a file processing unit 21 that reads or writes files in the storage area of ​​the storage in response to instructions from the host that uses the storage, a deduplication unit 22 that performs deduplication processing on the files that are the subject of reading or writing so that there are no duplicates in the storage area, and a duplication permission unit 23 that identifies the access status to the files in the storage area and, if the identified access status satisfies a set condition, excludes the corresponding file from the deduplication processing.

Techniques for keeping on-disk, physical deduplication (“dedupe”) index levels up to date to increase the efficiency of data deduplication in storage systems. The techniques include providing a plurality of dedupe indexes across a memory level and a disk level of a storage system, in which the plurality of dedupe indexes include an in-memory dedupe index at the memory level, and an on-disk dedupe index at the disk level. The techniques include applying a plurality of eviction rules regarding evicting entries of the on-disk dedupe index, in which the plurality of eviction rules avoid random evictions of entries from the on-disk dedupe index. The techniques include, in response to applying the plurality of eviction rules, identifying a plurality of entries of the on-disk dedupe index as eviction candidates. The techniques include evicting the identified eviction candidates from the on-disk dedupe index.

The invention discloses a duplication data deletion metadata management method based on a persistent memory. The method comprises the following steps: managing a global fingerprint table and a logic address to a fingerprint mapping cache in a DRAM (Dynamic Random Access Memory), and managing an additional redo log and a global mapping table in the persistent memory. The write-in path firstly writes metadata update into the DRAM and forms a transaction record, the transaction record is sequentially added to a persistent memory log in a group submission mode, persistence is aligned according to the read-write granularity of the persistent memory, and random write and lock competition is reduced; in the checkpoint stage, an aggregation strategy is dynamically selected according to the redundancy rate and the coverage write proportion obtained through online load detection, and therefore metadata write amplification is remarkably reduced on the premise that collapse consistency is guaranteed. Meanwhile, three self-adaptive deduplication modes are provided, fingerprint management is bypassed in the low redundancy stage, redundancy is sampled and estimated in the medium redundancy stage, and full deduplication is started in the high redundancy stage, so that the metadata management cost is stably reduced under different loads, and the system throughput is improved.

The invention discloses a distributed heterogeneous data deduplication method for a low-altitude Internet of Things, and the method comprises the steps: constructing an efficient distributed data unified management mechanism through the credible bookkeeping capability of a block chain, and combining the feature vector extraction of multi-modal heterogeneous data and the mapping of a shared semantic space, thereby achieving the deduplication of the distributed heterogeneous data. Semantic reservation and similarity label generation of any modal data are achieved, then key generation and recovery based on semantics are supported, the fusion management capability of heterogeneous data is improved while data security is guaranteed, meanwhile, distributed device relation clustering is carried out through a graph clustering method, and the clustering efficiency is improved. Localized heterogeneous data de-duplication is flexibly executed in combination with the device cooperation relation and the data similarity, the storage space is saved, and the management efficiency is improved.

This application provides a data deduplication method, apparatus, device, storage medium, and program. The method includes: segmenting a target data stream into multiple data blocks to be stored; determining candidate data blocks from the multiple data blocks based on a first-level fingerprint index table; determining a set of anchor point feature values ​​corresponding to the candidate data blocks, and performing duplicate verification on the candidate data blocks using the anchor point index table according to the set of anchor point feature values, obtaining a first verification result; if the first verification result indicates that the candidate data block is non-duplicate data, then performing duplicate verification on the candidate data block using a second-level fingerprint index table, obtaining a second verification result; and marking candidate data blocks where the first verification result indicates duplicate data or the second verification result indicates duplicate data for deduplication. The method of this application can improve the efficiency of online deduplication.

A method for data deduplication and compression in untrusted storage systems is provided for more efficient storage of large amounts of data in a secure manner by maintaining data integrity. The data deduplication and compression in untrusted storage systems is performed with a system comprising a set of trusted servers that are configured to trust each other and share a common encryption key.

The invention discloses a deduplication and fusion method and device for roadside sensing target data, and relates to the technical field of intelligent traffic data processing. The road side sensing target data de-duplication and fusion method comprises the following steps: S1, collecting and preprocessing motion data, spatial position data and image feature data in a vehicle monitoring process, and constructing a standardized monitoring state data set; s2, performing association analysis on spatial association degrees of the target in different cameras, and dynamically adjusting a merging process of a cross-camera detection frame; s3, carrying out coherence evaluation on the motion coherence degree of the target in adjacent time periods, and updating a recording mode of a target track in real time; and S4, comprehensively evaluating the consistency of the multi-source features of the target, and performing linkage correction on the fusion write-in process of the target state and the duplicate record rejection strategy. The problem that the conditions of traditional merging rule failure, target overlapping and boundary blur in a traffic flow dense scene are difficult to process is solved.

This disclosure relates to ransomware detection in deduplication storage systems. Example implementations relate to deduplication operations in storage systems. Examples include: receiving data units of a backup stream to be stored in a deduplication storage system; performing a first matching operation for matching data units against a first container index among a plurality of container indexes, wherein the plurality of container indexes include metadata indicating the storage location of data units previously stored in persistent storage; calculating a first ratio, the first ratio indicating the amount of deduplication that occurred during the first matching operation against the first container index; and identifying at least a portion of the backup stream as potentially affected by a ransomware attack in response to determining that the first ratio violates a condition of local ratio history data regarding the first container index.

A system call of a Filesystem in Userspace (FUSE) library at a client is mapped to first and second operations of a deduplication filesystem, the system call performing an in-kernel copy between source and destination file descriptors. A filesystem mount point of the deduplication filesystem is presented to an application residing at the client. Upon the application triggering the system call on the mount point, the system call is intercepted. First and second parameters of the system call are examined. The first parameter indicates an offset location in the source file descriptor at which a range of data to copy should start. The second parameter indicates an offset location in the destination file descriptor at which the range of data copied should be inserted. Based on the examination, one of a first or second API of a client-side library residing at the client is called.

Example implementations relate to deduplication operations in a storage system. An example includes initiating a housekeeping job to delete a backup item stored in a deduplication storage system; identifying a set of container indexes affected by the housekeeping job, the set of container indexes including at least a first container index; in response to a determination that a housekeeping work map includes a first work entry associated with the first container index, determining an ingest count of new data units added to a first container during an ingest of the backup item into the deduplication storage system, the first container associated with the first container index; and adding the determined ingest count to the cumulative counter included in the first work entry associated with the first container index.

Access object (AOB) and deduplication object (DOB) services of a deduplication filesystem are provisioned across a cluster. A client-side library receives a request to restore a file, the file being divided into chunks, the chunks being assigned to similarity groups, and the DOBs owning subsets of the similarity groups. Offset ranges of the file are assigned to the AOBs such that each offset range assigned to an AOB hosted on a node corresponds to chunks of a similarity group owned by a DOB that is also hosted on the node. Prefetches are issued to the AOBs. Multiple internal read-ahead streams are opened from the AOBs to the DOBs. The streams prefetch chunks read by the DOBs to populate read-ahead caches maintained at the AOBs. The request is serviced using the read-ahead caches.

The invention relates to the technical field of spatio-temporal data retrieval, in particular to a distributed massive spatio-temporal data quick retrieval method, which comprises the following steps of: analyzing spatio-temporal data; constructing RowKey, and creating a column family; constructing a metadata object; metadata storage is achieved through a BulkLoad mechanism, and user retrieval conditions are analyzed; the method comprises the following steps of: screening through a RegionServer, and carrying out space filtering through an Observer coprocessor deployed at a RegionServer end; and returning the local screening results returned by the RegionServers back to the driving node in a unified manner, executing data de-duplication and time sorting operations in a global dimension, and constructing a final retrieval result set. According to the method, RowKey indexing and coprocessor end-side filtering are utilized, a two-stage retrieval mechanism of global screening and local fine detection is achieved, and it is ensured that quick retrieval is achieved in a massive spatio-temporal data scene.

The invention relates to a data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: receiving data packets respectively acquired by a plurality of acquisition devices for the same target device in the same acquisition period; wherein under the condition that the acquisition equipment does not acquire complete data for the equipment index of the target equipment, the uploaded data packet carries an interval identifier of an unacquired data interval representing the equipment index; taking the first received data packet as a basic data packet, reporting collected data in the basic data packet under the condition that the basic data packet carries the interval identifier, and caching the interval identifier; and taking a data packet received after the basic data packet as a supplementary data packet, under the condition that coincident data exists between the collected data in the supplementary data packet and the data indicated by the cached interval identifier, updating the cached interval identifier based on the coincident data, and reporting the coincident data. By adopting the method, the data deduplication efficiency and reliability can be improved.

This invention relates to the field of data deduplication technology, and discloses a method and system for deduplicating redundant information based on data feature recognition. The method includes: acquiring terminal feature data of the target object; analyzing the correlation between features based on the preprocessed terminal feature data; generating a correlation contribution degree representing the correlation between features; this scheme can retain the nonlinear correlation between features through the correlation contribution degree; adaptively distinguishing the degree of redundancy through the generated feature correlation redundancy set; further fusing the same-direction locking value and the stripping binding coefficient to obtain the redundancy confidence degree; thereby eliminating common environmental interference; and finally obtaining the deduplicated feature set through natural segmentation points and reverse feature pairs. Compared with existing linear dimensionality reduction methods such as PCA, this scheme completely preserves the local structure and high-order correlation information in high-dimensional sparse data, which can reduce the missed and false judgment rates and improve the deduplication accuracy of smart home Wi-Fi router terminal feature data.

The application relates to the technical field of big data processing and network security, and discloses a ubiquitous network data deduplication system supporting similarity data ownership verification, which comprises an encryption module, a label generation module, a deduplication decision module, a challenge generation module, a proof calculation module, a verification module and an integrity evidence module; the system derives a high-entropy session key through two-dimensional chaotic mapping to encrypt plaintext data blocks, generates a similarity label with a threshold, and realizes efficient deduplication based on three-stage Hamming distance determination; the dynamic challenge-response mechanism is used to provide an auditable ownership proof for each download request, and the tree and block chain evidence technology are combined to realize multi-granularity checking of data integrity and rapid positioning of damaged blocks, so that the storage efficiency, security and usability are considered.

The application discloses a data deduplication method based on a storage system, which comprises the following steps: S1, dividing data in the storage system into a plurality of data blocks; S2, calculating a unique data fingerprint for each data block; S3, constructing a fingerprint index structure, storing a fingerprint set of the existing data blocks, and adding each newly generated fingerprint F i to the fingerprint set; S4, retaining a data block copy and marking the position of the duplicate data block in the index. i If Fi exists in the fingerprint set, it indicates that the corresponding data block is duplicate, and the step S4 is jumped to; if Fi does not exist in the fingerprint set, it indicates that the corresponding data block is a new data block, and Fi is added to the fingerprint set; S4, retaining a data block copy and marking the position of the duplicate data block in the index. i The scheme realizes the identification and deletion of the duplicate data, thereby saving the storage space and improving the utilization rate of the storage system.

The invention discloses a data deduplication method and device, electronic equipment, a storage medium and a product. The method comprises the following steps of: for data to be processed in a set batch range, performing two rounds of de-weighting on the data based on a normal mode; for the to-be-processed data not in the set batch range, one round or two rounds of deduplication are carried out according to a specified execution mode, and the specified execution mode is a normal mode or a pipeline mode and is determined according to a first round of deduplication result in the set batch range; wherein an operator of the first round of de-duplication in the normal mode is valid; the operators of the first round of de-duplication in the pipeline mode are invalid. According to the technical scheme, two rounds of deduplication are carried out on the previous batch based on the normal mode, one round or two rounds of deduplication are determined to be carried out on the subsequent batch according to the first round of deduplication result of the batches, on the basis that two rounds of deduplication are preferentially adopted to guarantee the deduplication comprehensiveness, the proper execution mode can be re-assigned in time according to the actual situation, and the deduplication efficiency is improved. And the duplicate removal efficiency and reliability are improved.

Deterministic encryption of data is provided that facilitates subsequent data deduplication. An input data block is read. A compressibility of the input data block is determined. The data block is compressed when the compressibility exceeds a predetermined threshold. The data block is encrypted based on a keyed hash. The encrypted data block is provided with associated metadata, the associated metadata comprising an indication of the compressibility and the keyed hash.