160 results about "Erasure code" patented technology

The invention discloses an enterprise financial consultation platform based on financial data analysis, and relates to the technical field of enterprise financial data analysis, and the platform comprises a storage medium health monitoring module which collects physical health indexes of storage equipment in real time, constructs a dynamic portrait of the health degree of a storage medium, marks a high-risk storage area, and stores the physical health indexes; generating a dynamically updated storage medium health thermodynamic diagram; according to the method, the physical health indexes of the storage equipment are collected in real time, and the dynamic portrait of the health degree of the storage medium is constructed, so that the aging risk of the storage medium is accurately identified, invisible hardware aging is converted into quantifiable risk information, the risk identification capability is improved, dual-mode verification of a bit layer and a semantic layer is adopted, and the risk identification efficiency is improved. The functions of identifying high-risk silent data errors and capturing local damage of the storage medium are realized; and through a hierarchical repair strategy, the document repair effect is improved by utilizing an erasure code decoder and a FinDoc-RepairGAN processor.

The invention relates to the field of distributed storage, and discloses a distributed memory system based on erasure codes and a dynamic repair mechanism. Comprising a data node cluster, a metadata management cluster, a bandwidth token management module, a dynamic repair composer and a repair agent node. And the data node cluster divides the object into strip generation data fragments and verification fragments and dispersedly stores the data fragments and the verification fragments according to a placement rule. And the metadata management cluster maintains object stripe mapping, a fragment position set, erasure code protection configuration and a version epoch, and records a repair intention log. And the bandwidth token management module grants repair execution permission according to the fault domain budget. The dynamic repair composer generates availability repair and durability repair tasks, the temporary check fragments are written in when the number of the available check fragments is lower than a threshold value in the availability stage, and the configuration fragments are recovered and the temporary fragments are recovered in the durability stage. And the consistency of version epochs is verified before the agent node is repaired and written back, and if not, recalculation is carried out.

The invention discloses an efficient double-link block chain consensus method and device, equipment and a storage medium, and belongs to the field of information security. The method comprises an initialization stage, an input stage, a coding stage, a broadcasting stage and a consensus stage. According to the invention, the parallel design of a coding broadcast link and a consensus confirmation link is adopted, so that the occurrence frequency of transaction congestion is reduced; the integrity required by the asynchronous Byzantine consensus process is realized through three-step broadcasting, the updating problem of the post-quantum component does not need to be considered, and the security is higher; an asynchronous network and a Byzantine opponent model are adopted, so that malicious node attacks which do not exceed the total number of nodes can be tolerated on the premise of not setting a network time delay upper limit; in the encoding stage, the generation of the redundancy share through the RS erasure code can be realized by adopting preprocessing, the data redundancy expansion multiple in the broadcasting stage is far smaller than that of the same type of protocol, and the original transaction can be recovered by collecting part of effective shares in the consensus stage, so that the system time delay is effectively reduced, and the system throughput is improved.

The invention provides a distributed cluster security backup method and system oriented to a cloud-side environment, and relates to the technical field of distributed storage systems, and the method comprises the steps: obtaining cloud-side cluster data, and adaptively selecting a backup mode based on an erasure code and a multi-copy hybrid redundancy strategy; establishing a workload-aware incremental replication mechanism, and optimizing fault recovery by using idle resources; the problem of lock competition is solved by adopting a multi-copy consistency guarantee mechanism based on two-stage submission, and lock-free reading is realized based on MVCC. According to the invention, the data backup flexibility, the fault recovery efficiency and the system consistency guarantee capability are improved.

The invention discloses a distributed data storage and access control method based on a block chain, and relates to the technical field of data security. The method comprises the following steps: fragmenting user data, encrypting the fragmented user data by using an encryption algorithm, and storing the encrypted user data in a plurality of distributed nodes; taking the hash value and the storage position of the data fragment as metadata to be recorded in a block chain, and performing integrity verification through a smart contract; when a user accesses, user attributes are verified based on an attribute-based encryption technology, and an access control strategy is dynamically executed through an intelligent contract; and an erasure code technology is adopted to carry out redundancy storage so as to realize data recovery and fault tolerance. The method effectively solves the problems of single-point failure, privacy disclosure and inflexible access control in traditional centralized storage, has the beneficial effects of improving data security, guaranteeing integrity, realizing fine-grained dynamic access control and enhancing system reliability, and is suitable for data security management in cloud computing, Internet of Things and big data environments.

The embodiment of the invention relates to a data processing method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the coding operation of an erasure code on first data under the condition of receiving a write-in request for the first data, and obtaining a plurality of logic units and a plurality of fragments; according to the first identifier of the logic unit and the second identifier of each fragment, writing the plurality of fragments into a storage node; and under the condition that a reading request is received, obtaining a target first identifier and a target second identifier of the second data, reading the plurality of target fragments of the second data from the plurality of storage nodes, and carrying out merging processing to obtain the second data. Therefore, the data can be divided into the fragments with the unique identifiers for storage in the writing stage, and the target fragments can be quickly positioned according to the logic addresses and the recovered data can be merged in the reading stage, so that the metadata query overhead is reduced, the reading and writing efficiency is improved, and the storage and network transmission cost is reduced; and the overall performance of the distributed storage system in a high-concurrency scene is improved.

The invention provides a temporary check block repairing method and system based on erasure codes in distributed block storage, and the method comprises the steps that a distributed block storage system detects that a check block fails to be written in, and a metadata service creates a temporary check block according to an erasure code mode; the temporary check block records an operation log and executes data writing, and replays IO operation according to a data version number; repairing the inconsistency of the data blocks by using the operation log of the temporary check block; and quickly recovering the check block data through the data bitmap of the temporary check block. According to the rapid recovery processing method for the temporary check block, the check block data can be directly recovered without a coding mode, the internal IO is reduced, and the recovery speed of the erasure code check block is increased.

A fragment and shuffle erasure coding operation is disclosed. In a transformative operation, encrypted fragments are generated from an input file. The transformative operation includes one or more iterations of chunking the file and mixing the chunks to generate a mixed sequence. The mixed sequence is sliced into encrypted fragments. The encrypted fragments are shuffled in a storage system such that physical locations of the encrypted fragments change over time.

In some embodiments, a method for die-level monitoring is provided. The method includes distributing user data throughout a plurality of storage nodes through erasure coding, wherein the plurality of storage nodes are housed within a chassis that couples the storage nodes. Each of the storage nodes has a non-volatile solid-state storage with non-volatile memory and the user data is accessible via the erasure coding from a remainder of the storage nodes in event of two of the storage nodes being unreachable. The method includes producing diagnostic information that diagnoses the non-volatile memory on a basis of per package, per die, per plane, per block, or per page, the producing performed by each of the plurality of storage nodes. The method includes writing the diagnostic information to a memory in the storage cluster.

The invention belongs to the technical field of computer storage, and particularly discloses an encryption difference compressed data restoration method and device based on similarity perception. According to the method and the device, the differential data block and the reference block which are subjected to aggregation storage are read from the continuous disk space at one time; in the data recovery process, pipelining processing is carried out on a difference decoding task and an erasure code decoding task; and performing data restoration according to the size of the data slice, the difference data block and the reference block by executing the pipelined difference decoding task and the pipelined erasure code decoding task. According to the mode, the whole aggregation storage data block containing the differential data block and the reference block is read from the continuous disk space, compared with respective reading, one-time extra disk overhead can be reduced, then the differential decoding task and the erasure code decoding task are subjected to pipelining processing, and the decoding efficiency is improved. And data recovery is carried out by executing the decoding task, so that the data recovery performance of the encryption difference compression system can be effectively improved.

The invention discloses an HDFS (Hadoop Distributed File System) intelligent hierarchical storage method and system based on multi-dimensional popularity perception and tape library integration, and belongs to the technical field of distributed file systems, and the method comprises the following steps: expanding HDFS metadata to construct a unified namespace; acquiring data block access features and dynamically dividing hot, warm and cold data by using a K-means clustering algorithm; executing a self-adaptive grading strategy: adding a copy to hot data and storing in an SSD (Solid State Disk), maintaining a default copy to warm data and storing in a disk, and migrating cold data to a tape library and storing by adopting an erasure code; scattered storage and fault tolerance of relevance perception are realized through a message queue; metadata persistence and periodical re-evaluation are carried out, and automatic migration and dynamic adjustment of data among the SSD layer, the disk layer and the tape library layer are achieved. By adopting the method and the system, the storage strategy and the data access value are accurately matched, the storage cost is finally reduced, and the system I / O performance and the resource utilization rate are improved.

The invention relates to the technical field of computer science, and discloses an efficient file storage system with a distributed architecture, which comprises the following modules: a distributed file storage module for segmenting file data and distributing the file data to a plurality of storage nodes by adopting a distributed storage technology to ensure high availability and expansibility of the data; the intelligent data management module is used for dynamically managing data loads among storage nodes based on a data distribution algorithm and supporting automatic expansion and optimization of storage resources; the high-speed file access module is used for ensuring that a user can efficiently perform file read-write operation through a distributed access mechanism and an optimized data transmission protocol; and the data redundancy and backup module is used for providing a flexible redundancy strategy and combining a replica redundancy technology and an erasure code technology. By optimizing the file storage technology, the storage efficiency can be improved, the storage space can be reasonably utilized, dynamic expansion of the system is supported, and the problems that an existing system is slow in response and poor in expansibility when coping with large-scale data are solved.

The technical scheme provides a data error correction and performance optimization method based on multiple optical disks, comprising the following steps: S1, generating a logical volume which is in a temporary data state before data recording and comprises a data block and a verification block; s2, converting the logical volume into a physical volume through a recombination algorithm, wherein the physical volume is a mirror image file which can be directly recorded to an optical disc; s3, respectively burning the physical volume to a plurality of corresponding optical disks, and synchronously mounting the optical disks through a plurality of CD drivers to realize concurrent read-write operation on the physical volume; and S4, when the data block in any optical disc is damaged, reversely calculating the position of the damaged data block in the logical volume through the recombination algorithm, calling undamaged data blocks and verification blocks corresponding to the damaged data block in other optical discs, and recovering the damaged data block based on an erasure code algorithm. According to the technical scheme, the reading performance of the large file can be optimized, and data protection can be provided.

The application relates to the technical field of distributed storage, and discloses a data storage method and device, computer equipment and a storage medium, the method comprising the following steps: in the case of using a storage node to perform erasure code calculation, whenever receiving to-be-stored data, randomly selecting a storage node and sending the to-be-stored data to the storage node; using the storage node to receive the to-be-stored data, obtaining instruction set information of the highest level supported by a central processing unit of the storage node, obtaining an encoding function matched with the instruction set information, performing encoding processing on the to-be-stored data based on the encoding function, and obtaining an erasure code corresponding to the to-be-stored data; and associating the erasure code with the to-be-stored data. The application adaptively selects the instruction set information of the highest level supported by the central processing unit, uses the encoding function corresponding to the instruction set information to perform erasure code calculation, and obtains the erasure code, so that the calculation efficiency of the erasure code is improved.

The invention discloses a threat intelligence synchronization erasure code redundancy check and incremental error correction method, which relates to the technical field of network security, and comprises the following steps: S1, obtaining a threat intelligence data block sequence, extracting an original hash value and cyclic redundancy check of each data block from a distributed transmission path, and obtaining a threat intelligence data block sequence; s2, according to the preliminary verification result, judging whether the integrity index is lower than a preset threshold value or not, if the integrity index is lower than the preset threshold value, positioning an abnormal data block in the sequence, and determining a pollution position by comparing a hash value of each data block with parity check verification; according to the erasure code redundancy check and incremental error correction method for threat intelligence synchronization, high-reliability transmission and rapid recovery of threat intelligence data in a distributed environment are realized by integrating error correction, verification and synchronization mechanisms, and the data security and the system robustness are effectively improved.

This application belongs to the field of computer storage technology, specifically disclosing a cross-rack sensing invalid data reclamation method and apparatus based on erasure coding. The method includes: determining the cross-rack update network traffic generated by simulating the reclamation of invalid data blocks in each erasure coding stripe based on the invalid data blocks and valid check blocks stored in each rack; dividing each erasure coding stripe into filling stripes and reclamation stripes based on the cross-rack update network traffic corresponding to each erasure coding stripe; migrating the valid data blocks of each reclamation stripe to the storage nodes where the invalid data blocks in each filling stripe are located, and releasing the storage space of each reclamation stripe; and updating the valid check blocks of each filling stripe. This application achieves a more scientific and efficient invalid data reclamation method, effectively reducing network traffic throughout the invalid data reclamation process, improving invalid data reclamation efficiency, while ensuring system storage efficiency.

A method for proactively rebuilding user data in a plurality of storage nodes of a storage cluster is provided. The method includes distributing user data and metadata throughout the plurality of storage nodes such that the plurality of storage nodes can read the user data, using erasure coding, despite loss of two of the storage nodes. The method includes determining that one of the storage nodes is unreachable and determining to rebuild the user data for the one of the storage nodes that is unreachable. The method includes reading the user data across a remainder of the plurality of storage nodes, using the erasure coding and writing the user data across the remainder of the plurality of storage nodes, using the erasure coding. A plurality of storage nodes within a single chassis that can proactively rebuild the user data stored within the storage nodes is also provided.

The present application relates to a kind of data reliable storage methods based on erasure code adaptive coding, the application is saved by being encoded as erasure code to data, and under different workloads, different coding strategies are converted, so that the storage system can adapt to different working environment, maintain the reliability of data.The present application is composed of two components of improved erasure code and adaptive coding, improved erasure code is improved based on existing erasure code, the performance of existing erasure code is enhanced by the XOR grouping coding XGC proposed in the present application, to provide the basis for adaptive coding.Component two judges work load and changes coding strategy based on the hot and cold characteristics of data and the use rate of storage system.

The invention provides an asynchronous data distribution method, computing equipment and a storage medium. The method comprises the steps that a first node applies an erasure code algorithm to original data to generate n data fragments; wherein the first node is any network node with original data; the first node applies an error correction code algorithm to the n data fragments to generate n verification fragments; the first node broadcasts the first specified data fragment and the first specified verification fragment to other nodes, and sends the n verification fragments to the corresponding network nodes according to the first corresponding relation, so that the second node determines original data according to the received multiple data fragments and multiple verification fragments; wherein the first corresponding relation is used for indicating the corresponding relation between the data fragment and the verification fragment and the network node; the second node is any network node without original data. According to the scheme, data can be efficiently and reliably distributed in an unreliable asynchronous network.

The application discloses an ordered repairing and deleting method of a distributed storage system, and comprises the following steps: constructing a repairing partition key for repairing information, and storing the repairing information in the distributed storage in order based on the repairing partition key; constructing a deleting partition key for deleting information, and storing the deleting information in the distributed storage in order based on the deleting partition key; distributing each erasure code group to a corresponding repairing work process and deleting work process, so that the repairing work process is configured with the repairing partition key, the corresponding repairing information is pulled from the distributed storage, a data repairing operation is performed, the deleting work process is configured with the deleting partition key, the corresponding deleting information is pulled from the distributed storage, and a data deleting operation is performed. The application effectively avoids problems such as excessive system memory pressure, data reliability decline and unstable system operation caused by unordered organization and unordered scheduling, and improves the stability and scheduling efficiency of the repairing and deleting tasks under the Quorum intelligent read-write architecture.

The application discloses a kind of unmanned plane identity anonymization and safe data outsourcing method and system, method is based on integrated supervision platform and unmanned plane, including system initialization stage, integrated supervision platform generates main private key and public parameter;User anonymous and private key generation stage, unmanned plane registers anonymous identity and obtains user private key;Data preprocessing and outsourcing stage, unmanned plane divides data into block and sub-block and encrypts, generates coding vector using erasure code coding, generates verification label for each coding data block, constructs random distribution table according to the number of cloud server, distributes coding vector and label to multiple cloud server storage.In this way, the unmanned plane identity anonymization and safe data outsourcing method described reduces storage redundancy, ensures that data can be reconstructed when multiple servers fail, and guarantees data availability and robust recovery.

The application relates to the computer technical field and discloses a data processing method based on erasure code and related devices. The application obtains a plurality of original data blocks; divides the plurality of original data blocks into two sequential groups and a plurality of cross-group data block groups; the original data block serial numbers in the sequential groups are continuous, each cross-group data block group contains the same number of original data blocks, and the cross-group data block serial numbers are discontinuous; a plurality of check blocks of the plurality of original data blocks are generated, the plurality of check blocks include two global check blocks, sequential group check blocks corresponding to each sequential group, and cross-group check blocks corresponding to each cross-group data block group. Thus, in the case of occupying fewer hard disks, data recovery can be performed in a local scope in the data block recovery process, global data recovery is maximally reduced, reconstruction cost is effectively reduced, and higher fault tolerance is achieved under the same condition.

The invention discloses an industrial data disaster recovery backup method and system based on edge computing, and relates to the technical field of industrial internet and data security. A decentralized collaborative backup grid is constructed, so that scattered industrial edge nodes can be autonomously discovered and connected to form a dynamic backup neighbor relation network; a grid topology is automatically constructed among nodes through mutual perception and authentication of a service discovery protocol, on this basis, data to be backed up generated by any node can be encoded into fragmented data blocks and verification blocks through an erasure code technology, and the fragmented data blocks and the verification blocks are intelligently scheduled to a plurality of backup neighbor nodes for distributed storage. By means of continuous heartbeat detection and multi-dimensional health index analysis, the states of all nodes in a grid are monitored in real time, once it is detected that the nodes are abnormal, a data reconstruction process based on erasure codes is automatically triggered without a central instruction, original data are recovered through surviving coding blocks, and a self-organizing and self-healing disaster recovery system is formed.

This invention relates to the field of distributed storage technology and provides a data collaboration method and system for distributed transaction terminals. The method involves each transaction terminal performing erasure coding on its local ledger data to generate a set of data fragments consisting of original data fragments and verification data fragments. These data fragments are then distributed and stored across different transaction terminals according to distributed storage rules, and the terminals exchange online status and fragment integrity information. When a request to access the target ledger is received, the terminal storing its fragments is located based on the ledger identifier, and the fragments are retrieved in parallel to reconstruct the target ledger. When a terminal is detected to be offline or a fragment is corrupted, online terminals collaborate to reconstruct the original data using surviving fragments, re-encode new fragments, and distribute them to healthy terminals. This invention, through erasure coding and distributed collaborative storage, reduces storage overhead while improving system fault tolerance and data availability, achieving efficient and reliable distributed ledger data collaborative management.

The invention discloses a data storage method and device, equipment, a medium and a program product, and relates to the technical field of electronic computers.The data storage method comprises the steps that to-be-stored original data are obtained, the original data are divided into first-level data strips according to the (K, M) redundancy ratio, first-level verification strips are obtained through coding calculation, and the first-level verification strips are stored in the first-level verification strips; the first-level data strip and the second-level check strip form a first-level erasure code, the first-level erasure code serves as original data, a second-level erasure code is constructed according to the (K-1, M) redundancy ratio, and the first-level erasure code and the second-level erasure code are stored in different storage nodes. According to the method, the multi-level erasure codes are constructed, so that the data recovery capability, the system fault-tolerant capability and the node utilization rate are remarkably improved when the problems that the data reliability is insufficient and the fault-tolerant capability is limited in a storage system are faced.

A data synchronization method, a medium, a computer device and a program product are applied to metadata nodes of a distributed storage system, a plurality of data nodes of the distributed storage system are used for storing erasure code blocks, and the erasure code blocks comprise at least one erasure code corresponding to at least one piece of original data. The metadata node is used for storing length information of original data written into the distributed storage system; the method comprises the steps that at least one piece of length information sent by a client of the butt-joint distributed storage system is obtained, the at least one piece of length information is used for representing the length of at least one piece of original data corresponding to at least one erasure code in a target erasure code block, and the target erasure code block is the erasure code block with the kth length in the multiple erasure code blocks, k is greater than or equal to the minimum number of erasure codes required for reconstructing the original data; and determining target length information from the at least one piece of length information, and synchronizing the length information of the original data written into the distributed storage system based on the target length information.

The invention relates to the technical field of big data, and particularly provides a data conversion method and device, computing equipment and a product, and the data conversion method comprises the steps: aggregating data of a data layer into a data stream; segmenting the data stream into at least one data segment; segmenting each data segment into at least one data block, wherein the lengths of the plurality of data blocks are different; according to the serial number corresponding to the at least one data block, one or more target data blocks with continuous serial numbers are divided into the same erasure code fragment, K erasure code fragments are obtained, and the serial numbers of the data blocks in the same erasure code fragment are continuous. In the embodiment of the invention, when the storage mode of the copy format is converted into the storage mode of the erasure code, the data blocks in the erasure code fragment adopt a column storage scheme, so that the data blocks in the same erasure code fragment are continuous, the conversion mode is more transparent, and the problem of format conversion is solved.

The application discloses a data reconstruction method and device, electronic equipment and storage medium, and relates to the technical field of data storage and recovery. In the case that a to-be-reconstructed disk exists in a distributed storage system, the redundancy of the to-be-reconstructed disk is determined based on state data of the to-be-reconstructed disk; a target redundancy level corresponding to the redundancy of the to-be-reconstructed disk is determined from a plurality of preset redundancy levels; different redundancy levels correspond to different erasure code types; the erasure code types corresponding to different redundancy levels represent different combinations of data recovery rates and storage resource occupation ratios; the erasure code type of the to-be-reconstructed disk is switched to a target erasure code type corresponding to the target redundancy level, and the target erasure code type is used to transmit a fault data block in the to-be-reconstructed disk to a specified disk for data reconstruction, thereby solving the problem that data recovery performance, resource consumption and network bandwidth resource occupation cannot be dynamically balanced in the data reconstruction process in the related art.

The application discloses a data processing method and an electronic device. The method comprises the following steps: in response to a write / read data request, determining a storage address corresponding to to-be-written / read data; determining a bitmap mark corresponding to the storage address, the bitmap mark being used for representing whether data calculated by an erasure code exists in the storage address; and writing the to-be-written data in the storage address or reading the to-be-read data from the storage address according to the bitmap mark. In this way, when data is saved by using a large-scale strip erasure code, each time data is written in or read from the storage address, only the bitmap mark corresponding to the storage address needs to be confirmed, and the data is written in or read from the storage address based on the confirmation result of the bitmap mark, that is, the data can be directly written in the strip by using a local updating mode without reconstructing the data, so that time spent on reconstructing the data in the strip is saved, and user experience is improved.