18 results about "Erasure code" patented technology

The application discloses an ordered repairing and deleting method of a distributed storage system, and comprises the following steps: constructing a repairing partition key for repairing information, and storing the repairing information in the distributed storage in order based on the repairing partition key; constructing a deleting partition key for deleting information, and storing the deleting information in the distributed storage in order based on the deleting partition key; distributing each erasure code group to a corresponding repairing work process and deleting work process, so that the repairing work process is configured with the repairing partition key, the corresponding repairing information is pulled from the distributed storage, a data repairing operation is performed, the deleting work process is configured with the deleting partition key, the corresponding deleting information is pulled from the distributed storage, and a data deleting operation is performed. The application effectively avoids problems such as excessive system memory pressure, data reliability decline and unstable system operation caused by unordered organization and unordered scheduling, and improves the stability and scheduling efficiency of the repairing and deleting tasks under the Quorum intelligent read-write architecture.

The application discloses a kind of unmanned plane identity anonymization and safe data outsourcing method and system, method is based on integrated supervision platform and unmanned plane, including system initialization stage, integrated supervision platform generates main private key and public parameter;User anonymous and private key generation stage, unmanned plane registers anonymous identity and obtains user private key;Data preprocessing and outsourcing stage, unmanned plane divides data into block and sub-block and encrypts, generates coding vector using erasure code coding, generates verification label for each coding data block, constructs random distribution table according to the number of cloud server, distributes coding vector and label to multiple cloud server storage.In this way, the unmanned plane identity anonymization and safe data outsourcing method described reduces storage redundancy, ensures that data can be reconstructed when multiple servers fail, and guarantees data availability and robust recovery.

This invention relates to the field of distributed storage technology and provides a data collaboration method and system for distributed transaction terminals. The method involves each transaction terminal performing erasure coding on its local ledger data to generate a set of data fragments consisting of original data fragments and verification data fragments. These data fragments are then distributed and stored across different transaction terminals according to distributed storage rules, and the terminals exchange online status and fragment integrity information. When a request to access the target ledger is received, the terminal storing its fragments is located based on the ledger identifier, and the fragments are retrieved in parallel to reconstruct the target ledger. When a terminal is detected to be offline or a fragment is corrupted, online terminals collaborate to reconstruct the original data using surviving fragments, re-encode new fragments, and distribute them to healthy terminals. This invention, through erasure coding and distributed collaborative storage, reduces storage overhead while improving system fault tolerance and data availability, achieving efficient and reliable distributed ledger data collaborative management.

The application discloses a data reconstruction method and device, electronic equipment and storage medium, and relates to the technical field of data storage and recovery. In the case that a to-be-reconstructed disk exists in a distributed storage system, the redundancy of the to-be-reconstructed disk is determined based on state data of the to-be-reconstructed disk; a target redundancy level corresponding to the redundancy of the to-be-reconstructed disk is determined from a plurality of preset redundancy levels; different redundancy levels correspond to different erasure code types; the erasure code types corresponding to different redundancy levels represent different combinations of data recovery rates and storage resource occupation ratios; the erasure code type of the to-be-reconstructed disk is switched to a target erasure code type corresponding to the target redundancy level, and the target erasure code type is used to transmit a fault data block in the to-be-reconstructed disk to a specified disk for data reconstruction, thereby solving the problem that data recovery performance, resource consumption and network bandwidth resource occupation cannot be dynamically balanced in the data reconstruction process in the related art.

This invention discloses a method, apparatus, device, and medium for handling erasure coding data read failures, relating to the field of distributed storage systems. The method includes: if a target number of available data blocks exist, and any object storage device among the target number of object storage devices reports an error causing data read failure, then the primary object storage device sends an error message to the client, marks the status of the erroneous object storage device, and synchronizes the status marking information to the target object list so that the client can determine whether to perform data erasure coding; if the erroneous object storage device recovers or a new object storage device is added, then the target number of available data blocks are read to obtain the original data, the target data for which the read failed is determined based on the original data, and the target data is written to the target object storage device; the status of the target object storage device is modified by the primary object storage device so that the client can re-initiate the data read request. This improves the overall stability of the storage system.

The specification provides a data processing method, an electronic device, a storage medium and a program product. A master storage node of to-be-written data in a distributed storage cluster determines backup storage nodes based on a master-slave backup party matrix, sends a copy of the to-be-written data to the backup storage nodes, and after determining that each backup storage node has received the copy, sends a write completion response to a client, so that the performance of a data write operation on a critical path in the distributed storage cluster is comparable to that of a replica mechanism, achieving high-performance data storage. At the same time, under the condition that the backup storage node determines that it has received the copy of the to-be-written data sent by the master storage node corresponding to each check row number, the backup storage node calculates and stores a corresponding check block based on each received copy, so as to convert k data copies into a check block, realizing the calculation and storage of the check block in the erasure code mechanism, and the storage overhead is comparable to that of the erasure code mechanism, which is much lower than that of the replica mechanism. In summary, a redundant mechanism with high performance and low storage space overhead is realized.

This invention relates to an erasure coding method and system based on two-layer optimization. The method includes: obtaining storage parameters of the data to be stored; constructing an initial generator matrix structure and a reference table for characterizing the bit-level characteristics of elements; constructing a two-layer optimization model and using the two-layer optimization model to collaboratively optimize the initial generator matrix structure; the two-layer optimization model includes: an upper-layer optimization model configured to adjust the structural distribution of the generator matrix; and a lower-layer optimization model configured to schedule the bit-level operation paths after the generator matrix expansion based on the reference table and calculate the actual bit-level computational overhead; feeding back the bit-level computational overhead of the lower-layer optimization model to the upper-layer optimization model; and encoding the data to be stored based on the comprehensive evaluation result of the algebraic characteristics and the bit-level computational overhead. This invention achieves the collaborative unification of the generator matrix structure and XOR scheduling optimization through the two-layer optimization model, improving the coding rationality, practicality, and throughput.

Embodiments of the present specification provide an information processing method and device, wherein the information processing method comprises: obtaining erasure code stripe information corresponding to an erasure code file, and writing the erasure code stripe information into a stripe information record table; in the case of detecting an operation request associated with the erasure code file, reading the erasure code stripe information in the stripe information record table; determining data address information of a target data node corresponding to the erasure code file according to the erasure code stripe information; creating operation description information according to the operation request and the data address information, and sending to the target data node.

This invention discloses an asynchronous network consensus method and system based on a reputation model, belonging to the field of consensus algorithm technology. It aims to solve the problems of low efficiency, high latency, and frequent leader election failures in asynchronous network environments. The method is divided into a broadcast phase and a consensus phase: In the broadcast phase, broadcast nodes encode erasure codes into message fragments and distribute them; receiving nodes verify these fragments and store them in the corresponding priority queues. In the consensus phase, each node generates a binary value vector based on its priority queue and distributes it across the network. A leader is elected based on the reputation value calculated by each node. An asynchronous binary consensus protocol is used to reach a consensus on the leader's validity, determining a multi-value Byzantine consensus result. Finally, the original message is requested and restored from the node indicated by the result. Erasure coding reduces communication overhead, and the reputation model reduces the number of consensus failures, improving system throughput, reducing latency, and enhancing stability and efficiency in asynchronous network environments.

This invention provides a data storage method and system for distributed object storage. The method includes: encoding the original data of the target object using erasure coding to generate a group of shards to be written, containing k data shards and m check shards; writing each shard to independent physical extended shards on different storage nodes, wherein the extended shards are stored hierarchically using record blocks and check information blocks; generating a write success response when the number of successfully written shards reaches a preset threshold (≥k and ≤k+m); and performing a sealing operation on the extended shards after writing, using the minimum logical commit length of each successfully written shard as the trust boundary, truncating excess data and locking write permissions. This application solves the dilemma of existing systems where some shards cannot be securely confirmed after writing, eliminates node blocking, balances write efficiency and data consistency, and improves the availability, integrity, and efficiency of the storage system.

This invention provides an SSD and storage array based on erasure coding mechanism. The SSD includes a NAND flash memory array and a controller chip. The controller chip includes SRAM, a CPU, and an erasure coding hardware accelerator. The erasure coding hardware accelerator is used to generate parity blocks and repair data, and includes a parity data generation module and a data repair module. The SSD provided by this invention integrates an erasure coding hardware accelerator into the controller chip, thereby applying erasure coding technology to the SSD and improving its storage efficiency and reliability.

This application discloses a data processing method, apparatus, and electronic device based on blockchain. The method includes: upon receiving a data backup instruction sent by a client, obtaining version data of a database from a trusted execution environment through a proxy module, wherein the version data includes at least one of the following: a first timestamp associated with data stored in the database, first database operation information within a first time range, and first verification information for data verification; verifying the version data based on verification data stored in the blockchain; if the version data passes verification, performing erasure coding operations on the first timestamp and the first database operation information to generate multiple first data block fragments, and sending the first data block fragments to their respective nodes; and, if the nodes achieve Byzantine consensus, storing and backing up the first data block fragments according to a first order. This method improves the efficiency of data backup.

This invention discloses a data storage method, apparatus, device, and medium for an expanded erasure coding storage system. The method includes: splitting a file to be stored into a first number of candidate fragment files based on a preset erasure coding structure; wherein each candidate fragment file includes a second number of data fragment files and a third number of verification fragment files; determining a fourth number of target fragment files from the candidate fragment files; wherein the fourth number is determined based on the third number, and the fourth number is greater than or equal to 1; storing the target fragment files in the expanded storage node, and storing the remaining fragment files in the original storage node. By adopting the technical solution of this application, after expanding the erasure coding storage system with new nodes, data balancing among the nodes can be quickly performed without compromising data integrity and erasure redundancy, without consuming additional system resources, and without affecting real-time storage services.

The embodiment of the specification provides a transaction data pre-transmission method, a block chain system and a consensus node, including: creating a local sub-chain based on transactions stored in a local transaction pool; wherein the local sub-chain includes a plurality of sub-blocks for generating a proposal block, which are created based on transactions stored in the local transaction pool; based on an erasure code algorithm, each sub-block in the local sub-chain is respectively divided into a specified number of data shards, and the specified number of data shards are respectively unicast to each other consensus node, so that each other consensus node performs legality verification on the received data shard, and in response to the legality verification passing, continues to broadcast the data shard to each other consensus node, and each other consensus node performs data recovery on the received data shard to obtain each sub-block.

This invention relates to the field of data communication and provides a distributed communication method based on a dual-plane architecture and adaptive erasure coding. The method includes constructing a global scheduler, creating logical mailboxes within the global scheduler, and establishing exchange links; processing received data by embedding data length and padding with zeros to obtain encoded data; performing target sandbox delivery on the encoded data according to a data block threshold to obtain decoded data; enqueuing the data and promoting it from the lower network plane to the upper application plane; calculating sequence number differences; accumulating these sequence number differences to calculate packet loss assessment parameters; updating the data block threshold based on the packet loss assessment parameters to obtain an iterative data block threshold; writing the iterative data block threshold to a pending configuration buffer; obtaining indicator parameters; retrieving the iterative data block threshold from the pending configuration buffer; and updating the parameters using the iterative data block threshold. This invention can complete data communication.