324 results about "Erasure code" patented technology

The invention relates to the technical field of judicial data security, and discloses a judicial system confidential data security circulation method based on a block chain technology. Collecting multi-source judicial data, identifying sensitive information through a large language model, and performing differential privacy desensitization processing; constructing an SM4 encryption and TLS 1.3 end-to-end secure channel, generating a data hash fingerprint, and writing the data hash fingerprint into an alliance chain evidence based on a PBFT consensus mechanism; designing a multi-modal classification engine to perform feature extraction and intelligent classification; establishing a hybrid authority model to integrate an XACML strategy and a Kafka queue, and implementing dynamic authority control in combination with an RBAC / ABAC mechanism; a hierarchical encryption storage architecture is constructed, and homomorphic encryption retrieval and erasure code distributed storage are adopted; constructing a judicial knowledge graph based on a BERT model; and deploying a block chain auditing system, and combining LSTM anomaly detection and a DREAD risk assessment model to form a closed-loop risk control system. According to the method, the problems of security risk and privacy disclosure in judicial data cross-department circulation are solved.

The invention relates to the technical field of big data, in particular to a cloud computing-based tumor early screening data sharing platform construction method and system. The sharing platform construction method comprises the following steps: carrying out association modeling on multi-dimensional early screening data by adopting a knowledge graph technology; encrypting the sensitive field of the metadata index database, and synchronously generating an irreversible anonymous identifier to replace the original identifier; partitioning and arranging the metadata index database in a distributed storage system corresponding to a cloud platform according to tumor types, and performing cross-region redundant backup by adopting an erasure code technology; and constructing an access control smart contract based on a block chain technology, verifying the qualification of a requester through a digital identity authentication module, and after verifying the validity of a digital certificate of the requester, dynamically decrypting and transmitting a data copy conforming to an authority level. Through organic fusion of the knowledge graph, the encryption technology, distributed storage and the block chain, a safe, efficient and reliable tumor early screening data sharing platform is constructed.

The invention provides an erasure code compatible read-write method and system based on a bidirectional data access agency, and relates to the technical field of distributed storage, and the method comprises the steps: employing multi-agent reinforcement learning to construct a heterogeneous resource portrait and optimize task allocation, combining a space-time graph convolutional network and a federated learning framework to construct a data format conversion model, and constructing an erasure code compatible read-write model; data features are analyzed through an encoder based on an attention mechanism, a fragmentation strategy is generated, self-adaptive encoding is achieved through a capsule neural network, data integrity verification and repairing are conducted through a double-layer space attention network and a graph attention network, the data access efficiency of a heterogeneous storage system is improved, data format compatibility is enhanced, and the data access efficiency is improved. And the data integrity is ensured.

The invention discloses a data management system and method based on distributed cloud storage, and relates to the technical field of distributed cloud storage, and the system comprises an access log collection module, a node state monitoring module, a data thermal evaluation module, a redundancy strategy decision module and a storage execution scheduling module. A fragmentation access behavior is recorded through an access log acquisition module, a node state monitoring module acquires a node resource state in real time, and a data thermal evaluation module constructs a three-dimensional thermal model based on an access frequency, a network hop count and a storage cost to calculate a thermal value and supports dynamic weight adjustment; the redundancy strategy decision module divides the fragments into high frequency, medium frequency and low frequency according to the thermodynamic value, and single copy + local cache, erasure code storage and dynamic adjustment strategies are adopted respectively; the storage execution scheduling module executes fragmentation operation and optimizes reading node selection, and data access efficiency is improved through load awareness and a bandwidth-delay joint model; according to the invention, dynamic balance between data access efficiency and storage cost is realized.

The present application discloses a data management method, apparatus and system for a distributed storage system, and an electronic device. The method comprises: traversing all storage paths in a target distributed storage cluster, and acquiring file storage information of the storage paths during traversal; on the basis of the file storage information, calculating a redundancy policy score of each storage path, and on the basis of the redundancy policy score and a first redundancy policy score threshold, determining a storage policy adapted to the storage path; when it is determined that the storage policy adapted to the storage path is an erasure code storage policy, determining a data recovery mode of the storage path on the basis of the architecture information of the storage path; and when it is determined that the storage policy adapted to the storage path is a multi-copy storage policy, determining the number of copies of the storage path on the basis of a service importance degree of the storage path and the redundancy policy score.

The invention relates to the technical field of information security, in particular to a computing method and system for information security. The method comprises the following steps of: partitioning and fragmenting input data, and distributing the data to a plurality of storage nodes through randomization, distributed storage and Hash mapping to ensure the safety of data storage and the distribution flexibility; establishing an index table for all the fragments; regularly triggering a data confusion mechanism, and adjusting the storage position of data fragments through a pseudo-random function; the integrity of the data is detected by generating a check code and an erasure code, it is ensured that the data is not tampered or damaged in the transmission or storage process, and the data is recovered through a redundant copy when the data is lost; the identity verification and the data integrity when the user accesses the data are ensured through authority authentication; and monitoring storage nodes and data access behaviors in real time, detecting potential threats and taking dynamic adjustment measures. According to the invention, through a multi-level security policy, the security and fault-tolerant capability of data processing are effectively improved.

The invention discloses a data dynamic migration and storage method, device and equipment and a medium, and relates to the technical field of computers. The method comprises the steps that environment parameters of a current mechanical hard disk in a distributed cluster are collected; wherein the environment parameters comprise a vibration parameter, a temperature parameter and a noise parameter; quantifying the input and output delay performance of the current mechanical hard disk by using the environmental parameters of the current mechanical hard disk; if the input and output delay performance of the current mechanical hard disk is greater than a preset threshold value, searching a target mechanical hard disk combination in the distributed cluster; and migrating and storing the target data block of the current mechanical hard disk into the target mechanical hard disk combination based on an erasure code and a multi-copy mechanism. Through the scheme, the reliability of storing the data in the mechanical hard disk in the distributed cluster is remarkably improved.

The invention relates to the field of distributed storage, in particular to an erasure code algorithm processing method for cold and hot fragmented data in a distributed storage system in a local area network or a wide area network, and solves the problems caused by using an erasure code algorithm to process cold and hot data in the prior art. The method comprises the steps of data block partitioning, a parameter strategy of data cold and hot judgment, a parameter strategy of node cold and hot judgment, data writing, an erasure code algorithm processing mechanism of different types of data, processing after erasure code algorithm processing, data cold and hot conversion processing and the like. Meanwhile, normal read-write performance is kept not affected by erasure code processing in the EC coding processing process, and the rationality of cold and hot data division in the distributed storage system is improved; layered storage and migration are performed according to cold and hot data, the read-write performance of distributed storage is considered, and the capacity utilization rate is increased; and the reliability, fault tolerance and performance of the storage system are improved.

The invention provides a data management method and system based on optomagnetic fusion storage, and belongs to the field of data management.The method comprises the steps that target data are obtained, data feature information is collected, and data popularity analysis is carried out; a basic storage scheme is determined according to data popularity, and intelligent decision-making of optical storage, magnetic storage or fusion storage is realized through data criticality analysis and popularity critical coefficient evaluation; predicting a storage loss parameter according to the data popularity and the storage scheme, and optimizing an error correction code or erasure code parameter in combination with the data criticality; and finally, implementing data management according to the determined storage scheme and optimization parameters. The technical problems that a traditional storage system cannot intelligently distribute storage media according to data characteristics, and access efficiency and storage safety are difficult to balance are solved, self-adaptive selection of a storage scheme and optimization of storage quality are achieved, and efficiency and reliability of data management are improved.

The embodiment of the invention relates to the technical field of data storage, in particular to a data storage method and device, equipment and a storage medium, and aims at improving the data read-write efficiency of distributed storage. The method comprises the following steps: receiving a data writing request sent by a client; performing erasure code calculation on business data corresponding to the data writing request to obtain storage data; through a preset network transmission protocol, the storage data is written into a hard disk space pre-applied from at least one data storage node, and the minimum storage unit of the hard disk space is a data block; and recording a mapping relationship between the business object range corresponding to the storage data and the internal object range in the data storage node in the metadata storage node.

The invention discloses a satellite Internet of Things distributed node parallel transmission acceleration method, which relates to the technical field of satellite Internet of Things, and comprises the following steps: dynamically fragmenting, splitting into N fragments according to a network load rate and a node state, adding a serial number, a check code and redundant information, and marking a head as {INJ}; the intelligent routing is used for distributing fragments to distributed nodes through a scoring function based on satellite ephemeris, node loads and link quality; dynamic path switching: monitoring link and node states, and switching to a better path when the performance does not reach the standard; fault-tolerant recombination is carried out, recombination is carried out according to sequence numbers, if missing fragments are few, erasure codes are used for recovery, and if missing fragments are more, local retransmission is carried out. Through parallel transmission, dynamic switching and fault-tolerant recombination, the transmission rate, throughput and robustness are improved, node loads are balanced, and the performance of the satellite Internet of Things is optimized.

The invention relates to the technical field of intelligent capacity expansion, in particular to an intelligent capacity expansion method and system for an enterprise-level solid state disk, and the method comprises the following steps: an I / O controller detects a newly accessed enterprise-level solid state disk and sets a total write-in amount threshold value; according to the method, the total write-in quantity threshold is set after the enterprise-level solid state disk is accessed, and the dynamic delay portrait is constructed in combination with the real-time delay data, so that the performance state of the target hard disk is precisely mastered in the capacity expansion process, and the logic data block and the check code are synchronously embedded in the data write-in stage; and an erasure code check block is synchronously generated in combination with a write-in voucher and written into a far-end address, so that a main and auxiliary path parallel write-in mechanism is realized. And comparing data consistency by reading main and auxiliary addresses, constructing an inconsistent item set, obtaining a corresponding physical region error log by relying on an NVMe interface, extracting an abnormal sector, and mapping to form an unstable physical region coordinate, so that the system has the capability of actively avoiding an unstable region.

The invention provides a fast recovery method for hadoop cluster data, which relates to the technical field of erasure codes, and comprises the following steps of: calculating an importance score of a data block to be recovered, determining a target copy number based on a nonlinear copy number adjustment function, calculating a load weight in combination with network topology information, and selecting candidate nodes; and predicting a node fault probability by using a long short-term memory network, calculating a migration priority based on the importance score and the fault probability, and selecting an optimal target node to execute data recovery. According to the invention, the intelligence and differentiation of the data recovery process are realized, and the data recovery efficiency and reliability are improved.

The invention discloses a data processing method and device, electronic equipment, a storage medium and a product. The method comprises the steps of obtaining a disk dynamic mapping request; in response to the disk dynamic mapping request, a data repair migration task is generated, and the data repair migration task is used for triggering disk dynamic mapping and data repair migration processes of the erasure code redundant group; and executing the data restoration migration task.

One or more data segments to be stored in a storage system are formed. A first data segment of the one or more data segments is written to regions of flash memory of a first storage device of the storage system using an erasure code that divides the first data segment into data shards. Writing the first data segment includes calculating at least one intra-device recovery data shard corresponding to the data shards of the first data segment to be stored in the first storage device that protects the data shards. The data shards of the first data segment and the at least one intra-device recovery data shard are organized and stored into the flash memory of the first storage device based on fault boundaries in flash architectures for writing to flash cells of the flash memory.

The invention discloses an enterprise financial consultation platform based on financial data analysis, and relates to the technical field of enterprise financial data analysis, and the platform comprises a storage medium health monitoring module which collects physical health indexes of storage equipment in real time, constructs a dynamic portrait of the health degree of a storage medium, marks a high-risk storage area, and stores the physical health indexes; generating a dynamically updated storage medium health thermodynamic diagram; according to the method, the physical health indexes of the storage equipment are collected in real time, and the dynamic portrait of the health degree of the storage medium is constructed, so that the aging risk of the storage medium is accurately identified, invisible hardware aging is converted into quantifiable risk information, the risk identification capability is improved, dual-mode verification of a bit layer and a semantic layer is adopted, and the risk identification efficiency is improved. The functions of identifying high-risk silent data errors and capturing local damage of the storage medium are realized; and through a hierarchical repair strategy, the document repair effect is improved by utilizing an erasure code decoder and a FinDoc-RepairGAN processor.

The invention relates to the technical field of databases, in particular to an incremental data writing method and system based on incremental erasure codes. The method comprises the following steps: acquiring a write-in position (offset value) and a data length of incremental data, calculating an incremental erasure code according to incremental change data of each data block starting from the write-in position and having the length of the data length, and updating verification data at a corresponding position of a verification block; according to the method, calculation and updating of the incremental erasure codes in the incremental data writing process are achieved, and therefore in a distributed storage scene, it is not needed to obtain full data of data blocks for erasure code calculation, and occupation of data transmission and calculation resources is reduced.

The invention relates to the field of distributed storage, and discloses a distributed memory system based on erasure codes and a dynamic repair mechanism. Comprising a data node cluster, a metadata management cluster, a bandwidth token management module, a dynamic repair composer and a repair agent node. And the data node cluster divides the object into strip generation data fragments and verification fragments and dispersedly stores the data fragments and the verification fragments according to a placement rule. And the metadata management cluster maintains object stripe mapping, a fragment position set, erasure code protection configuration and a version epoch, and records a repair intention log. And the bandwidth token management module grants repair execution permission according to the fault domain budget. The dynamic repair composer generates availability repair and durability repair tasks, the temporary check fragments are written in when the number of the available check fragments is lower than a threshold value in the availability stage, and the configuration fragments are recovered and the temporary fragments are recovered in the durability stage. And the consistency of version epochs is verified before the agent node is repaired and written back, and if not, recalculation is carried out.

The invention discloses an efficient double-link block chain consensus method and device, equipment and a storage medium, and belongs to the field of information security. The method comprises an initialization stage, an input stage, a coding stage, a broadcasting stage and a consensus stage. According to the invention, the parallel design of a coding broadcast link and a consensus confirmation link is adopted, so that the occurrence frequency of transaction congestion is reduced; the integrity required by the asynchronous Byzantine consensus process is realized through three-step broadcasting, the updating problem of the post-quantum component does not need to be considered, and the security is higher; an asynchronous network and a Byzantine opponent model are adopted, so that malicious node attacks which do not exceed the total number of nodes can be tolerated on the premise of not setting a network time delay upper limit; in the encoding stage, the generation of the redundancy share through the RS erasure code can be realized by adopting preprocessing, the data redundancy expansion multiple in the broadcasting stage is far smaller than that of the same type of protocol, and the original transaction can be recovered by collecting part of effective shares in the consensus stage, so that the system time delay is effectively reduced, and the system throughput is improved.

The invention provides a distributed cluster security backup method and system oriented to a cloud-side environment, and relates to the technical field of distributed storage systems, and the method comprises the steps: obtaining cloud-side cluster data, and adaptively selecting a backup mode based on an erasure code and a multi-copy hybrid redundancy strategy; establishing a workload-aware incremental replication mechanism, and optimizing fault recovery by using idle resources; the problem of lock competition is solved by adopting a multi-copy consistency guarantee mechanism based on two-stage submission, and lock-free reading is realized based on MVCC. According to the invention, the data backup flexibility, the fault recovery efficiency and the system consistency guarantee capability are improved.

The invention provides an erasure code data updating method and system of a distributed hybrid storage system. The erasure code data updating method comprises the steps of constructing a generation matrix; judging whether the effective data of the performance layer form a complete erasure code strip or not; if yes, reading a whole-strip data block of a performance layer, calculating and generating a new erasure code check block, writing the data block and the check block into a capacity layer in a one-time coverage manner, and repairing by adopting a whole-strip updating consistency repairing strategy after a fault occurs; and if not, reading the data to be updated of the performance layer and the old data blocks and all the check blocks at the corresponding positions of the capacity layer, writing the calculated new erasure code check blocks and new data blocks into the capacity layer according to a certain sequence, and repairing by adopting a non-full stripe updating consistency repair strategy after the fault occurs. According to the method, extra I / O operation required for maintaining new and old version data on the HDD is avoided, the utilization rate of storage resources is improved, and it is ensured that consistency and integrity can be quickly and efficiently recovered when write-in fails and possible data is lost due to superposition.

Methods and apparatus are provided. In an example aspect, a method of managing data storage in a communication system is provided. The communication system includes a plurality of clusters of network nodes, and the network nodes in each cluster of network node include one or more data storage nodes. The method comprises receiving data to be stored from a data source node, generating, from the data to be stored, a first subset of erasure coded chunks, causing the first subset of erasure coded chunks to be stored in one or more of the data storage nodes in at least one first cluster of the plurality of clusters, and sending, to a second cluster of the plurality of clusters, the data to be stored.

The invention provides a transmission performance optimization method and device, equipment, a storage medium and a program product, and relates to the technical field of data transmission. The method comprises the following steps: generating a first file corresponding to each path according to an original data file and erasure code parameters; transmitting the corresponding first file through each path, and acquiring a transmission performance parameter of each path; determining the weight of each path according to the transmission performance parameter of each path; and adjusting the number of data blocks transmitted by each path according to the weight of each path. According to the scheme, the real transmission performance data is obtained in the multi-path transmission process through the first file, the transmission performance of the paths is evaluated from the transmission result level, the distribution strategy of the data blocks of all the paths is dynamically adjusted according to the fed-back real transmission performance parameters, the data transmission efficiency and reliability are effectively improved, and transmission delay is reduced.

Disclosed in the present application are an erasure code data updating method and apparatus. The method comprises: in response to a write operation for a target shard of a target object in a main object storage device, acquiring a state identifier of the target shard; if the state identifier is a first state identifier, storing, in a reserved object in a parity object storage device, initial data in the target shard and initial parity data for the target object, using updated data to overwrite the initial data, and modifying the state identifier to a second state identifier; if the state identifier is the second state identifier, using the updated data to overwrite current data in the target shard and updated data that is stored in the reserved object; and when a distributed object storage system is idle, determining target parity data on the basis of the initial parity data and the initial data and the updated data that are currently stored in the reserved object, and modifying the state identifier to the first state identifier.

The invention discloses a data reconstruction method and device, electronic equipment and a storage medium, and relates to the technical field of data storage and recovery, and the data reconstruction method comprises the steps that under the condition that a to-be-reconstructed disk exists in a distributed storage system, the redundancy of the to-be-reconstructed disk is determined based on state data of the to-be-reconstructed disk; determining a target redundancy level corresponding to the redundancy of the to-be-reconstructed disk from a plurality of preset redundancy levels; different redundancy levels correspond to different erasure code types; the erasure code types corresponding to different redundancy levels represent different combinations of data recovery rates and storage resource occupation proportions; and switching the erasure code type of the to-be-reconstructed disk to a target erasure code type corresponding to the target redundancy level, and according to the target erasure code type, transmitting the fault data block in the to-be-reconstructed disk to a specified disk for data reconstruction. The problem that data recovery performance, resource consumption and network bandwidth resource occupation cannot be dynamically balanced in the data reconstruction process in the prior art is solved.

The invention discloses a distributed data storage and access control method based on a block chain, and relates to the technical field of data security. The method comprises the following steps: fragmenting user data, encrypting the fragmented user data by using an encryption algorithm, and storing the encrypted user data in a plurality of distributed nodes; taking the hash value and the storage position of the data fragment as metadata to be recorded in a block chain, and performing integrity verification through a smart contract; when a user accesses, user attributes are verified based on an attribute-based encryption technology, and an access control strategy is dynamically executed through an intelligent contract; and an erasure code technology is adopted to carry out redundancy storage so as to realize data recovery and fault tolerance. The method effectively solves the problems of single-point failure, privacy disclosure and inflexible access control in traditional centralized storage, has the beneficial effects of improving data security, guaranteeing integrity, realizing fine-grained dynamic access control and enhancing system reliability, and is suitable for data security management in cloud computing, Internet of Things and big data environments.

The embodiment of the invention relates to a data processing method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the coding operation of an erasure code on first data under the condition of receiving a write-in request for the first data, and obtaining a plurality of logic units and a plurality of fragments; according to the first identifier of the logic unit and the second identifier of each fragment, writing the plurality of fragments into a storage node; and under the condition that a reading request is received, obtaining a target first identifier and a target second identifier of the second data, reading the plurality of target fragments of the second data from the plurality of storage nodes, and carrying out merging processing to obtain the second data. Therefore, the data can be divided into the fragments with the unique identifiers for storage in the writing stage, and the target fragments can be quickly positioned according to the logic addresses and the recovered data can be merged in the reading stage, so that the metadata query overhead is reduced, the reading and writing efficiency is improved, and the storage and network transmission cost is reduced; and the overall performance of the distributed storage system in a high-concurrency scene is improved.

The invention provides a temporary check block repairing method and system based on erasure codes in distributed block storage, and the method comprises the steps that a distributed block storage system detects that a check block fails to be written in, and a metadata service creates a temporary check block according to an erasure code mode; the temporary check block records an operation log and executes data writing, and replays IO operation according to a data version number; repairing the inconsistency of the data blocks by using the operation log of the temporary check block; and quickly recovering the check block data through the data bitmap of the temporary check block. According to the rapid recovery processing method for the temporary check block, the check block data can be directly recovered without a coding mode, the internal IO is reduced, and the recovery speed of the erasure code check block is increased.

A fragment and shuffle erasure coding operation is disclosed. In a transformative operation, encrypted fragments are generated from an input file. The transformative operation includes one or more iterations of chunking the file and mixing the chunks to generate a mixed sequence. The mixed sequence is sliced into encrypted fragments. The encrypted fragments are shuffled in a storage system such that physical locations of the encrypted fragments change over time.

A method for adjustable error correction in a storage cluster is provided. The method includes determining health of a non-volatile memory of a non-volatile solid-state storage unit of each of a plurality of storage nodes in a storage cluster on a basis of per flash package, per flash die, per flash plane, per flash block, or per flash page. The determining is performed by the storage cluster. The plurality of storage nodes is housed within a chassis that couples the storage nodes as the storage cluster. The method includes adjusting erasure coding across the plurality of storage nodes based on the health of the non-volatile memory and distributing user data throughout the plurality of storage nodes through the erasure coding. The user data is accessible via the erasure coding from a remainder of the plurality of storage nodes if any of the plurality of storage nodes are unreachable.