265 results about "Erasure code" patented technology

The invention relates to the technical field of judicial data security, and discloses a judicial system confidential data security circulation method based on a block chain technology. Collecting multi-source judicial data, identifying sensitive information through a large language model, and performing differential privacy desensitization processing; constructing an SM4 encryption and TLS 1.3 end-to-end secure channel, generating a data hash fingerprint, and writing the data hash fingerprint into an alliance chain evidence based on a PBFT consensus mechanism; designing a multi-modal classification engine to perform feature extraction and intelligent classification; establishing a hybrid authority model to integrate an XACML strategy and a Kafka queue, and implementing dynamic authority control in combination with an RBAC / ABAC mechanism; a hierarchical encryption storage architecture is constructed, and homomorphic encryption retrieval and erasure code distributed storage are adopted; constructing a judicial knowledge graph based on a BERT model; and deploying a block chain auditing system, and combining LSTM anomaly detection and a DREAD risk assessment model to form a closed-loop risk control system. According to the method, the problems of security risk and privacy disclosure in judicial data cross-department circulation are solved.

The invention discloses a data management system and method based on distributed cloud storage, and relates to the technical field of distributed cloud storage, and the system comprises an access log collection module, a node state monitoring module, a data thermal evaluation module, a redundancy strategy decision module and a storage execution scheduling module. A fragmentation access behavior is recorded through an access log acquisition module, a node state monitoring module acquires a node resource state in real time, and a data thermal evaluation module constructs a three-dimensional thermal model based on an access frequency, a network hop count and a storage cost to calculate a thermal value and supports dynamic weight adjustment; the redundancy strategy decision module divides the fragments into high frequency, medium frequency and low frequency according to the thermodynamic value, and single copy + local cache, erasure code storage and dynamic adjustment strategies are adopted respectively; the storage execution scheduling module executes fragmentation operation and optimizes reading node selection, and data access efficiency is improved through load awareness and a bandwidth-delay joint model; according to the invention, dynamic balance between data access efficiency and storage cost is realized.

The invention discloses a data dynamic migration and storage method, device and equipment and a medium, and relates to the technical field of computers. The method comprises the steps that environment parameters of a current mechanical hard disk in a distributed cluster are collected; wherein the environment parameters comprise a vibration parameter, a temperature parameter and a noise parameter; quantifying the input and output delay performance of the current mechanical hard disk by using the environmental parameters of the current mechanical hard disk; if the input and output delay performance of the current mechanical hard disk is greater than a preset threshold value, searching a target mechanical hard disk combination in the distributed cluster; and migrating and storing the target data block of the current mechanical hard disk into the target mechanical hard disk combination based on an erasure code and a multi-copy mechanism. Through the scheme, the reliability of storing the data in the mechanical hard disk in the distributed cluster is remarkably improved.

The invention provides a data management method and system based on optomagnetic fusion storage, and belongs to the field of data management.The method comprises the steps that target data are obtained, data feature information is collected, and data popularity analysis is carried out; a basic storage scheme is determined according to data popularity, and intelligent decision-making of optical storage, magnetic storage or fusion storage is realized through data criticality analysis and popularity critical coefficient evaluation; predicting a storage loss parameter according to the data popularity and the storage scheme, and optimizing an error correction code or erasure code parameter in combination with the data criticality; and finally, implementing data management according to the determined storage scheme and optimization parameters. The technical problems that a traditional storage system cannot intelligently distribute storage media according to data characteristics, and access efficiency and storage safety are difficult to balance are solved, self-adaptive selection of a storage scheme and optimization of storage quality are achieved, and efficiency and reliability of data management are improved.

The invention discloses a satellite Internet of Things distributed node parallel transmission acceleration method, which relates to the technical field of satellite Internet of Things, and comprises the following steps: dynamically fragmenting, splitting into N fragments according to a network load rate and a node state, adding a serial number, a check code and redundant information, and marking a head as {INJ}; the intelligent routing is used for distributing fragments to distributed nodes through a scoring function based on satellite ephemeris, node loads and link quality; dynamic path switching: monitoring link and node states, and switching to a better path when the performance does not reach the standard; fault-tolerant recombination is carried out, recombination is carried out according to sequence numbers, if missing fragments are few, erasure codes are used for recovery, and if missing fragments are more, local retransmission is carried out. Through parallel transmission, dynamic switching and fault-tolerant recombination, the transmission rate, throughput and robustness are improved, node loads are balanced, and the performance of the satellite Internet of Things is optimized.

The invention relates to the technical field of intelligent capacity expansion, in particular to an intelligent capacity expansion method and system for an enterprise-level solid state disk, and the method comprises the following steps: an I / O controller detects a newly accessed enterprise-level solid state disk and sets a total write-in amount threshold value; according to the method, the total write-in quantity threshold is set after the enterprise-level solid state disk is accessed, and the dynamic delay portrait is constructed in combination with the real-time delay data, so that the performance state of the target hard disk is precisely mastered in the capacity expansion process, and the logic data block and the check code are synchronously embedded in the data write-in stage; and an erasure code check block is synchronously generated in combination with a write-in voucher and written into a far-end address, so that a main and auxiliary path parallel write-in mechanism is realized. And comparing data consistency by reading main and auxiliary addresses, constructing an inconsistent item set, obtaining a corresponding physical region error log by relying on an NVMe interface, extracting an abnormal sector, and mapping to form an unstable physical region coordinate, so that the system has the capability of actively avoiding an unstable region.

The invention provides a fast recovery method for hadoop cluster data, which relates to the technical field of erasure codes, and comprises the following steps of: calculating an importance score of a data block to be recovered, determining a target copy number based on a nonlinear copy number adjustment function, calculating a load weight in combination with network topology information, and selecting candidate nodes; and predicting a node fault probability by using a long short-term memory network, calculating a migration priority based on the importance score and the fault probability, and selecting an optimal target node to execute data recovery. According to the invention, the intelligence and differentiation of the data recovery process are realized, and the data recovery efficiency and reliability are improved.

The invention discloses a data processing method and device, electronic equipment, a storage medium and a product. The method comprises the steps of obtaining a disk dynamic mapping request; in response to the disk dynamic mapping request, a data repair migration task is generated, and the data repair migration task is used for triggering disk dynamic mapping and data repair migration processes of the erasure code redundant group; and executing the data restoration migration task.

The invention discloses an enterprise financial consultation platform based on financial data analysis, and relates to the technical field of enterprise financial data analysis, and the platform comprises a storage medium health monitoring module which collects physical health indexes of storage equipment in real time, constructs a dynamic portrait of the health degree of a storage medium, marks a high-risk storage area, and stores the physical health indexes; generating a dynamically updated storage medium health thermodynamic diagram; according to the method, the physical health indexes of the storage equipment are collected in real time, and the dynamic portrait of the health degree of the storage medium is constructed, so that the aging risk of the storage medium is accurately identified, invisible hardware aging is converted into quantifiable risk information, the risk identification capability is improved, dual-mode verification of a bit layer and a semantic layer is adopted, and the risk identification efficiency is improved. The functions of identifying high-risk silent data errors and capturing local damage of the storage medium are realized; and through a hierarchical repair strategy, the document repair effect is improved by utilizing an erasure code decoder and a FinDoc-RepairGAN processor.

The invention relates to the field of distributed storage, and discloses a distributed memory system based on erasure codes and a dynamic repair mechanism. Comprising a data node cluster, a metadata management cluster, a bandwidth token management module, a dynamic repair composer and a repair agent node. And the data node cluster divides the object into strip generation data fragments and verification fragments and dispersedly stores the data fragments and the verification fragments according to a placement rule. And the metadata management cluster maintains object stripe mapping, a fragment position set, erasure code protection configuration and a version epoch, and records a repair intention log. And the bandwidth token management module grants repair execution permission according to the fault domain budget. The dynamic repair composer generates availability repair and durability repair tasks, the temporary check fragments are written in when the number of the available check fragments is lower than a threshold value in the availability stage, and the configuration fragments are recovered and the temporary fragments are recovered in the durability stage. And the consistency of version epochs is verified before the agent node is repaired and written back, and if not, recalculation is carried out.

The invention discloses an efficient double-link block chain consensus method and device, equipment and a storage medium, and belongs to the field of information security. The method comprises an initialization stage, an input stage, a coding stage, a broadcasting stage and a consensus stage. According to the invention, the parallel design of a coding broadcast link and a consensus confirmation link is adopted, so that the occurrence frequency of transaction congestion is reduced; the integrity required by the asynchronous Byzantine consensus process is realized through three-step broadcasting, the updating problem of the post-quantum component does not need to be considered, and the security is higher; an asynchronous network and a Byzantine opponent model are adopted, so that malicious node attacks which do not exceed the total number of nodes can be tolerated on the premise of not setting a network time delay upper limit; in the encoding stage, the generation of the redundancy share through the RS erasure code can be realized by adopting preprocessing, the data redundancy expansion multiple in the broadcasting stage is far smaller than that of the same type of protocol, and the original transaction can be recovered by collecting part of effective shares in the consensus stage, so that the system time delay is effectively reduced, and the system throughput is improved.

The invention provides a distributed cluster security backup method and system oriented to a cloud-side environment, and relates to the technical field of distributed storage systems, and the method comprises the steps: obtaining cloud-side cluster data, and adaptively selecting a backup mode based on an erasure code and a multi-copy hybrid redundancy strategy; establishing a workload-aware incremental replication mechanism, and optimizing fault recovery by using idle resources; the problem of lock competition is solved by adopting a multi-copy consistency guarantee mechanism based on two-stage submission, and lock-free reading is realized based on MVCC. According to the invention, the data backup flexibility, the fault recovery efficiency and the system consistency guarantee capability are improved.

Methods and apparatus are provided. In an example aspect, a method of managing data storage in a communication system is provided. The communication system includes a plurality of clusters of network nodes, and the network nodes in each cluster of network node include one or more data storage nodes. The method comprises receiving data to be stored from a data source node, generating, from the data to be stored, a first subset of erasure coded chunks, causing the first subset of erasure coded chunks to be stored in one or more of the data storage nodes in at least one first cluster of the plurality of clusters, and sending, to a second cluster of the plurality of clusters, the data to be stored.

The invention provides a transmission performance optimization method and device, equipment, a storage medium and a program product, and relates to the technical field of data transmission. The method comprises the following steps: generating a first file corresponding to each path according to an original data file and erasure code parameters; transmitting the corresponding first file through each path, and acquiring a transmission performance parameter of each path; determining the weight of each path according to the transmission performance parameter of each path; and adjusting the number of data blocks transmitted by each path according to the weight of each path. According to the scheme, the real transmission performance data is obtained in the multi-path transmission process through the first file, the transmission performance of the paths is evaluated from the transmission result level, the distribution strategy of the data blocks of all the paths is dynamically adjusted according to the fed-back real transmission performance parameters, the data transmission efficiency and reliability are effectively improved, and transmission delay is reduced.

Disclosed in the present application are an erasure code data updating method and apparatus. The method comprises: in response to a write operation for a target shard of a target object in a main object storage device, acquiring a state identifier of the target shard; if the state identifier is a first state identifier, storing, in a reserved object in a parity object storage device, initial data in the target shard and initial parity data for the target object, using updated data to overwrite the initial data, and modifying the state identifier to a second state identifier; if the state identifier is the second state identifier, using the updated data to overwrite current data in the target shard and updated data that is stored in the reserved object; and when a distributed object storage system is idle, determining target parity data on the basis of the initial parity data and the initial data and the updated data that are currently stored in the reserved object, and modifying the state identifier to the first state identifier.

The invention discloses a data reconstruction method and device, electronic equipment and a storage medium, and relates to the technical field of data storage and recovery, and the data reconstruction method comprises the steps that under the condition that a to-be-reconstructed disk exists in a distributed storage system, the redundancy of the to-be-reconstructed disk is determined based on state data of the to-be-reconstructed disk; determining a target redundancy level corresponding to the redundancy of the to-be-reconstructed disk from a plurality of preset redundancy levels; different redundancy levels correspond to different erasure code types; the erasure code types corresponding to different redundancy levels represent different combinations of data recovery rates and storage resource occupation proportions; and switching the erasure code type of the to-be-reconstructed disk to a target erasure code type corresponding to the target redundancy level, and according to the target erasure code type, transmitting the fault data block in the to-be-reconstructed disk to a specified disk for data reconstruction. The problem that data recovery performance, resource consumption and network bandwidth resource occupation cannot be dynamically balanced in the data reconstruction process in the prior art is solved.

The invention discloses a distributed data storage and access control method based on a block chain, and relates to the technical field of data security. The method comprises the following steps: fragmenting user data, encrypting the fragmented user data by using an encryption algorithm, and storing the encrypted user data in a plurality of distributed nodes; taking the hash value and the storage position of the data fragment as metadata to be recorded in a block chain, and performing integrity verification through a smart contract; when a user accesses, user attributes are verified based on an attribute-based encryption technology, and an access control strategy is dynamically executed through an intelligent contract; and an erasure code technology is adopted to carry out redundancy storage so as to realize data recovery and fault tolerance. The method effectively solves the problems of single-point failure, privacy disclosure and inflexible access control in traditional centralized storage, has the beneficial effects of improving data security, guaranteeing integrity, realizing fine-grained dynamic access control and enhancing system reliability, and is suitable for data security management in cloud computing, Internet of Things and big data environments.

The embodiment of the invention relates to a data processing method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the coding operation of an erasure code on first data under the condition of receiving a write-in request for the first data, and obtaining a plurality of logic units and a plurality of fragments; according to the first identifier of the logic unit and the second identifier of each fragment, writing the plurality of fragments into a storage node; and under the condition that a reading request is received, obtaining a target first identifier and a target second identifier of the second data, reading the plurality of target fragments of the second data from the plurality of storage nodes, and carrying out merging processing to obtain the second data. Therefore, the data can be divided into the fragments with the unique identifiers for storage in the writing stage, and the target fragments can be quickly positioned according to the logic addresses and the recovered data can be merged in the reading stage, so that the metadata query overhead is reduced, the reading and writing efficiency is improved, and the storage and network transmission cost is reduced; and the overall performance of the distributed storage system in a high-concurrency scene is improved.

The invention provides a temporary check block repairing method and system based on erasure codes in distributed block storage, and the method comprises the steps that a distributed block storage system detects that a check block fails to be written in, and a metadata service creates a temporary check block according to an erasure code mode; the temporary check block records an operation log and executes data writing, and replays IO operation according to a data version number; repairing the inconsistency of the data blocks by using the operation log of the temporary check block; and quickly recovering the check block data through the data bitmap of the temporary check block. According to the rapid recovery processing method for the temporary check block, the check block data can be directly recovered without a coding mode, the internal IO is reduced, and the recovery speed of the erasure code check block is increased.

A fragment and shuffle erasure coding operation is disclosed. In a transformative operation, encrypted fragments are generated from an input file. The transformative operation includes one or more iterations of chunking the file and mixing the chunks to generate a mixed sequence. The mixed sequence is sliced into encrypted fragments. The encrypted fragments are shuffled in a storage system such that physical locations of the encrypted fragments change over time.

The invention discloses a data offline encryption storage method and system based on a mobile terminal, and the method is executed in a network connection-free environment, and comprises the following steps: dynamically dividing original data into N data blocks; performing double encryption on each data block to generate a composite ciphertext; splitting the master key into a plurality of fragments, and distributing the fragments to a local security module and a trusted execution environment for independent storage through a secret sharing algorithm; writing the composite ciphertext into an encrypted database in a tree index structure; during data reading, carrying out integrity verification on the data block based on the Hash tree structure; and when the data is detected to be damaged, performing data reconstruction by using redundant data and an erasure code algorithm. The method disclosed by the invention has the beneficial effects that the security, efficiency and reliability of offline data storage of the mobile terminal can be remarkably improved.

In some embodiments, a method for die-level monitoring is provided. The method includes distributing user data throughout a plurality of storage nodes through erasure coding, wherein the plurality of storage nodes are housed within a chassis that couples the storage nodes. Each of the storage nodes has a non-volatile solid-state storage with non-volatile memory and the user data is accessible via the erasure coding from a remainder of the storage nodes in event of two of the storage nodes being unreachable. The method includes producing diagnostic information that diagnoses the non-volatile memory on a basis of per package, per die, per plane, per block, or per page, the producing performed by each of the plurality of storage nodes. The method includes writing the diagnostic information to a memory in the storage cluster.

The present invention, in the field of blockchain and erasure code technology, specifically discloses a blockchain data storage and reading method based on block aggregation coding. The method comprises: aggregating and encoding multiple blocks using erasure codes based on block storage characteristics to obtain coded blocks; recoding the coded blocks based on data migration characteristics when the blockchain node scale changes to complete blockchain data storage; and reading the blocks based on the data reading characteristics of the coded blocks using an optimized reading process based on local node download, node download, and remote coding group block download. This invention solves the problems of high blockchain data storage redundancy and low data processing efficiency when the number of nodes in the system changes.

The invention belongs to the technical field of computer storage, and particularly discloses an encryption difference compressed data restoration method and device based on similarity perception. According to the method and the device, the differential data block and the reference block which are subjected to aggregation storage are read from the continuous disk space at one time; in the data recovery process, pipelining processing is carried out on a difference decoding task and an erasure code decoding task; and performing data restoration according to the size of the data slice, the difference data block and the reference block by executing the pipelined difference decoding task and the pipelined erasure code decoding task. According to the mode, the whole aggregation storage data block containing the differential data block and the reference block is read from the continuous disk space, compared with respective reading, one-time extra disk overhead can be reduced, then the differential decoding task and the erasure code decoding task are subjected to pipelining processing, and the decoding efficiency is improved. And data recovery is carried out by executing the decoding task, so that the data recovery performance of the encryption difference compression system can be effectively improved.

The invention discloses an HDFS (Hadoop Distributed File System) intelligent hierarchical storage method and system based on multi-dimensional popularity perception and tape library integration, and belongs to the technical field of distributed file systems, and the method comprises the following steps: expanding HDFS metadata to construct a unified namespace; acquiring data block access features and dynamically dividing hot, warm and cold data by using a K-means clustering algorithm; executing a self-adaptive grading strategy: adding a copy to hot data and storing in an SSD (Solid State Disk), maintaining a default copy to warm data and storing in a disk, and migrating cold data to a tape library and storing by adopting an erasure code; scattered storage and fault tolerance of relevance perception are realized through a message queue; metadata persistence and periodical re-evaluation are carried out, and automatic migration and dynamic adjustment of data among the SSD layer, the disk layer and the tape library layer are achieved. By adopting the method and the system, the storage strategy and the data access value are accurately matched, the storage cost is finally reduced, and the system I / O performance and the resource utilization rate are improved.

The invention relates to the technical field of computer science, and discloses an efficient file storage system with a distributed architecture, which comprises the following modules: a distributed file storage module for segmenting file data and distributing the file data to a plurality of storage nodes by adopting a distributed storage technology to ensure high availability and expansibility of the data; the intelligent data management module is used for dynamically managing data loads among storage nodes based on a data distribution algorithm and supporting automatic expansion and optimization of storage resources; the high-speed file access module is used for ensuring that a user can efficiently perform file read-write operation through a distributed access mechanism and an optimized data transmission protocol; and the data redundancy and backup module is used for providing a flexible redundancy strategy and combining a replica redundancy technology and an erasure code technology. By optimizing the file storage technology, the storage efficiency can be improved, the storage space can be reasonably utilized, dynamic expansion of the system is supported, and the problems that an existing system is slow in response and poor in expansibility when coping with large-scale data are solved.

The technical scheme provides a data error correction and performance optimization method based on multiple optical disks, comprising the following steps: S1, generating a logical volume which is in a temporary data state before data recording and comprises a data block and a verification block; s2, converting the logical volume into a physical volume through a recombination algorithm, wherein the physical volume is a mirror image file which can be directly recorded to an optical disc; s3, respectively burning the physical volume to a plurality of corresponding optical disks, and synchronously mounting the optical disks through a plurality of CD drivers to realize concurrent read-write operation on the physical volume; and S4, when the data block in any optical disc is damaged, reversely calculating the position of the damaged data block in the logical volume through the recombination algorithm, calling undamaged data blocks and verification blocks corresponding to the damaged data block in other optical discs, and recovering the damaged data block based on an erasure code algorithm. According to the technical scheme, the reading performance of the large file can be optimized, and data protection can be provided.

The application relates to the technical field of distributed storage, and discloses a data storage method and device, computer equipment and a storage medium, the method comprising the following steps: in the case of using a storage node to perform erasure code calculation, whenever receiving to-be-stored data, randomly selecting a storage node and sending the to-be-stored data to the storage node; using the storage node to receive the to-be-stored data, obtaining instruction set information of the highest level supported by a central processing unit of the storage node, obtaining an encoding function matched with the instruction set information, performing encoding processing on the to-be-stored data based on the encoding function, and obtaining an erasure code corresponding to the to-be-stored data; and associating the erasure code with the to-be-stored data. The application adaptively selects the instruction set information of the highest level supported by the central processing unit, uses the encoding function corresponding to the instruction set information to perform erasure code calculation, and obtains the erasure code, so that the calculation efficiency of the erasure code is improved.

The invention discloses a threat intelligence synchronization erasure code redundancy check and incremental error correction method, which relates to the technical field of network security, and comprises the following steps: S1, obtaining a threat intelligence data block sequence, extracting an original hash value and cyclic redundancy check of each data block from a distributed transmission path, and obtaining a threat intelligence data block sequence; s2, according to the preliminary verification result, judging whether the integrity index is lower than a preset threshold value or not, if the integrity index is lower than the preset threshold value, positioning an abnormal data block in the sequence, and determining a pollution position by comparing a hash value of each data block with parity check verification; according to the erasure code redundancy check and incremental error correction method for threat intelligence synchronization, high-reliability transmission and rapid recovery of threat intelligence data in a distributed environment are realized by integrating error correction, verification and synchronization mechanisms, and the data security and the system robustness are effectively improved.

This application belongs to the field of computer storage technology, specifically disclosing a cross-rack sensing invalid data reclamation method and apparatus based on erasure coding. The method includes: determining the cross-rack update network traffic generated by simulating the reclamation of invalid data blocks in each erasure coding stripe based on the invalid data blocks and valid check blocks stored in each rack; dividing each erasure coding stripe into filling stripes and reclamation stripes based on the cross-rack update network traffic corresponding to each erasure coding stripe; migrating the valid data blocks of each reclamation stripe to the storage nodes where the invalid data blocks in each filling stripe are located, and releasing the storage space of each reclamation stripe; and updating the valid check blocks of each filling stripe. This application achieves a more scientific and efficient invalid data reclamation method, effectively reducing network traffic throughout the invalid data reclamation process, improving invalid data reclamation efficiency, while ensuring system storage efficiency.