50 results about "Dispersed storage" patented technology

The invention discloses a distributed storage method and system for data security, and belongs to the technical field of data storage security, the method comprises the following steps: firstly, fragmenting original data, and generating a unique identifier for each data fragment; for each fragment, an encryption algorithm and key credentials are adaptively selected, and the key credentials are stored in a key management system. By performing homomorphic encryption on the key voucher, an encryption handle is generated, and efficient positioning and retrieval of the key are realized. According to the method, a differentiated encryption strategy is adopted, a most suitable encryption scheme is selected according to fragment characteristics, and encrypted data fragments are dispersed and stored in a plurality of distributed storage nodes. A global mapping index is constructed in a metadata management server, and a corresponding relation between an original data global identifier and metadata of each data fragment is recorded, including information such as identifiers before and after encryption, a storage position, an encryption algorithm, a key handle and the like. According to the method, the data leakage risk is effectively reduced, and the safety and reliability of the storage system are improved.

Systems and methods for decentralized data storage and retrieval within a distributed computing system include partitioning datasets into encrypted data shards that are each associated with a unique identifier. A processing system operating within the network may distribute these shards across storage nodes selected based on performance metrics such as latency, storage utilization, and reliability. Metadata for each shard, including its identifier, unique cryptographic key, and node location, may be recorded in a tamper-evident distributed ledger. Upon receiving a retrieval request, the system may query the ledger to identify shard locations, retrieve the shards in parallel, validate their integrity by comparing ledger metadata with metadata from storage nodes, decrypt the shards using their respective cryptographic keys, and reconstruct the dataset. These systems and methods enhance data security, integrity, and fault tolerance while allowing efficient and verifiable decentralized storage and retrieval in dynamic, distributed environments.

The invention discloses a data backup and recovery method, which relates to the technical field of data backup and recovery, and comprises the following steps: S1, pre-analyzing data to be backed up; s2, connecting a storage medium; s3, data transmission backup; s4, data backup recording; and S5, data recovery. According to the data backup recovery method, a plurality of independent storage media are adopted, only one medium is connected with a computer at the same time, the media are physically isolated, data are not intercommunicated, even if a certain medium is damaged, backup data of other media are still intact, and a single-point fault is avoided; the physical movement requirement is reduced to prevent the storage medium from being lost, all data blocks of a single file can be ensured to be completely stored in the same medium by analyzing the file structure and the incidence relation in advance, it is ensured that the data logic relation can be completely restored during concentrated storage and recovery of related files, and the incidence failure caused by scattered storage is avoided.

The invention relates to the field of distributed storage, and discloses a distributed memory system based on erasure codes and a dynamic repair mechanism. Comprising a data node cluster, a metadata management cluster, a bandwidth token management module, a dynamic repair composer and a repair agent node. And the data node cluster divides the object into strip generation data fragments and verification fragments and dispersedly stores the data fragments and the verification fragments according to a placement rule. And the metadata management cluster maintains object stripe mapping, a fragment position set, erasure code protection configuration and a version epoch, and records a repair intention log. And the bandwidth token management module grants repair execution permission according to the fault domain budget. The dynamic repair composer generates availability repair and durability repair tasks, the temporary check fragments are written in when the number of the available check fragments is lower than a threshold value in the availability stage, and the configuration fragments are recovered and the temporary fragments are recovered in the durability stage. And the consistency of version epochs is verified before the agent node is repaired and written back, and if not, recalculation is carried out.

The invention belongs to the technical field of spaceflight electronic integrated circuits, and particularly relates to a satellite load FPGA configuration data error correction method and system capable of resisting single event upset, in particular to a bit-symbol double-layer error correction method of RS decoding operator error correction based on finite field algebra cascading on the basis of two-out-of-three majority voting logic. Single-particle independent upset and multi-copy common-mode failure can be corrected at the same time, and the problem of traditional TMR error locking is solved. Meanwhile, when configuration data are written into a nonvolatile memory, a cross-sector space staggered and dispersed storage mechanism is introduced, symbols in the same RS code word are separated as far as possible in physical positions, and therefore aggregated physical damage is converted into dispersed symbol-level errors.

A distributed storage integrity system in a dispersed storage network includes a scanning agent and a control unit. The scanning agent identifies an encoded data slice that requires rebuilding, wherein the encoded data slice is one of a plurality of encoded data slices generated from a data segment using an error encoding dispersal function. The control unit retrieves at least a number T of encoded data slices needed to reconstruct the data segment based on the error encoding dispersal function. The control unit is operable to reconstruct the data segment from at least the number T of the encoded data slices and generate a rebuilt encoded data slice from the reconstructed data segment. The scanning agent is located in a storage unit and the control unit is located in the storage unit or in a storage integrity processing unit, a dispersed storage processing unit or a dispersed storage managing unit.

A computing device for use in a storage network includes one or more network interfaces, memory with operational instructions and a processing module. The processing module is configured to execute the operational instructions receive a storage request for data and select a plurality of memory devices for storage of the data. The processing module partitions the data into a plurality of chunk sets and encode each chunk set of the plurality of chunk sets according to a dispersed storage error coded function to produce a plurality of chunks. Finally, the processing module facilitates storage of each chunk of the plurality of chunks in a memory device of the plurality of memory devices.

The invention discloses an HDFS (Hadoop Distributed File System) intelligent hierarchical storage method and system based on multi-dimensional popularity perception and tape library integration, and belongs to the technical field of distributed file systems, and the method comprises the following steps: expanding HDFS metadata to construct a unified namespace; acquiring data block access features and dynamically dividing hot, warm and cold data by using a K-means clustering algorithm; executing a self-adaptive grading strategy: adding a copy to hot data and storing in an SSD (Solid State Disk), maintaining a default copy to warm data and storing in a disk, and migrating cold data to a tape library and storing by adopting an erasure code; scattered storage and fault tolerance of relevance perception are realized through a message queue; metadata persistence and periodical re-evaluation are carried out, and automatic migration and dynamic adjustment of data among the SSD layer, the disk layer and the tape library layer are achieved. By adopting the method and the system, the storage strategy and the data access value are accurately matched, the storage cost is finally reduced, and the system I / O performance and the resource utilization rate are improved.

A method for a storage network computing device includes receiving a request to store a data object and generating a plurality of write slice requests, where the plurality of write slice requests includes a set of encoded data slices to be stored in the storage network and at least a decode threshold number of encoded data slices of the set of encoded data slices is required to recover a corresponding portion of a data object. The method continues by identifying a set of storage devices (from a plurality) for storage of the set of encoded data slices, where the plurality of storage devices includes one or more first memory devices local to the computing device and one or more second memory devices that are remote from the computing device. The method then continues, by storing the set of encoded data slices in the identified set of storage devices, generating a set of write slice responses relating to the set of encoded data slices and outputting the set of write slice responses.

A processing system of a storage network operates by receiving a write request associated with a data object; identifying dispersed storage error encoding parameters; and determining a number of available storage units. When the number of available storage units is determined to exceed a first number, the processing system operates by: determining a first write threshold number in accordance with the dispersed storage error encoding parameters; dispersed error encoding the data object to produce a first write threshold number of encoded data slices; and writing, to available storage units, the first write threshold number of encoded data slices. When the number of available storage units is determined to not exceed the first number, the processing system operates by: determining a second write threshold number in accordance with the dispersed storage error encoding parameters associated, wherein the second write threshold number is less than the first write threshold number; dispersed error encoding the data object to produce a second write threshold number of encoded data slices; and writing, to the available storage units, the second write threshold number of encoded data slices.

This invention relates to the field of distributed storage technology and provides a data collaboration method and system for distributed transaction terminals. The method involves each transaction terminal performing erasure coding on its local ledger data to generate a set of data fragments consisting of original data fragments and verification data fragments. These data fragments are then distributed and stored across different transaction terminals according to distributed storage rules, and the terminals exchange online status and fragment integrity information. When a request to access the target ledger is received, the terminal storing its fragments is located based on the ledger identifier, and the fragments are retrieved in parallel to reconstruct the target ledger. When a terminal is detected to be offline or a fragment is corrupted, online terminals collaborate to reconstruct the original data using surviving fragments, re-encode new fragments, and distribute them to healthy terminals. This invention, through erasure coding and distributed collaborative storage, reduces storage overhead while improving system fault tolerance and data availability, achieving efficient and reliable distributed ledger data collaborative management.

The invention discloses a data storage method and a related device, and the method comprises the steps: firstly obtaining factory data which needs to be stored through a TLC NAND technology, then compressing the factory data to obtain compressed data, dividing the compressed data into a plurality of strips, then generating a corresponding XOR check code for each strip, and storing the XOR check code in a database; and finally, the compressed data and the XOR check code corresponding to each strip are written into different physical units, so that the factory data are compressed and then put into a disk, the written data volume can be reduced, the abrasion of the embedded storage equipment can be relieved, the service life can be prolonged, and the storage efficiency can be improved by utilizing the XOR check code. The XOR check code and the survivor data are stored in different physical units in a dispersed manner, so that data recovery and reconstruction can be performed through the XOR check code and the survivor data when part of data in the strip is lost, the data integrity requirement of the embedded storage device for sudden power failure is met, the persistent retention capability of data storage is further improved, and the data storage efficiency is improved. And the reliability of data storage is improved.

The application relates to the technical field of computer application, and discloses a mass file data stream storage method, a storage system and a big data processing system. A rotation state conversion model is arranged between a data stream and a storage node, and storage pressure is dispersed through multi-node caching. The storage method comprises the following steps: presetting a file size demarcation point; inputting mass file data; judging whether the size of the input file is smaller than or equal to the demarcation point; storing file data smaller than or equal to the demarcation point in a selected storage node through the rotation state conversion model; generating index information of all small files, and merging the index information into an index file; organizing data in the form of a B+ tree for each index file to obtain an index data bucket; storing a plurality of index data buckets in the form of data blocks in the storage node; and saving all index data buckets in sequence as an index data bucket queue. The application can effectively solve the problems of traditional distributed storage systems, such as write performance decline and low efficiency when facing mass files.

One or more techniques and / or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and / or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and / or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and / or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and / or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items.

The invention relates to the technical field of computer application, and discloses a mass file data stream storage method, a mass file data stream storage system and a big data processing system, a rotation state conversion model is deployed between a data stream and a storage node, and the storage pressure is dispersed through multi-node cache. The storage method comprises the following steps: presetting a file size demarcation point; inputting massive file data, and judging whether the size of the input file is smaller than or equal to the demarcation point one by one; file data smaller than or equal to the demarcation point is stored in a selected storage node through a rotation state conversion model; generating index information of all the small files, combining the index information to index files, and organizing data in a B + tree form for each index file to obtain an index data bucket; a plurality of index data buckets are stored in a storage node in a data block form, and all the index data buckets are stored as an index data bucket queue in sequence. The method can effectively solve the problems that the writing performance is reduced and the efficiency is low when a traditional distributed storage system faces massive files.

The application belongs to the technical field of aerospace electronic integrated circuits, and particularly relates to a satellite load FPGA configuration data error correction method and system resisting single event upsets, a double-layer error correction method of "bit-symbol" based on RS decoding operator error correction of a finite field algebra on the basis of a "two out of three" majority voting logic, which can correct single particle independent upsets and multiple copy common mode failures at the same time, and solve the traditional TMR error locking problem. Meanwhile, when configuration data is written into a non-volatile memory, a cross-sector spatial staggered dispersion storage mechanism is introduced, so that the symbols in the same RS code word are separated as much as possible in the physical position, so as to convert the aggregation physical damage into dispersed symbol level errors.

The invention discloses a data storage method and system, electronic equipment and a storage medium, and relates to the technical field of data storage.The whole data storage system is divided into a plurality of storage units, each storage unit comprises a buffer and a storage tree, newly-written data are firstly stored in the buffers, and then the newly-written data are stored in the storage tree; according to the method and the device, the buffer is divided into a plurality of storage sub-trees, the buffer is subsequently subjected to data down-brushing to brush the data down to the storage tree, so that batch writing of the data of the storage tree is realized, write amplification of the storage tree is reduced, and when the storage tree level is high, the high-level storage tree is divided into the plurality of storage sub-trees, so that scattered storage of the data is realized, the situation that the storage tree level is too high is avoided, and the storage efficiency is improved. The data access time delay of the storage tree is reduced, and a foundation is laid for improving the concurrent access amount and the access efficiency of the data.

The invention discloses an electric power engineering data centralized filing processing system and method based on cloud storage. The method comprises the following steps: carrying out digital formatting processing on all electric power engineering data through a scanner; uploading the digitized electric power engineering data to a cloud storage platform based on a distributed storage technology; analyzing relevance among the data to obtain a relevance matrix; constructing a hash space of distributed storage based on a hash ring; based on the obtained relevance matrix, data are stored in a hash ring in a scattered mode through a distributed hash algorithm according to relevance; and carrying out dynamic archiving management on the electric power engineering data based on load balancing storage. The method has the advantages that high-relevance data are stored in a scattered mode by analyzing data relevance, data safety is improved, efficient scattered storage and dynamic management of electric power engineering data are achieved through the distributed hash algorithm and the load balancing technology, and data storage reliability and system operation efficiency are improved.

A processing system operates by: detecting an access anomaly associated with an access request from a requestor for a set of encoded data slices, the access anomaly having an unfavorable access pattern, wherein the set of encoded data slices is dispersed storage error encoded and stored in at least one storage unit of the storage network; denying the access request in response to detecting the access anomaly; generating, based on the unfavorable access pattern, an anomaly detection indicator identifying the requestor; and sending the anomaly detection indicator to other devices of the storage network.

A computing device of a storage network includes a network interface, memory and a processing module operably coupled to the memory and the network interface, with the processing module configured to execute the operational instructions to receive a storage request for a data file and then determine a data type for the data file. Based on the data type, the computing device determines whether to partition the data file into variable-sized data blocks and in response to a determination to do so, partitions the datafile according to a partition function into a plurality of data blocks, encodes each data block according to a dispersed storage error coded function to produce a plurality of data fragments and facilitates storage of the plurality of data fragments in a plurality of storage network nodes.

A computing system is operable to decode a plurality of incoming streams of digitally encoded data packets to extract a plurality of sets of input data from the plurality of incoming streams of digitally encoded data. A plurality of encrypted user data is generated based on encrypting at least some of the plurality of sets of input data and each of the plurality of encrypted user data is stored via a plurality of different storage locations. The plurality of sets of input data are processed generate a plurality of automatically generated textual data. A plurality of further outgoing streams of digitally encoded data packets are generated and transmitted via processing the plurality of automatically generated textual data.

The invention discloses a data collaborative storage management method and system for credit investigation data, and relates to the technical field of storage systems. The data collaborative storage management method and system for the credit investigation data comprises the steps that S1, the credit investigation data are collected and preprocessed, the credit investigation data are stored, and a credit investigation database is constructed; s2, carrying out consistency risk analysis through the fragmented data and the write-in operation data; s3, carrying out write-in orderliness analysis in combination with the global write-in sequence, the sorting priority and the consistency risk data; s4, repairing the fragmented data, and carrying out consistency repairing verification analysis based on the fragmented data before and after repairing; and S5, according to the fragmentation state, operating snapshots and repairing log data, archiving credit investigation data and continuously completing risk management and control. The problems that data are stored in a scattered mode, dirty writing conflicts occur frequently and the fragment consistency guarantee capacity is insufficient in the high-concurrency writing environment of the multi-service system are solved.

The utility model relates to the field of storage devices, in particular to an art tool integrated storage device which comprises a storage box and a storage assembly. A storage assembly used for classified storage according to art tools is installed in the storage box. The storage assembly comprises a sealing plate, a paint frame, a storage cabinet and a storage plate. The drawing board, tools and pigment are integrally and intensively stored through the pigment frame, the storage cabinet and the storage plate, limited space is fully utilized through reasonable layout, space waste caused by scattered storage is avoided, the disordered situation is reduced, the number of objects needing to be carried is reduced, follow-up movable creation is facilitated, and the drawing board is convenient to use. Meanwhile, all art tools are stored in a classified mode, a user can conveniently manage and search needed articles, the use convenience is improved, the use device is integrally stored, good sealing performance and protectiveness can be provided for the tools and the drawing board, paint is prevented from drying up or being affected with damp, the drawing board and the tools are protected against damage, and the overall storage effect is improved.

A processing system of a storage network operates by receiving a write request associated with a data object; identifying dispersed storage error encoding parameters; and determining a number of available storage units. When the number of available storage units is determined to exceed a first number, the processing system operates by: determining a first write threshold number in accordance with the dispersed storage error encoding parameters; dispersed error encoding the data object to produce a first write threshold number of encoded data slices; and writing, to available storage units, the first write threshold number of encoded data slices. When the number of available storage units is determined to not exceed the first number, the processing system operates by: determining a second write threshold number in accordance with the dispersed storage error encoding parameters associated, wherein the second write threshold number is less than the first write threshold number; dispersed error encoding the data object to produce a second write threshold number of encoded data slices; and writing, to the available storage units, the second write threshold number of encoded data slices.

One or more techniques and / or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and / or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and / or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and / or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and / or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items.

A method for execution by a computing device of a storage network includes appending at least a decode threshold number of encoded key slices of a set of encoded key slices to at least some encrypted data segments of a plurality of encrypted data segments to produce secure packages. The method further includes error encoding, in accordance with error encoding parameters, the secure packages to produce sets of encoded data slices, where a first secure package of the secure packages is dispersed storage error encoded using an error encoding function of the error encoding parameters to produce a first set of encoded data slices of the sets of encoded data slices. The method further includes outputting the sets of encoded data slices for storage in memory of the storage network.

This invention provides a method for transmitting railway Building Information Modeling (BIM) model files based on distributed storage. The method includes: lightweight decomposing the railway BIM model file into multiple sub-files; distributing these sub-files across edge server nodes using a cloud server; transmitting the sub-file list to a gateway; constructing a BIMdex index model; training the BIMdex index model using a training dataset; deploying the trained BIMdex index model locally on the gateway; and when an end device sends a file request carrying a file identifier to the gateway, the BIMdex index model in the gateway queries the sub-file list based on the file identifier, obtains the corresponding BIMdex address index, and returns it to the end device. This invention significantly improves data transmission efficiency by distributing sub-files of the same railway BIM model across multiple physical nodes and combining this with load balancing technology.

The invention discloses a network data distributed storage management system, relates to the technical field of data storage, and adopts the technical scheme that the network data distributed storage management system comprises a distributed storage management module, the system is characterized in that the distributed storage management module comprises a distributed architecture module, a fragmentation management module, a data replication application module, a fault-tolerant mechanism trigger module, an operation and maintenance management and control module, a distributed event processing module, a data concurrency control module, a storage performance optimization module, a query processing optimization module and a data cloud combination management module; the method has the beneficial effects that a plurality of storage servers are utilized to share storage load, data are stored on each node in a scattered manner, no single-point fault exists, the reliability and availability of the system are improved, the system can be flexibly expanded, the large-scale storage requirement is met, the risk of the single-point fault is reduced, the management process is simplified, the automation of operation and maintenance tasks is realized, and the operation and maintenance efficiency is improved. The security and privacy of data storage are improved, the storage utilization rate is high, and the data recovery speed is high.

The invention relates to the technical field of civil aviation data security, in particular to a data security protection method and system for a civil aviation data intermediate station, and the method comprises the steps: obtaining first data; dividing the high-secrecy-level data into N data sub-blocks according to data types, and storing the N data sub-blocks in N storage modules in a dispersed manner; generating a confusion variable by adopting a random number generation algorithm, decomposing the confusion variable into N confusion sub-blocks, and storing the N confusion sub-blocks in N storage modules; destroying the confusion variables; generating a limited access strategy through a relevance blocking method; and when the formatting instruction is received, backing up the confusion sub-blocks stored in the storage module which receives the formatting instruction to the storage module which does not receive the formatting instruction, and then formatting. According to the method, the high-secrecy-level data is stored in a distributed manner, and the confusion sub-blocks are generated to confuse the relevance feature data, so that the data security is improved on the premise that the data value of the relevance feature data is brought into full play.

A method for execution in a storage network begins by determining to adjust a version number associated with a data object stored in the storage network, where the data object was dispersed storage error encoded to produce a set of encoded data slices. The method continues by identifying a number of versions associated with the data object and based on the number of versions identified and the adjusted version number associated with the data object, generating metadata associated with the data object. The method then continues by issuing a write request for each encoded data slice of the set of encoded data slices to the set of storage units, where each write request includes the metadata associated with the data object.