61 results about "Dispersed storage" patented technology

The invention discloses a distributed storage method and system for data security, and belongs to the technical field of data storage security, the method comprises the following steps: firstly, fragmenting original data, and generating a unique identifier for each data fragment; for each fragment, an encryption algorithm and key credentials are adaptively selected, and the key credentials are stored in a key management system. By performing homomorphic encryption on the key voucher, an encryption handle is generated, and efficient positioning and retrieval of the key are realized. According to the method, a differentiated encryption strategy is adopted, a most suitable encryption scheme is selected according to fragment characteristics, and encrypted data fragments are dispersed and stored in a plurality of distributed storage nodes. A global mapping index is constructed in a metadata management server, and a corresponding relation between an original data global identifier and metadata of each data fragment is recorded, including information such as identifiers before and after encryption, a storage position, an encryption algorithm, a key handle and the like. According to the method, the data leakage risk is effectively reduced, and the safety and reliability of the storage system are improved.

A method begins by a computing device of a storage network receiving a storage network access request and determining whether the storage network access request requires multiple access steps to a plurality of storage units, where a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed storage error encoding parameters to produce a set of encoded data slices that are distributedly stored among a plurality of storage units, and where a decode threshold number of encoded data slices are needed to recover the data segment. When the storage network access request requires multiple access steps to a plurality of storage units, the method continues with the computing device determining to delegate at least as portion of the multiple access steps to a delegation agent issuing a multi-step object access partial task to the delegation agent; and receiving a multistep object access result.

Systems and methods for decentralized data storage and retrieval within a distributed computing system include partitioning datasets into encrypted data shards that are each associated with a unique identifier. A processing system operating within the network may distribute these shards across storage nodes selected based on performance metrics such as latency, storage utilization, and reliability. Metadata for each shard, including its identifier, unique cryptographic key, and node location, may be recorded in a tamper-evident distributed ledger. Upon receiving a retrieval request, the system may query the ledger to identify shard locations, retrieve the shards in parallel, validate their integrity by comparing ledger metadata with metadata from storage nodes, decrypt the shards using their respective cryptographic keys, and reconstruct the dataset. These systems and methods enhance data security, integrity, and fault tolerance while allowing efficient and verifiable decentralized storage and retrieval in dynamic, distributed environments.

The invention discloses a data backup and recovery method, which relates to the technical field of data backup and recovery, and comprises the following steps: S1, pre-analyzing data to be backed up; s2, connecting a storage medium; s3, data transmission backup; s4, data backup recording; and S5, data recovery. According to the data backup recovery method, a plurality of independent storage media are adopted, only one medium is connected with a computer at the same time, the media are physically isolated, data are not intercommunicated, even if a certain medium is damaged, backup data of other media are still intact, and a single-point fault is avoided; the physical movement requirement is reduced to prevent the storage medium from being lost, all data blocks of a single file can be ensured to be completely stored in the same medium by analyzing the file structure and the incidence relation in advance, it is ensured that the data logic relation can be completely restored during concentrated storage and recovery of related files, and the incidence failure caused by scattered storage is avoided.

The invention relates to the field of distributed storage, and discloses a distributed memory system based on erasure codes and a dynamic repair mechanism. Comprising a data node cluster, a metadata management cluster, a bandwidth token management module, a dynamic repair composer and a repair agent node. And the data node cluster divides the object into strip generation data fragments and verification fragments and dispersedly stores the data fragments and the verification fragments according to a placement rule. And the metadata management cluster maintains object stripe mapping, a fragment position set, erasure code protection configuration and a version epoch, and records a repair intention log. And the bandwidth token management module grants repair execution permission according to the fault domain budget. The dynamic repair composer generates availability repair and durability repair tasks, the temporary check fragments are written in when the number of the available check fragments is lower than a threshold value in the availability stage, and the configuration fragments are recovered and the temporary fragments are recovered in the durability stage. And the consistency of version epochs is verified before the agent node is repaired and written back, and if not, recalculation is carried out.

Storing data for machine learning and artificial intelligence applications in a decentralized storage network, including: identifying a plurality of decentralized storage networks that a storage system can utilize for storing data, each of the plurality of decentralized storage networks comprising a collection of network connected computers operating as cooperative participants without employing dedicated servers for the storage of data; selecting, based characteristics of each decentralized storage network, one or more decentralized storage networks for storing the data; and initiating storage of the data on the selected one of more decentralized storage networks.

One or more techniques and / or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and / or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and / or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and / or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and / or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items.

The invention belongs to the technical field of spaceflight electronic integrated circuits, and particularly relates to a satellite load FPGA configuration data error correction method and system capable of resisting single event upset, in particular to a bit-symbol double-layer error correction method of RS decoding operator error correction based on finite field algebra cascading on the basis of two-out-of-three majority voting logic. Single-particle independent upset and multi-copy common-mode failure can be corrected at the same time, and the problem of traditional TMR error locking is solved. Meanwhile, when configuration data are written into a nonvolatile memory, a cross-sector space staggered and dispersed storage mechanism is introduced, symbols in the same RS code word are separated as far as possible in physical positions, and therefore aggregated physical damage is converted into dispersed symbol-level errors.

A distributed storage integrity system in a dispersed storage network includes a scanning agent and a control unit. The scanning agent identifies an encoded data slice that requires rebuilding, wherein the encoded data slice is one of a plurality of encoded data slices generated from a data segment using an error encoding dispersal function. The control unit retrieves at least a number T of encoded data slices needed to reconstruct the data segment based on the error encoding dispersal function. The control unit is operable to reconstruct the data segment from at least the number T of the encoded data slices and generate a rebuilt encoded data slice from the reconstructed data segment. The scanning agent is located in a storage unit and the control unit is located in the storage unit or in a storage integrity processing unit, a dispersed storage processing unit or a dispersed storage managing unit.

A computing device for use in a storage network includes one or more network interfaces, memory with operational instructions and a processing module. The processing module is configured to execute the operational instructions receive a storage request for data and select a plurality of memory devices for storage of the data. The processing module partitions the data into a plurality of chunk sets and encode each chunk set of the plurality of chunk sets according to a dispersed storage error coded function to produce a plurality of chunks. Finally, the processing module facilitates storage of each chunk of the plurality of chunks in a memory device of the plurality of memory devices.

The invention discloses an HDFS (Hadoop Distributed File System) intelligent hierarchical storage method and system based on multi-dimensional popularity perception and tape library integration, and belongs to the technical field of distributed file systems, and the method comprises the following steps: expanding HDFS metadata to construct a unified namespace; acquiring data block access features and dynamically dividing hot, warm and cold data by using a K-means clustering algorithm; executing a self-adaptive grading strategy: adding a copy to hot data and storing in an SSD (Solid State Disk), maintaining a default copy to warm data and storing in a disk, and migrating cold data to a tape library and storing by adopting an erasure code; scattered storage and fault tolerance of relevance perception are realized through a message queue; metadata persistence and periodical re-evaluation are carried out, and automatic migration and dynamic adjustment of data among the SSD layer, the disk layer and the tape library layer are achieved. By adopting the method and the system, the storage strategy and the data access value are accurately matched, the storage cost is finally reduced, and the system I / O performance and the resource utilization rate are improved.

A processing system operates by: detecting an access anomaly associated with an access request from a requestor for a set of encoded data slices, the access anomaly having an unfavorable access pattern, wherein the set of encoded data slices is dispersed storage error encoded and stored in at least one storage unit of the storage network; denying the access request in response to detecting the access anomaly; generating, based on the unfavorable access pattern, an anomaly detection indicator identifying the requestor; and sending the anomaly detection indicator to other devices of the storage network.

A method for a storage network computing device includes receiving a request to store a data object and generating a plurality of write slice requests, where the plurality of write slice requests includes a set of encoded data slices to be stored in the storage network and at least a decode threshold number of encoded data slices of the set of encoded data slices is required to recover a corresponding portion of a data object. The method continues by identifying a set of storage devices (from a plurality) for storage of the set of encoded data slices, where the plurality of storage devices includes one or more first memory devices local to the computing device and one or more second memory devices that are remote from the computing device. The method then continues, by storing the set of encoded data slices in the identified set of storage devices, generating a set of write slice responses relating to the set of encoded data slices and outputting the set of write slice responses.

A processing system of a storage network operates by receiving a write request associated with a data object; identifying dispersed storage error encoding parameters; and determining a number of available storage units. When the number of available storage units is determined to exceed a first number, the processing system operates by: determining a first write threshold number in accordance with the dispersed storage error encoding parameters; dispersed error encoding the data object to produce a first write threshold number of encoded data slices; and writing, to available storage units, the first write threshold number of encoded data slices. When the number of available storage units is determined to not exceed the first number, the processing system operates by: determining a second write threshold number in accordance with the dispersed storage error encoding parameters associated, wherein the second write threshold number is less than the first write threshold number; dispersed error encoding the data object to produce a second write threshold number of encoded data slices; and writing, to the available storage units, the second write threshold number of encoded data slices.

This invention relates to the field of distributed storage technology and provides a data collaboration method and system for distributed transaction terminals. The method involves each transaction terminal performing erasure coding on its local ledger data to generate a set of data fragments consisting of original data fragments and verification data fragments. These data fragments are then distributed and stored across different transaction terminals according to distributed storage rules, and the terminals exchange online status and fragment integrity information. When a request to access the target ledger is received, the terminal storing its fragments is located based on the ledger identifier, and the fragments are retrieved in parallel to reconstruct the target ledger. When a terminal is detected to be offline or a fragment is corrupted, online terminals collaborate to reconstruct the original data using surviving fragments, re-encode new fragments, and distribute them to healthy terminals. This invention, through erasure coding and distributed collaborative storage, reduces storage overhead while improving system fault tolerance and data availability, achieving efficient and reliable distributed ledger data collaborative management.

The invention discloses a data storage method and a related device, and the method comprises the steps: firstly obtaining factory data which needs to be stored through a TLC NAND technology, then compressing the factory data to obtain compressed data, dividing the compressed data into a plurality of strips, then generating a corresponding XOR check code for each strip, and storing the XOR check code in a database; and finally, the compressed data and the XOR check code corresponding to each strip are written into different physical units, so that the factory data are compressed and then put into a disk, the written data volume can be reduced, the abrasion of the embedded storage equipment can be relieved, the service life can be prolonged, and the storage efficiency can be improved by utilizing the XOR check code. The XOR check code and the survivor data are stored in different physical units in a dispersed manner, so that data recovery and reconstruction can be performed through the XOR check code and the survivor data when part of data in the strip is lost, the data integrity requirement of the embedded storage device for sudden power failure is met, the persistent retention capability of data storage is further improved, and the data storage efficiency is improved. And the reliability of data storage is improved.

The application relates to the technical field of computer application, and discloses a mass file data stream storage method, a storage system and a big data processing system. A rotation state conversion model is arranged between a data stream and a storage node, and storage pressure is dispersed through multi-node caching. The storage method comprises the following steps: presetting a file size demarcation point; inputting mass file data; judging whether the size of the input file is smaller than or equal to the demarcation point; storing file data smaller than or equal to the demarcation point in a selected storage node through the rotation state conversion model; generating index information of all small files, and merging the index information into an index file; organizing data in the form of a B+ tree for each index file to obtain an index data bucket; storing a plurality of index data buckets in the form of data blocks in the storage node; and saving all index data buckets in sequence as an index data bucket queue. The application can effectively solve the problems of traditional distributed storage systems, such as write performance decline and low efficiency when facing mass files.

One or more techniques and / or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and / or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and / or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and / or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and / or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items.

Systems and methods for securely and remotely storing data in a remote, distributed redundant array of independent drives (RAID) is provided. RAID storage is accomplished through a series of mapped drives, non-routable Internet protocol (IP) addresses, and routable IP addresses. In addition, authorization to access a RAID controller, network address translation (NAT) system, and domain name system (DNS) system may all be separated, increasing security and allowing storage to be securely distributed among a variety of dispersed storage locations.

The invention relates to the technical field of computer application, and discloses a mass file data stream storage method, a mass file data stream storage system and a big data processing system, a rotation state conversion model is deployed between a data stream and a storage node, and the storage pressure is dispersed through multi-node cache. The storage method comprises the following steps: presetting a file size demarcation point; inputting massive file data, and judging whether the size of the input file is smaller than or equal to the demarcation point one by one; file data smaller than or equal to the demarcation point is stored in a selected storage node through a rotation state conversion model; generating index information of all the small files, combining the index information to index files, and organizing data in a B + tree form for each index file to obtain an index data bucket; a plurality of index data buckets are stored in a storage node in a data block form, and all the index data buckets are stored as an index data bucket queue in sequence. The method can effectively solve the problems that the writing performance is reduced and the efficiency is low when a traditional distributed storage system faces massive files.

The application belongs to the technical field of aerospace electronic integrated circuits, and particularly relates to a satellite load FPGA configuration data error correction method and system resisting single event upsets, a double-layer error correction method of "bit-symbol" based on RS decoding operator error correction of a finite field algebra on the basis of a "two out of three" majority voting logic, which can correct single particle independent upsets and multiple copy common mode failures at the same time, and solve the traditional TMR error locking problem. Meanwhile, when configuration data is written into a non-volatile memory, a cross-sector spatial staggered dispersion storage mechanism is introduced, so that the symbols in the same RS code word are separated as much as possible in the physical position, so as to convert the aggregation physical damage into dispersed symbol level errors.

The invention discloses a data storage method and system, electronic equipment and a storage medium, and relates to the technical field of data storage.The whole data storage system is divided into a plurality of storage units, each storage unit comprises a buffer and a storage tree, newly-written data are firstly stored in the buffers, and then the newly-written data are stored in the storage tree; according to the method and the device, the buffer is divided into a plurality of storage sub-trees, the buffer is subsequently subjected to data down-brushing to brush the data down to the storage tree, so that batch writing of the data of the storage tree is realized, write amplification of the storage tree is reduced, and when the storage tree level is high, the high-level storage tree is divided into the plurality of storage sub-trees, so that scattered storage of the data is realized, the situation that the storage tree level is too high is avoided, and the storage efficiency is improved. The data access time delay of the storage tree is reduced, and a foundation is laid for improving the concurrent access amount and the access efficiency of the data.

The invention discloses an electric power engineering data centralized filing processing system and method based on cloud storage. The method comprises the following steps: carrying out digital formatting processing on all electric power engineering data through a scanner; uploading the digitized electric power engineering data to a cloud storage platform based on a distributed storage technology; analyzing relevance among the data to obtain a relevance matrix; constructing a hash space of distributed storage based on a hash ring; based on the obtained relevance matrix, data are stored in a hash ring in a scattered mode through a distributed hash algorithm according to relevance; and carrying out dynamic archiving management on the electric power engineering data based on load balancing storage. The method has the advantages that high-relevance data are stored in a scattered mode by analyzing data relevance, data safety is improved, efficient scattered storage and dynamic management of electric power engineering data are achieved through the distributed hash algorithm and the load balancing technology, and data storage reliability and system operation efficiency are improved.

A processing system operates by: detecting an access anomaly associated with an access request from a requestor for a set of encoded data slices, the access anomaly having an unfavorable access pattern, wherein the set of encoded data slices is dispersed storage error encoded and stored in at least one storage unit of the storage network; denying the access request in response to detecting the access anomaly; generating, based on the unfavorable access pattern, an anomaly detection indicator identifying the requestor; and sending the anomaly detection indicator to other devices of the storage network.

A computing device of a storage network includes a network interface, memory and a processing module operably coupled to the memory and the network interface, with the processing module configured to execute the operational instructions to receive a storage request for a data file and then determine a data type for the data file. Based on the data type, the computing device determines whether to partition the data file into variable-sized data blocks and in response to a determination to do so, partitions the datafile according to a partition function into a plurality of data blocks, encodes each data block according to a dispersed storage error coded function to produce a plurality of data fragments and facilitates storage of the plurality of data fragments in a plurality of storage network nodes.

A computing system is operable to decode a plurality of incoming streams of digitally encoded data packets to extract a plurality of sets of input data from the plurality of incoming streams of digitally encoded data. A plurality of encrypted user data is generated based on encrypting at least some of the plurality of sets of input data and each of the plurality of encrypted user data is stored via a plurality of different storage locations. The plurality of sets of input data are processed generate a plurality of automatically generated textual data. A plurality of further outgoing streams of digitally encoded data packets are generated and transmitted via processing the plurality of automatically generated textual data.

The invention discloses a data collaborative storage management method and system for credit investigation data, and relates to the technical field of storage systems. The data collaborative storage management method and system for the credit investigation data comprises the steps that S1, the credit investigation data are collected and preprocessed, the credit investigation data are stored, and a credit investigation database is constructed; s2, carrying out consistency risk analysis through the fragmented data and the write-in operation data; s3, carrying out write-in orderliness analysis in combination with the global write-in sequence, the sorting priority and the consistency risk data; s4, repairing the fragmented data, and carrying out consistency repairing verification analysis based on the fragmented data before and after repairing; and S5, according to the fragmentation state, operating snapshots and repairing log data, archiving credit investigation data and continuously completing risk management and control. The problems that data are stored in a scattered mode, dirty writing conflicts occur frequently and the fragment consistency guarantee capacity is insufficient in the high-concurrency writing environment of the multi-service system are solved.

The utility model relates to the field of storage devices, in particular to an art tool integrated storage device which comprises a storage box and a storage assembly. A storage assembly used for classified storage according to art tools is installed in the storage box. The storage assembly comprises a sealing plate, a paint frame, a storage cabinet and a storage plate. The drawing board, tools and pigment are integrally and intensively stored through the pigment frame, the storage cabinet and the storage plate, limited space is fully utilized through reasonable layout, space waste caused by scattered storage is avoided, the disordered situation is reduced, the number of objects needing to be carried is reduced, follow-up movable creation is facilitated, and the drawing board is convenient to use. Meanwhile, all art tools are stored in a classified mode, a user can conveniently manage and search needed articles, the use convenience is improved, the use device is integrally stored, good sealing performance and protectiveness can be provided for the tools and the drawing board, paint is prevented from drying up or being affected with damp, the drawing board and the tools are protected against damage, and the overall storage effect is improved.

A processing system of a storage network operates by receiving a write request associated with a data object; identifying dispersed storage error encoding parameters; and determining a number of available storage units. When the number of available storage units is determined to exceed a first number, the processing system operates by: determining a first write threshold number in accordance with the dispersed storage error encoding parameters; dispersed error encoding the data object to produce a first write threshold number of encoded data slices; and writing, to available storage units, the first write threshold number of encoded data slices. When the number of available storage units is determined to not exceed the first number, the processing system operates by: determining a second write threshold number in accordance with the dispersed storage error encoding parameters associated, wherein the second write threshold number is less than the first write threshold number; dispersed error encoding the data object to produce a second write threshold number of encoded data slices; and writing, to the available storage units, the second write threshold number of encoded data slices.

A method for execution in a storage network begins by performing a key derivation function on a password to produce a key and issuing a set of passwords based on the key to a set of storage units, where a data object is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of encoded data slices and the set of encoded data slices are stored in the set of storage units. The method continues, by receiving at least a decode threshold number of confidential information responses from the set of storage units and regenerating a set of keys for the confidential information responses. The method then continues by decrypting the confidential information responses using the set of keys to reproduce a set of encoded data slices and dispersed storage error decoding a decode threshold number of the set of encoded data slices to produce recovered data.