40 results about "Dispersed storage" patented technology

Systems and methods for decentralized data storage and retrieval within a distributed computing system include partitioning datasets into encrypted data shards that are each associated with a unique identifier. A processing system operating within the network may distribute these shards across storage nodes selected based on performance metrics such as latency, storage utilization, and reliability. Metadata for each shard, including its identifier, unique cryptographic key, and node location, may be recorded in a tamper-evident distributed ledger. Upon receiving a retrieval request, the system may query the ledger to identify shard locations, retrieve the shards in parallel, validate their integrity by comparing ledger metadata with metadata from storage nodes, decrypt the shards using their respective cryptographic keys, and reconstruct the dataset. These systems and methods enhance data security, integrity, and fault tolerance while allowing efficient and verifiable decentralized storage and retrieval in dynamic, distributed environments.

The invention discloses a data backup and recovery method, which relates to the technical field of data backup and recovery, and comprises the following steps: S1, pre-analyzing data to be backed up; s2, connecting a storage medium; s3, data transmission backup; s4, data backup recording; and S5, data recovery. According to the data backup recovery method, a plurality of independent storage media are adopted, only one medium is connected with a computer at the same time, the media are physically isolated, data are not intercommunicated, even if a certain medium is damaged, backup data of other media are still intact, and a single-point fault is avoided; the physical movement requirement is reduced to prevent the storage medium from being lost, all data blocks of a single file can be ensured to be completely stored in the same medium by analyzing the file structure and the incidence relation in advance, it is ensured that the data logic relation can be completely restored during concentrated storage and recovery of related files, and the incidence failure caused by scattered storage is avoided.

The invention relates to the field of distributed storage, and discloses a distributed memory system based on erasure codes and a dynamic repair mechanism. Comprising a data node cluster, a metadata management cluster, a bandwidth token management module, a dynamic repair composer and a repair agent node. And the data node cluster divides the object into strip generation data fragments and verification fragments and dispersedly stores the data fragments and the verification fragments according to a placement rule. And the metadata management cluster maintains object stripe mapping, a fragment position set, erasure code protection configuration and a version epoch, and records a repair intention log. And the bandwidth token management module grants repair execution permission according to the fault domain budget. The dynamic repair composer generates availability repair and durability repair tasks, the temporary check fragments are written in when the number of the available check fragments is lower than a threshold value in the availability stage, and the configuration fragments are recovered and the temporary fragments are recovered in the durability stage. And the consistency of version epochs is verified before the agent node is repaired and written back, and if not, recalculation is carried out.

The invention belongs to the technical field of spaceflight electronic integrated circuits, and particularly relates to a satellite load FPGA configuration data error correction method and system capable of resisting single event upset, in particular to a bit-symbol double-layer error correction method of RS decoding operator error correction based on finite field algebra cascading on the basis of two-out-of-three majority voting logic. Single-particle independent upset and multi-copy common-mode failure can be corrected at the same time, and the problem of traditional TMR error locking is solved. Meanwhile, when configuration data are written into a nonvolatile memory, a cross-sector space staggered and dispersed storage mechanism is introduced, symbols in the same RS code word are separated as far as possible in physical positions, and therefore aggregated physical damage is converted into dispersed symbol-level errors.

A computing device for use in a storage network includes one or more network interfaces, memory with operational instructions and a processing module. The processing module is configured to execute the operational instructions receive a storage request for data and select a plurality of memory devices for storage of the data. The processing module partitions the data into a plurality of chunk sets and encode each chunk set of the plurality of chunk sets according to a dispersed storage error coded function to produce a plurality of chunks. Finally, the processing module facilitates storage of each chunk of the plurality of chunks in a memory device of the plurality of memory devices.

The invention discloses an HDFS (Hadoop Distributed File System) intelligent hierarchical storage method and system based on multi-dimensional popularity perception and tape library integration, and belongs to the technical field of distributed file systems, and the method comprises the following steps: expanding HDFS metadata to construct a unified namespace; acquiring data block access features and dynamically dividing hot, warm and cold data by using a K-means clustering algorithm; executing a self-adaptive grading strategy: adding a copy to hot data and storing in an SSD (Solid State Disk), maintaining a default copy to warm data and storing in a disk, and migrating cold data to a tape library and storing by adopting an erasure code; scattered storage and fault tolerance of relevance perception are realized through a message queue; metadata persistence and periodical re-evaluation are carried out, and automatic migration and dynamic adjustment of data among the SSD layer, the disk layer and the tape library layer are achieved. By adopting the method and the system, the storage strategy and the data access value are accurately matched, the storage cost is finally reduced, and the system I / O performance and the resource utilization rate are improved.

A method for a storage network computing device includes receiving a request to store a data object and generating a plurality of write slice requests, where the plurality of write slice requests includes a set of encoded data slices to be stored in the storage network and at least a decode threshold number of encoded data slices of the set of encoded data slices is required to recover a corresponding portion of a data object. The method continues by identifying a set of storage devices (from a plurality) for storage of the set of encoded data slices, where the plurality of storage devices includes one or more first memory devices local to the computing device and one or more second memory devices that are remote from the computing device. The method then continues, by storing the set of encoded data slices in the identified set of storage devices, generating a set of write slice responses relating to the set of encoded data slices and outputting the set of write slice responses.

A processing system of a storage network operates by receiving a write request associated with a data object; identifying dispersed storage error encoding parameters; and determining a number of available storage units. When the number of available storage units is determined to exceed a first number, the processing system operates by: determining a first write threshold number in accordance with the dispersed storage error encoding parameters; dispersed error encoding the data object to produce a first write threshold number of encoded data slices; and writing, to available storage units, the first write threshold number of encoded data slices. When the number of available storage units is determined to not exceed the first number, the processing system operates by: determining a second write threshold number in accordance with the dispersed storage error encoding parameters associated, wherein the second write threshold number is less than the first write threshold number; dispersed error encoding the data object to produce a second write threshold number of encoded data slices; and writing, to the available storage units, the second write threshold number of encoded data slices.

This invention relates to the field of distributed storage technology and provides a data collaboration method and system for distributed transaction terminals. The method involves each transaction terminal performing erasure coding on its local ledger data to generate a set of data fragments consisting of original data fragments and verification data fragments. These data fragments are then distributed and stored across different transaction terminals according to distributed storage rules, and the terminals exchange online status and fragment integrity information. When a request to access the target ledger is received, the terminal storing its fragments is located based on the ledger identifier, and the fragments are retrieved in parallel to reconstruct the target ledger. When a terminal is detected to be offline or a fragment is corrupted, online terminals collaborate to reconstruct the original data using surviving fragments, re-encode new fragments, and distribute them to healthy terminals. This invention, through erasure coding and distributed collaborative storage, reduces storage overhead while improving system fault tolerance and data availability, achieving efficient and reliable distributed ledger data collaborative management.

The invention discloses a data storage method and a related device, and the method comprises the steps: firstly obtaining factory data which needs to be stored through a TLC NAND technology, then compressing the factory data to obtain compressed data, dividing the compressed data into a plurality of strips, then generating a corresponding XOR check code for each strip, and storing the XOR check code in a database; and finally, the compressed data and the XOR check code corresponding to each strip are written into different physical units, so that the factory data are compressed and then put into a disk, the written data volume can be reduced, the abrasion of the embedded storage equipment can be relieved, the service life can be prolonged, and the storage efficiency can be improved by utilizing the XOR check code. The XOR check code and the survivor data are stored in different physical units in a dispersed manner, so that data recovery and reconstruction can be performed through the XOR check code and the survivor data when part of data in the strip is lost, the data integrity requirement of the embedded storage device for sudden power failure is met, the persistent retention capability of data storage is further improved, and the data storage efficiency is improved. And the reliability of data storage is improved.

The application relates to the technical field of computer application, and discloses a mass file data stream storage method, a storage system and a big data processing system. A rotation state conversion model is arranged between a data stream and a storage node, and storage pressure is dispersed through multi-node caching. The storage method comprises the following steps: presetting a file size demarcation point; inputting mass file data; judging whether the size of the input file is smaller than or equal to the demarcation point; storing file data smaller than or equal to the demarcation point in a selected storage node through the rotation state conversion model; generating index information of all small files, and merging the index information into an index file; organizing data in the form of a B+ tree for each index file to obtain an index data bucket; storing a plurality of index data buckets in the form of data blocks in the storage node; and saving all index data buckets in sequence as an index data bucket queue. The application can effectively solve the problems of traditional distributed storage systems, such as write performance decline and low efficiency when facing mass files.

One or more techniques and / or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and / or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and / or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and / or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and / or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items.

The invention relates to the technical field of computer application, and discloses a mass file data stream storage method, a mass file data stream storage system and a big data processing system, a rotation state conversion model is deployed between a data stream and a storage node, and the storage pressure is dispersed through multi-node cache. The storage method comprises the following steps: presetting a file size demarcation point; inputting massive file data, and judging whether the size of the input file is smaller than or equal to the demarcation point one by one; file data smaller than or equal to the demarcation point is stored in a selected storage node through a rotation state conversion model; generating index information of all the small files, combining the index information to index files, and organizing data in a B + tree form for each index file to obtain an index data bucket; a plurality of index data buckets are stored in a storage node in a data block form, and all the index data buckets are stored as an index data bucket queue in sequence. The method can effectively solve the problems that the writing performance is reduced and the efficiency is low when a traditional distributed storage system faces massive files.

The application belongs to the technical field of aerospace electronic integrated circuits, and particularly relates to a satellite load FPGA configuration data error correction method and system resisting single event upsets, a double-layer error correction method of "bit-symbol" based on RS decoding operator error correction of a finite field algebra on the basis of a "two out of three" majority voting logic, which can correct single particle independent upsets and multiple copy common mode failures at the same time, and solve the traditional TMR error locking problem. Meanwhile, when configuration data is written into a non-volatile memory, a cross-sector spatial staggered dispersion storage mechanism is introduced, so that the symbols in the same RS code word are separated as much as possible in the physical position, so as to convert the aggregation physical damage into dispersed symbol level errors.

A computing device of a storage network includes a network interface, memory and a processing module operably coupled to the memory and the network interface, with the processing module configured to execute the operational instructions to receive a storage request for a data file and then determine a data type for the data file. Based on the data type, the computing device determines whether to partition the data file into variable-sized data blocks and in response to a determination to do so, partitions the datafile according to a partition function into a plurality of data blocks, encodes each data block according to a dispersed storage error coded function to produce a plurality of data fragments and facilitates storage of the plurality of data fragments in a plurality of storage network nodes.

A computing system is operable to decode a plurality of incoming streams of digitally encoded data packets to extract a plurality of sets of input data from the plurality of incoming streams of digitally encoded data. A plurality of encrypted user data is generated based on encrypting at least some of the plurality of sets of input data and each of the plurality of encrypted user data is stored via a plurality of different storage locations. The plurality of sets of input data are processed generate a plurality of automatically generated textual data. A plurality of further outgoing streams of digitally encoded data packets are generated and transmitted via processing the plurality of automatically generated textual data.

The invention discloses a data collaborative storage management method and system for credit investigation data, and relates to the technical field of storage systems. The data collaborative storage management method and system for the credit investigation data comprises the steps that S1, the credit investigation data are collected and preprocessed, the credit investigation data are stored, and a credit investigation database is constructed; s2, carrying out consistency risk analysis through the fragmented data and the write-in operation data; s3, carrying out write-in orderliness analysis in combination with the global write-in sequence, the sorting priority and the consistency risk data; s4, repairing the fragmented data, and carrying out consistency repairing verification analysis based on the fragmented data before and after repairing; and S5, according to the fragmentation state, operating snapshots and repairing log data, archiving credit investigation data and continuously completing risk management and control. The problems that data are stored in a scattered mode, dirty writing conflicts occur frequently and the fragment consistency guarantee capacity is insufficient in the high-concurrency writing environment of the multi-service system are solved.

The utility model relates to the field of storage devices, in particular to an art tool integrated storage device which comprises a storage box and a storage assembly. A storage assembly used for classified storage according to art tools is installed in the storage box. The storage assembly comprises a sealing plate, a paint frame, a storage cabinet and a storage plate. The drawing board, tools and pigment are integrally and intensively stored through the pigment frame, the storage cabinet and the storage plate, limited space is fully utilized through reasonable layout, space waste caused by scattered storage is avoided, the disordered situation is reduced, the number of objects needing to be carried is reduced, follow-up movable creation is facilitated, and the drawing board is convenient to use. Meanwhile, all art tools are stored in a classified mode, a user can conveniently manage and search needed articles, the use convenience is improved, the use device is integrally stored, good sealing performance and protectiveness can be provided for the tools and the drawing board, paint is prevented from drying up or being affected with damp, the drawing board and the tools are protected against damage, and the overall storage effect is improved.

One or more techniques and / or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and / or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and / or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and / or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and / or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items.

A method for execution by a computing device of a storage network includes appending at least a decode threshold number of encoded key slices of a set of encoded key slices to at least some encrypted data segments of a plurality of encrypted data segments to produce secure packages. The method further includes error encoding, in accordance with error encoding parameters, the secure packages to produce sets of encoded data slices, where a first secure package of the secure packages is dispersed storage error encoded using an error encoding function of the error encoding parameters to produce a first set of encoded data slices of the sets of encoded data slices. The method further includes outputting the sets of encoded data slices for storage in memory of the storage network.

This invention provides a method for transmitting railway Building Information Modeling (BIM) model files based on distributed storage. The method includes: lightweight decomposing the railway BIM model file into multiple sub-files; distributing these sub-files across edge server nodes using a cloud server; transmitting the sub-file list to a gateway; constructing a BIMdex index model; training the BIMdex index model using a training dataset; deploying the trained BIMdex index model locally on the gateway; and when an end device sends a file request carrying a file identifier to the gateway, the BIMdex index model in the gateway queries the sub-file list based on the file identifier, obtains the corresponding BIMdex address index, and returns it to the end device. This invention significantly improves data transmission efficiency by distributing sub-files of the same railway BIM model across multiple physical nodes and combining this with load balancing technology.

The invention relates to the technical field of civil aviation data security, in particular to a data security protection method and system for a civil aviation data intermediate station, and the method comprises the steps: obtaining first data; dividing the high-secrecy-level data into N data sub-blocks according to data types, and storing the N data sub-blocks in N storage modules in a dispersed manner; generating a confusion variable by adopting a random number generation algorithm, decomposing the confusion variable into N confusion sub-blocks, and storing the N confusion sub-blocks in N storage modules; destroying the confusion variables; generating a limited access strategy through a relevance blocking method; and when the formatting instruction is received, backing up the confusion sub-blocks stored in the storage module which receives the formatting instruction to the storage module which does not receive the formatting instruction, and then formatting. According to the method, the high-secrecy-level data is stored in a distributed manner, and the confusion sub-blocks are generated to confuse the relevance feature data, so that the data security is improved on the premise that the data value of the relevance feature data is brought into full play.

A data file store 100 stores multiple collections of files 102 provided by a user 101 and containing information specific to that user. The data file store 100 comprises a plurality of data blocks each comprising an encrypted file from the collection of files and a corresponding file identifier. Each encrypted file is encrypted using a separate key pair 108 comprising a private key 109 and a public key 110. The file identifier for each encrypted file includes an element from the encrypted file’s public key 110. The data blocks are scattered in different locations in the data file store and individual files 102 are retrievable by using the element of the public key 110 to determine to which of the collection of files the public key corresponds. The element of the public key 110 may comprise an alphanumeric string or a prime number. Each block may include an encrypted file expiry indicator identifying how long the file can be accessed. Blocks may be scattered. The collection of files may include a case identifier, which may be encrypted and stored within each block of the collection. Each block may contain a hash of all other blocks within the collection.

This application relates to a configuration file distribution method, apparatus, computer device, readable storage medium, and program product, falling under the field of network security technology. A connector in a second network generates an encrypted configuration file request based on a connector identifier. The configuration center determines the connector identifier and identity information based on the configuration file request, retrieves the target configuration file corresponding to the identity information from multiple configuration files, and configures the connector based on the target configuration file to handle financial transactions between the financial institution and platform to which the configuration center belongs. Compared to traditional data transmission over the public network, this solution, through a configuration center located in the first network, centrally manages the configuration files of connectors corresponding to various platforms. By authenticating the connector requesting the configuration file, it determines whether to distribute the corresponding configuration file, reducing the risk of arbitrary tampering due to dispersed storage of configuration files and improving the security of configuration file distribution.

A method for execution by one or more computing devices of a storage network includes dispersed storage error encoding a data segment of data into a set of encoded data slices in accordance with an information dispersal algorithm width number and a decode threshold number, where the decode threshold number of encoded data slices of the set of encoded data slices are needed to reconstruct the data segment. The method further includes storing the set of encoded data slices in primary storage slots of a set of storage units deployed across multiple sites of the storage network in accordance with a mapping of the primary storage slots to storage units of the set of storage units, where a number of the primary storage slots is equal to or greater than the information dispersal algorithm width number.

The application relates to the technical field of data storage, in particular to a multi-engine intelligent data warehouse storage model construction method and system, which solves the technical problem that, in the prior art, a sharding random dispersion storage strategy is adopted for unstructured data, resulting in a large input / output overhead and high access cost in a data access calling process. The method comprises the following steps: constructing a distributed storage architecture comprising a plurality of storage nodes; performing sharding processing on unstructured data to be stored to obtain a plurality of data shards and acquire data features of each data shard; comparing the similarity of the data features between each data shard and historical data to evaluate the access granularity of each data shard; establishing an intelligent matching rule between the data shards and the storage nodes according to the access granularity of each data shard and the node resources of the storage nodes; and storing each data shard to a corresponding storage node according to the intelligent matching rule to form a multi-engine intelligent data warehouse storage model.

A computing device includes an interface configured to interface and communicate with a dispersed storage network (DSN), a memory that stores operational instructions, and a processing module operably coupled to the interface and memory such that the processing module, when operable within the computing device based on the operational instructions, is configured to perform various operations. The computing device selects a subset of the other computing devices to perform a computing task on a data object. The computing device determines processing parameters of the data and determines task partitioning. The computing device also processes the data based on processing parameters to generate data slice groupings and partitions the task based on the task partitioning to generate partial tasks. The computing device obtains and processes at least the decode threshold number of the plurality of partial results generated by the subset of the other computing devices to generate a result.

A method includes determining rebuilding performance parameter values for a storage unit of a set of storage units, where data is dispersed storage error encoded into a plurality of sets of encoded data slices in accordance with error encoding parameters, and a set of encoded data slices are stored or are to be stored in the set of storage units. The method includes updating a rate of internal rebuilding of encoded data slices by the storage unit based on the rebuilding performance parameter values. During internally rebuilding, the method includes correcting detected storage errors for one or more first encoded data slices within the storage unit. The method includes receiving a rebuilt encoded data slice from another computing device of the storage network and when the received rebuilt encoded data slice is not included in the one or more first encoded data slices, storing the received rebuilt encoded data slice.

Methods and apparatus for maintaining availability of critical information in a storage network that includes a plurality of storage units. In an embodiment, a processing module(s) of the storage network identifies critical information stored in one or more memory devices of a computing device. Critical information can include, for example, information required for operation of one or more storage units to perform data access operations. The processing module obtains the critical information from the one or more memory devices, and dispersed storage error encodes the critical information to produce one or more sets of encoded critical slices. The processing module further selects a set of storage units of the storage network and facilitates storage of the one or more sets of encoded critical slices in the selected set of storage units.

Methods and apparatus for preference based selection of storage network memory for data storage. In an example, a computing device receives a data object for storage in memory of the storage network and determines a system level storage efficiency preference associated with the data object. The computing device selects a set of storage nodes of a plurality of sets of storage nodes for storage of the data object based, at least in part, on the system level storage efficiency preference. The computing device further determines dispersed storage error encoding parameters for the data object, encodes the data object in accordance with the dispersed storage error encoding parameters to produce encoded data slices, and generates system addressing information for the encoded data slices.