679 results about "Dispersed storage" patented technology

A digital data file storage system is disclosed in which original data files to be stored are dispersed using some form of information dispersal algorithm into a number of file “slices” or subsets in such a manner that the data in each file share is less usable or less recognizable or completely unusable or completely unrecognizable by itself except when combined with some or all of the other file shares. These file shares are stored on separate digital data storage devices as a way of increasing privacy and security. As dispersed file shares are being transferred to or stored on a grid of distributed storage locations, various grid resources may become non-operational or may operate below at a less than optimal level. When dispersed file shares are being written to a dispersed storage grid which not available, the grid clients designates the dispersed data shares that could not be written at that time on a Rebuild List. In addition when grid resources already storing dispersed data become non-available, a process within the dispersed storage grid designates the dispersed data shares that need to be recreated on the Rebuild List. At other points in time a separate process reads the set of Rebuild Lists used to create the corresponding dispersed data and stores that data on available grid resources.

Briefly, the present invention relates to an information dispersal sytem in which original data to be stored is separated into a number of data “slices” in such a manner that the data in each subset is less usable or less recognizable or completely unusable or completely unrecognizable by itself except when combined with some or all of the other data subsets. These data subsets are stored on separate storage devices as a way of increasing privacy and security. In accordance with an important aspect of the invention, a metadata management system stores and indexes user files across all of the storage nodes. A number of applications run on the servers supporting these storage nodes and are responsible for controlling the metadata. Metadata is the information about the data, the data slices or data subsets and the way in which these data subsets are dispersed among different storage nodes running over the network. As used herein, metadata includes data source names, their size, last modification date, authentication information etc. This information is required to keep track of dispersed data subsets among all the nodes in the system. Every time new data subsets are stored and old ones are removed from the storage nodes, the metadata is updated. In accordance with an important aspect of the invention, the metadata management system stores metadata for dispersed data where: The dispersed data is in several pieces. The metadata is in a separate dataspace from the dispersed data. Accordingly, the metadata management system is able to manage the metadata in a manner that is computationally efficient relative to known systems in order to enable broad use of the invention using the types of computers generally used by businesses, consumers and other organizations currently.

A method begins by a dispersed storage (DS) processing module selecting two or more correlated error messages of a plurality of error messages. The method continues with the DS processing module identifying two or more collection of records corresponding to the selected two or more correlated error messages such that a collection of records includes an event record including information regarding an event, a first record including information regarding a dispersed storage (DS) processing module processing an event request to produce a plurality of sub-event requests, and a plurality of records including information regarding a plurality of DS units processing the plurality of sub-event requests. The method continues with the DS processing module identifying a correlation cause of one or more errors corresponding to the two or more correlated error messages based on the two or more collections of records.

A rebuilder application operates on a dispersed data storage grid and rebuilds stored data segments that have been compromised in some manner. The rebuilder application actively scans for compromised data segments, and is also notified during partially failed writes to the dispersed data storage network, and during reads from the dispersed data storage network when a data slice is detected that is compromised. Records are created for compromised data segments, and put into a rebuild list, which the rebuilder application processes.

A method begins by a management unit of a distributed storage network (DSN) sending a software update notice to a plurality of dispersed storage (DS) units, wherein the plurality of DS units supports a plurality of digital storage vaults, and wherein a set of DS units of the plurality of DS units supports a digital storage vault of the plurality of digital storage vaults. The method continues with a DS processing module determining an update strategy for updating software of the plurality of DS units such that at least a decode threshold number of DS units of the set of DS units is continually available to service access requests to the digital storage vault and updating the software of at least some of the plurality of DS units in accordance with the update strategy.

An improved system is disclosed for ensuring the integrity of data stored on a dispersed data storage network. Checksums are used to ensure integrity of both data segments and data slices. Checksums appended to data slices are checked by receiving slice servers to ensure that no errors occurred during transmission. Slice servers also periodically recalculate checksums for stored data slices to ensure that data slices have not been corrupted during storage. Checksums appended to data segments are checked when data segments are read from the storage network.

A method begins by a processing module dispersed storage error encoding data to produce a set of encoded data slices and sending a set of write request messages to a set of dispersed storage (DS) units, wherein each of the set of write request messages includes an encoded data slice of the set of encoded data slices. The method continues with the processing module determining whether a pillar width number of favorable write response messages has been received within a write acknowledgement (ACK) time period. The method continues with the processing module executing a retry write process to at least one DS unit of the set of DS units from which a favorable write response message was not received during the write ACK time period when the pillar width number of favorable write response messages has not been received within the write ACK time period.

A method begins by a processing module receiving redundant array of independent disks (RAID) data and determining whether to store the RAID data in at least one of a RAID format and in a dispersed storage network (DSN) format. The method continues with the processing module converting at least a portion of the RAID data into at least one set of encoded data slices when the at least a portion of the RAID data is to be stored in the DSN format. The method continues with the processing module outputting the at least one set of encoded data slices to a DSN memory.

A dispersed storage (DS) processing module may include a gateway module operable to communicate data and / or corresponding information with a user device and may include an access module operable to segment outbound data of the data into one or more outbound data segments and aggregate one or more inbound data segments into inbound data of the data. The DS processing module may include a grid module operable to encode an outbound data segment of the one or more outbound data segments into a plurality of outbound encoded data slices and decode a plurality of inbound encoded data slices into an inbound data segment of the one or more inbound data segments. The DS processing module may include a storage module operable to output the plurality of outbound encoded data slices to a plurality of DS storage units and receive the plurality of inbound encoded data slices from the plurality of DS storage units.

A method begins by a processing module determining dispersed storage (DS) units that are storing a set of encoded data slices associated with a data segment and sending a revision level check request message to each of the DS units. The method continues with the processing module receiving revision level check response messages from at least some of the DS units within a time period to produce received revision level check response messages and determining concurrency of a revision level of the set of encoded data slices stored by the DS units based on the received revision level check response messages. The method continues with the processing module initiating a revision concurrency process when less than a threshold number of encoded data slices have concurrency of the revision level.

A method begins by a processing module determining whether one or more encoded data slices of a set of encoded data slices of a plurality of sets of encoded data slices are required encoded data slices to produce one or more required encoded data slices. When the one or more of the encoded data slices are required, the method continues with the processing module generating specific dispersed storage network (DSN) storage information for each of the one or more required encoded data slices, generating general DSN storage information for remaining encoded data slices of the set of encoded data slices, and sending the set of encoded data slices to a dispersed storage network in accordance with the specific DSN storage information and the general DSN storage information.

A method begins with a processing module identifying data for storage and sending a subscription request message regarding the data for storage. The method continues with the processing module receiving a subscription response message. The method continues with the processing module obtaining error coding dispersal storage function parameters when a dispersal method of the subscription response message indicates direct dispersal. The method continues with the processing module encoding at least a portion of the data in accordance with the error coding dispersal storage function parameters to produce a set of encoded data slices. The method continues with the processing module determining a plurality of storage locations and performing a dispersed storage write function to store the set of encoded data slices in the plurality of storage locations.

A dispersed storage system includes a plurality of storage units that each include a partial rebuild grid module. The partial rebuild grid module includes partial rebuilding functionality to reconstruct one of a plurality of encoded data slices wherein the plurality of encoded data slices are generated from a data segment based on an error encoding dispersal function. In the partial rebuilding process, a data slice is rebuilt by combining in any order slice partials generated from at least a threshold number T of the plurality of data slices.

A method begins by a processing module generating a deterministic key from data and encrypting the data using the deterministic key to produce encrypted data. The method continues with the processing module generating transformed data from the encrypted data and generating a masked key from the deterministic key and the transformed data. The method continues with the processing module combining the masked key and the encrypted data to produce a secure package.

A storage integrity system in a dispersed storage network scans an address range of data slices to identify errors in one of a plurality of encoded data slices, wherein the plurality of encoded data slices are generated from a data segment using an error encoding dispersal function. When the storage integrity system detects an error, it identifies one of the encoded data slices for rebuilding. The identified data slice is rebuilt in response to the type of error. For example, when the type of the error includes a temporary error, the storage integrity system waits a predetermined time period to determine whether the error still exists prior to rebuilding the identified data slice.

In a dispersed storage network, data objects are dispersed storage error encoded into pluralities of sets of encoded data slices that are each stored in corresponding sets of storage units. To recover a data object, a read threshold number of encoded data slices from each set of encoded data slices of the plurality of sets of encoded data slices are required. Upon determining that an update is available for the storage units, a dispersed storage managing unit determines an updating sequence pattern that ensures that while one or more storage units are taken off line for the update, a sufficient number of storage units remain on line such that at least the read threshold number of encoded data slices are available for each set of encoded data slices.

A method begins by a processing module receiving a file retrieval request for a file, wherein the file includes one or more data regions, and wherein a data region of the one or more data regions is divided into a plurality of data segments and stored as a plurality of sets of encoded data slices in a dispersed storage network (DSN) memory. The method continues with the processing module retrieving a segment allocation table (SAT), wherein a SAT entry of a plurality of SAT entries includes information regarding storing the data region in the DSN memory and a segmentation scheme regarding the dividing of the data region into the plurality of data segments. The method continues with the processing module identifying the plurality of sets of encoded data slices and retrieving at least a sufficient number of the plurality of sets of encoded data slices to regenerate the data region.

A method begins by a dispersed storage (DS) processing module encoding a data segment of multi-media content using a dispersed storage error coding function to produce a set of encoded data slices and partitioning the set of encoded data slices into a first sub-set of encoded data slices and a second sub-set of encoded data slices, wherein the first sub-set of encoded data slices include less than a decode threshold number of encoded data slices. The method continues with the DS processing module distributing the first sub-set of encoded data slices to a plurality of potential accessing devices and when accessing information from a device of the plurality of potential accessing devices is received, sending at least one of the encoded data slices of the second sub-set of encoded data slices to the device such that the device has the decode threshold number of encoded data slices.

A dispersed storage device manages a file system directory of a dispersed storage network by receiving a data object to be stored and a user file name of the data object, calculating a data compression function of the data object, creating a file identifier based on a result of the data compression function, creating a source name for the data object using the file identifier and linking the user file name to the source name in the file system directory.

A method begins by a processing module receiving a data retrieval request and determining a read threshold number of dispersed storage (DS). The method continues with the processing module sending read request messages to DS units and receiving encoded data slices to produce received encoded data slices. The method continues with the processing module determining an incremental number of encoded data slices based on the number of received encoded data slices, determining an incremental number of DS units, and sending a read request message to each of the incremental number of DS units when the number of received encoded data slices compares unfavorably to a decode threshold number. The method continues with the processing module dispersed storage error decoding the received encoded data slices to produce data when the number of received encoded data slices compares favorably to the decode threshold number of encoded data slices.

A method begins by a processing module sending list digest requests to a set of dispersed storage (DS) units. The method continues with the processing module receiving list digest responses from at least some of the set of DS units and determining whether an inconsistency exists between first and second list digest responses of the list digest responses. The method continues with the processing module requesting at least a portion of each of the slice name information lists from first and second DS units of the set of DS units and identifying a slice name information error associated with the inconsistency based on the at least a portion of each of the slices name information lists of the first and second DS units when the inconsistency exists between first and second list digest responses of the list digest responses.

A method begins by a processing module receiving a data storage request that includes data and determining dispersed storage error encoding parameters for dispersed error encoding storage of data. The method continues with the processing module dispersed storage error encoding the data in accordance with the dispersed storage error encoding parameters to produce a set of encoded data slices and establishing access control information for each encoded data slice of the set encoded data slices in accordance with the dispersed storage error encoding parameters to produce a set of access control information. The method continues with the processing module appending a corresponding one of the set of access control information to each of the set of encoded data slices to produce a set of appended slices and outputting the set of appended slices to a set of dispersed storage units.

A method for data transfer from a first computing device to a second computing device begins when the first computing device is paired with a transfer token module by the first computing device sending the data to the transfer token module. The method continues with the transfer token module encoding the data utilizing a dispersed storage error encoding function to produce a set of encoded data slices and sending the set of encoded data slices to a target destination. The method continues, when the second computing device is paired with the transfer token module, with the transfer token module retrieving the set of encoded data slices from the target destination and decoding the set of encoded data slices utilizing the dispersed storage error encoding function to recapture the data. The method continues with the second computing device storing the data.

The invention discloses a distributed storage oriented cloud storage security architecture and a data access method thereof. The distributed cloud storage oriented security architecture comprises a cloud storage server, a distributed storage manager, a wafer breaker and storage devices, wherein the distributed storage manager and the cloud storage server are respectively connected with a client-side; the distributed storage manager and the cloud storage server are connected by a fiber channel (FC) or a routing switch; the wafer breaker is imbedded in the distributed storage manager; the distributed storage manager is connected with the storage devices by the FC or Iscsi; and the storage devices are connected by a network attached storage (NAS) or a storage area network (SAN). The storage devices are located at different places and are redundant mutually, thus improving the fault-tolerant ability of the devices and the optimal storage use ratio; the wafer breaker is used for carrying out fragment on stored data, so that the data are fragmented into data fragments which can not be identified by other authentication systems, and the fragmented data have relative confidentiality and security in the process of network transmission and data storage; and the storage devices and the server are the devices with higher cost performance, thus reducing the cost.

A dispersed storage (DS) method begins by issuing a plurality of write commands to a plurality of DS storage units. The method continues by receiving a write acknowledgement from one of the plurality of DS storage units to produce a received write acknowledgement. The method continues by issuing a plurality of commit commands to the plurality of DS storage units when a write threshold number of the received write acknowledgements have been received. The method continues by receiving a commit acknowledgement from a DS storage unit of the plurality of DS storage units to produce a received commit acknowledgement. The method continues by issuing a plurality of finalize commands to the plurality of DS storage units when a write threshold number of the received commit acknowledgements have been received.

A method begins by a processing module of a dispersed storage network (DSN) receiving data in a redundant array of independent disks (RAID) format and converting from the RAID format to an original format of the data. The method continues with the processing module dispersed storage error encoding a data segment of the data in the original format to produce a set of encoded data slices, where a set of encoded data slices includes a decode threshold sub-set of encoded data slices and an error correcting sub-set of encoded data slices. The method continues with the processing module converting the decode threshold sub-set of encoded data slices into a RAID formatted data segment, storing the RAID formatted data segment in RAID memory, and storing at least the error correcting sub-set of encoded data slices in DSN memory.

A method begins with a processing module issuing a retrieval request, receiving secret shares of a set of secret shares to produce received secret shares, and receiving encoded data slices of a set of encoded data slices. The method continues with the processing module decoding the received secret shares to recapture a message authentication key when a threshold number of the secret shares is received. The method continues with the processing module identifying a received encoded data slice of the received encoded data slices having an authentication code associated therewith when a threshold number of the encoded data slices is received. The method continues with the processing module verifying the authentication code based on the message authentication key and the received encoded data slice. The method continues with the processing module decoding the received encoded data slices to recapture a data segment when the authentication code is verified.

A method begins by a processing module obtaining a unique retrieval matrix based on an identity of the playback device and sending a request for retrieval of a set of encoded broadcast data slices to a dispersed storage network (DSN) memory, wherein the request includes the unique retrieval matrix and identity of the set of encoded broadcast data slices. The method continues with the processing module receiving a subset of the set of encoded broadcast data slices from the DSN memory, wherein the subset of the set of encoded broadcast data slices is based on the unique retrieval matrix. The method continues with the processing module storing the subset of the sets of encoded broadcast data slices.