1151 results about "Data slice" patented technology

Briefly, the present invention relates to an information dispersal sytem in which original data to be stored is separated into a number of data “slices” in such a manner that the data in each subset is less usable or less recognizable or completely unusable or completely unrecognizable by itself except when combined with some or all of the other data subsets. These data subsets are stored on separate storage devices as a way of increasing privacy and security. In accordance with an important aspect of the invention, a metadata management system stores and indexes user files across all of the storage nodes. A number of applications run on the servers supporting these storage nodes and are responsible for controlling the metadata. Metadata is the information about the data, the data slices or data subsets and the way in which these data subsets are dispersed among different storage nodes running over the network. As used herein, metadata includes data source names, their size, last modification date, authentication information etc. This information is required to keep track of dispersed data subsets among all the nodes in the system. Every time new data subsets are stored and old ones are removed from the storage nodes, the metadata is updated. In accordance with an important aspect of the invention, the metadata management system stores metadata for dispersed data where: The dispersed data is in several pieces. The metadata is in a separate dataspace from the dispersed data. Accordingly, the metadata management system is able to manage the metadata in a manner that is computationally efficient relative to known systems in order to enable broad use of the invention using the types of computers generally used by businesses, consumers and other organizations currently.

An improved system for accessing data within a distributed data storage network (“DDSN”) is disclosed. In a system implementing the disclosed invention, traffic is routed to individual slice servers within the DDSN in accordance with objective criteria as well as user-defined policies. In accordance with one aspect of the disclosed invention, when a data segment is written to a DDSN, the segment is divided into multiple data slices, which are simultaneously transmitted to different slice servers. In accordance with another aspect of the disclosed invention, when a data segment is read from a DDSN, a list of slice servers, each containing a data slice that could be used to reconstruct the requested data segment, is assembled, and sorted in accordance with a preference rating assigned to each of the slice servers. Sufficient data slices to reconstruct the data segment are then read in accordance with the preference ranking of the slice servers.

A billing process is disclosed for a information dispersal system or digital data storage system. In one embodiment of the invention, the original data to be stored is separated into a number of data “slices” or shares in such a manner that the data in each subset is less usable or less recognizable or completely unusable or completely unrecognizable by itself except when combined with some or all of the other data subsets. These data subsets are stored on separate digital data storage devices as a way of increasing privacy and security. As dispersed file shares are being stored or removed from a grid of distributed storage locations, a set of metadata tables are created, separate from the dispersed file share storage, to maintain information about the original data size of each block, file or set of file shares dispersed on the grid. The original data size information in these separate metadata tables is used to determine usage information based upon the original file size even though the file has been dispersed onto a storage grid that contains file slices who size may not relate to the original file size and the file slices may have been compressed by the system in order to reduce storage space or improve transmission time. As such, the billing process is able to enable a broad range of commercial billing options for billing for commercial data services on an information dispersal grid.

A rebuilder application operates on a dispersed data storage grid and rebuilds stored data segments that have been compromised in some manner. The rebuilder application actively scans for compromised data segments, and is also notified during partially failed writes to the dispersed data storage network, and during reads from the dispersed data storage network when a data slice is detected that is compromised. Records are created for compromised data segments, and put into a rebuild list, which the rebuilder application processes.

An improved data networking technique and apparatus using a novel physically distributed but logically shared and data-sliced synchronized shared memory switching datapath architecture integrated with a novel distributed data control path architecture to provide ideal output-buffered switching of data in networking systems, such as routers and switches, to support the increasing port densities and line rates with maximized network utilization and with per flow bit-rate latency and jitter guarantees, all while maintaining optimal throughput and quality of services under all data traffic scenarios, and with features of scalability in terms of number of data queues, ports and line rates, particularly for requirements ranging from network edge routers to the core of the network, thereby to eliminate both the need for the complication of centralized control for gathering system-wide information and for processing the same for egress traffic management functions and the need for a centralized scheduler, and eliminating also the need for buffering other than in the actual shared memory itself,—all with complete non-blocking data switching between ingress and egress ports, under all circumstances and scenarios.

A distributed data storage method, apparatus, and system for reducing a data loss that may result from a single-point failure. The method includes: splitting a data file to generate K data slices, splitting each data slice of the K data slices to generate M data blocks for each data slice, and performing check coding on the M data blocks by using a redundancy algorithm to generate N check blocks; determining, by using a random algorithm, a first physical storage node corresponding to one block of the M data blocks and the N check blocks, and determining at least M+1 different physical storage nodes based on the determined first physical storage node and according to a first rule-based sorting manner; and storing at least M+1 blocks of the M data blocks and the N check blocks onto the at least M+1 different storage nodes, where K, M, and N are integers.

An improved system is disclosed for ensuring the integrity of data stored on a dispersed data storage network. Checksums are used to ensure integrity of both data segments and data slices. Checksums appended to data slices are checked by receiving slice servers to ensure that no errors occurred during transmission. Slice servers also periodically recalculate checksums for stored data slices to ensure that data slices have not been corrupted during storage. Checksums appended to data segments are checked when data segments are read from the storage network.

A method begins by a processing module dispersed storage error encoding data to produce a set of encoded data slices and sending a set of write request messages to a set of dispersed storage (DS) units, wherein each of the set of write request messages includes an encoded data slice of the set of encoded data slices. The method continues with the processing module determining whether a pillar width number of favorable write response messages has been received within a write acknowledgement (ACK) time period. The method continues with the processing module executing a retry write process to at least one DS unit of the set of DS units from which a favorable write response message was not received during the write ACK time period when the pillar width number of favorable write response messages has not been received within the write ACK time period.

A method begins by a processing module receiving redundant array of independent disks (RAID) data and determining whether to store the RAID data in at least one of a RAID format and in a dispersed storage network (DSN) format. The method continues with the processing module converting at least a portion of the RAID data into at least one set of encoded data slices when the at least a portion of the RAID data is to be stored in the DSN format. The method continues with the processing module outputting the at least one set of encoded data slices to a DSN memory.

A distributed storage integrity system in a dispersed storage network includes a scanning agent and a control unit. The scanning agent identifies an encoded data slice that requires rebuilding, wherein the encoded data slice is one of a plurality of encoded data slices generated from a data segment using an error encoding dispersal function. The control unit retrieves at least a number T of encoded data slices needed to reconstruct the data segment based on the error encoding dispersal function. The control unit is operable to reconstruct the data segment from at least the number T of the encoded data slices and generate a rebuilt encoded data slice from the reconstructed data segment. The scanning agent is located in a storage unit and the control unit is located in the storage unit or in a storage integrity processing unit, a dispersed storage processing unit or a dispersed storage managing unit.

A display control apparatus displays image data pieces based on an array in which the image data pieces are sorted in accordance with a specified rule. The display control apparatus extracts, with use of one image data piece from the array as a reference, image data pieces at a set interval from the image data pieces arranged in the array, and arranges and displays the extracted image data pieces that include the reference image data piece on a screen of a display unit based on an order of the array. If the reference image data piece or the interval used in the extraction of the image data pieces has been changed based on an instruction from a user, the display on the display unit is changed with use of the new reference and interval.

A dispersed storage (DS) processing module may include a gateway module operable to communicate data and / or corresponding information with a user device and may include an access module operable to segment outbound data of the data into one or more outbound data segments and aggregate one or more inbound data segments into inbound data of the data. The DS processing module may include a grid module operable to encode an outbound data segment of the one or more outbound data segments into a plurality of outbound encoded data slices and decode a plurality of inbound encoded data slices into an inbound data segment of the one or more inbound data segments. The DS processing module may include a storage module operable to output the plurality of outbound encoded data slices to a plurality of DS storage units and receive the plurality of inbound encoded data slices from the plurality of DS storage units.

A method begins by a processing module determining dispersed storage (DS) units that are storing a set of encoded data slices associated with a data segment and sending a revision level check request message to each of the DS units. The method continues with the processing module receiving revision level check response messages from at least some of the DS units within a time period to produce received revision level check response messages and determining concurrency of a revision level of the set of encoded data slices stored by the DS units based on the received revision level check response messages. The method continues with the processing module initiating a revision concurrency process when less than a threshold number of encoded data slices have concurrency of the revision level.

A method begins by a processing module determining whether one or more encoded data slices of a set of encoded data slices of a plurality of sets of encoded data slices are required encoded data slices to produce one or more required encoded data slices. When the one or more of the encoded data slices are required, the method continues with the processing module generating specific dispersed storage network (DSN) storage information for each of the one or more required encoded data slices, generating general DSN storage information for remaining encoded data slices of the set of encoded data slices, and sending the set of encoded data slices to a dispersed storage network in accordance with the specific DSN storage information and the general DSN storage information.

A method begins with a processing module identifying data for storage and sending a subscription request message regarding the data for storage. The method continues with the processing module receiving a subscription response message. The method continues with the processing module obtaining error coding dispersal storage function parameters when a dispersal method of the subscription response message indicates direct dispersal. The method continues with the processing module encoding at least a portion of the data in accordance with the error coding dispersal storage function parameters to produce a set of encoded data slices. The method continues with the processing module determining a plurality of storage locations and performing a dispersed storage write function to store the set of encoded data slices in the plurality of storage locations.

A dispersed storage system includes a plurality of storage units that each include a partial rebuild grid module. The partial rebuild grid module includes partial rebuilding functionality to reconstruct one of a plurality of encoded data slices wherein the plurality of encoded data slices are generated from a data segment based on an error encoding dispersal function. In the partial rebuilding process, a data slice is rebuilt by combining in any order slice partials generated from at least a threshold number T of the plurality of data slices.

A distributed storage network includes a distributed storage processing unit, which generates multiple data slices from a single data segment, and multiple different storage units to store the data slices. Each data slice is assigned to a particular storage unit for storage. The distributed storage processing unit determines which of the storage units are in a ready state, and sends data slices to the ready storage units. Data slices assigned to storage units that are not in a ready state are temporarily stored in a temporary memory, e.g. cache memory, until the storage unit becomes ready. If the intended storage unit remains in a non-ready status for too long, the slices stored in cache memory can be sent to a substitute storage unit, or discarded and reconstructed at a later time.

An automated data management system and method for logging, processing, and reporting a large volume of data having different file types, using different versions, stored on different media, and / or run by different operating systems, includes a first processor for restoring a plurality of received data files, the data files being capable of being different file types; a file organizing / categorizing processor for organizing the received data files into data slices, each data slice including an identification number and a descriptor that describes characteristics of the received data file; a file logging processor for logging the received data files into a first database based on the data slices; a data uploading processor for uploading the first database to a second database; a de-duplicate processor for calculating a SHA value of the received data files to determine whether the received data files have duplicates and flagging duplicated data files in the second database; an image conversion processor for converting at least a portion of the received data files into image files; and a second processor for exporting the image files.

A method begins by a router receiving data for storage and interpreting the data to determine whether the data is to be forwarded or error encoded. The method continues with the router obtaining a routing table when the data is to be error encoded. Next, the method continues with the router selecting a routing option from the plurality of routing options and determining error coding dispersal storage function parameters based on the routing option. Next, the method continues with the router encoding the data based on the error coding dispersal storage function parameters to produce a plurality of sets of encoded data slices. Next, the method continues with the router outputting at least some of the encoded data slices of a set of the plurality of sets of encoded data slices to an entry point of the routing option.

A method of securely storing data to a dispersed data storage system is disclosed. A data segment is arranged along the columns or rows of an appropriately sized matrix. Data slices are then created based on either the columns or the rows so that no consecutive data is stored in a data slice. Each data slice is then stored in a separate storage node.

A storage integrity system in a dispersed storage network scans an address range of data slices to identify errors in one of a plurality of encoded data slices, wherein the plurality of encoded data slices are generated from a data segment using an error encoding dispersal function. When the storage integrity system detects an error, it identifies one of the encoded data slices for rebuilding. The identified data slice is rebuilt in response to the type of error. For example, when the type of the error includes a temporary error, the storage integrity system waits a predetermined time period to determine whether the error still exists prior to rebuilding the identified data slice.

In a dispersed storage network, data objects are dispersed storage error encoded into pluralities of sets of encoded data slices that are each stored in corresponding sets of storage units. To recover a data object, a read threshold number of encoded data slices from each set of encoded data slices of the plurality of sets of encoded data slices are required. Upon determining that an update is available for the storage units, a dispersed storage managing unit determines an updating sequence pattern that ensures that while one or more storage units are taken off line for the update, a sufficient number of storage units remain on line such that at least the read threshold number of encoded data slices are available for each set of encoded data slices.

A method begins by a processing module receiving a file retrieval request for a file, wherein the file includes one or more data regions, and wherein a data region of the one or more data regions is divided into a plurality of data segments and stored as a plurality of sets of encoded data slices in a dispersed storage network (DSN) memory. The method continues with the processing module retrieving a segment allocation table (SAT), wherein a SAT entry of a plurality of SAT entries includes information regarding storing the data region in the DSN memory and a segmentation scheme regarding the dividing of the data region into the plurality of data segments. The method continues with the processing module identifying the plurality of sets of encoded data slices and retrieving at least a sufficient number of the plurality of sets of encoded data slices to regenerate the data region.

A method begins by a distributed storage (DS) processing module identifying encoded data slices of stored encoded data slices to transfer, wherein the stored encoded data slices are assigned addresses within a local distributed storage network (DSN) address range, wherein a global DSN address space is divided into a plurality of address sectors, and wherein the local DSN address range is a portion of an address sector. The method continues with the DS processing module determining whether another local DSN address range in the address sector exists and when the other local DSN address range in the address sector exists, determining whether to transfer identified encoded data slices into the other local DSN address range. When the at least some of the identified encoded data slices are to be transferred, the method continues with the DS processing module initiating a data transfer protocol to transfer the identified encoded data slices.

A method begins by a dispersed storage (DS) processing module encoding a data segment of multi-media content using a dispersed storage error coding function to produce a set of encoded data slices and partitioning the set of encoded data slices into a first sub-set of encoded data slices and a second sub-set of encoded data slices, wherein the first sub-set of encoded data slices include less than a decode threshold number of encoded data slices. The method continues with the DS processing module distributing the first sub-set of encoded data slices to a plurality of potential accessing devices and when accessing information from a device of the plurality of potential accessing devices is received, sending at least one of the encoded data slices of the second sub-set of encoded data slices to the device such that the device has the decode threshold number of encoded data slices.

A method begins by a processing module receiving a data retrieval request and determining a read threshold number of dispersed storage (DS). The method continues with the processing module sending read request messages to DS units and receiving encoded data slices to produce received encoded data slices. The method continues with the processing module determining an incremental number of encoded data slices based on the number of received encoded data slices, determining an incremental number of DS units, and sending a read request message to each of the incremental number of DS units when the number of received encoded data slices compares unfavorably to a decode threshold number. The method continues with the processing module dispersed storage error decoding the received encoded data slices to produce data when the number of received encoded data slices compares favorably to the decode threshold number of encoded data slices.

A distributed storage network generates a plurality of data segments from a data object and stores each of the plurality of data segments as a plurality of encoded data slices generated from an error encoding dispersal function. When the distributed storage network receives a modification request for the data object, it determines a size of the plurality of data segments of the data object from a segment size field and identifies one of the plurality of data segments requiring modification. The identified data segment is reconstructed from the plurality of encoded data slices and modified in accordance with the modification request.