6407 results about "Data chunk" patented technology

Described herein are systems and methods for multiplexing pipelined data for backup operations. Various data streams are combined such as by multiplexing by a multiplexing module. The multiplexing module combines the data from the various data streams received by receiver module(s) into a single stream of chunks. The multiplexing module may combine data from multiple archive files into a single chunk. Additional modules perform other operations on the chunks of data to be transported such as encryption, compression, etc. The data chunks are transmitted via a transport channel to a receive pipeline that includes a second receiver module and other modules. The data chunks are then stored in a backup medium. The chunks are later retrieved and separated such as by demultiplexing for restoring to a client or for further storage as auxiliary copies of the separated data streams or archive files.

A write interface in a file server provides permission management for concurrent access to data blocks of a file, ensures correct use and update of indirect blocks in a tree of the file, preallocates file blocks when the file is extended, solves access conflicts for concurrent reads and writes to the same block, and permits the use of pipelined processors. For example, a write operation includes obtaining a per file allocation mutex (mutually exclusive lock), preallocating a metadata block, releasing the allocation mutex, issuing an asynchronous write request for writing to the file, waiting for the asynchronous write request to complete, obtaining the allocation mutex, committing the preallocated metadata block, and releasing the allocation mutex. Since no locks are held during the writing of data to the on-disk storage and this data write takes the majority of the time, the method enhances concurrency while maintaining data integrity.

For enhanced access control, a client includes a token in each read or write command sent to a block storage device. The block storage device evaluates the token to determine whether or not read or write access is permitted at a specified logical block address. For example, the token is included in the logical block address field of a SCSI read or write command. The client may compute the token as a function of the logical block address of a data block to be accessed, or a metadata server may include the token in each block address of each extent reported to the client in response to a metadata request. For enhanced security, the token also is a function of a client identifier, a logical unit number, and access rights of the client to a particular extent of file system data blocks.

The invention provides a method for organizing a writing operation to a nonvolatile memory. The method comprises setting a specific cache area, into which a specific data belonging to a specific group of logical blocks is to be written. It is determined whether or not the writing operation is a random write. If the writing operation is the random write, then the following steps are performed: determining whether or not the writing operation is to write a data that is belonging to the specific group of logical blocks; and writing the data into the specific cache area if the data is belonging to the specific group of logical blocks. As a result, a swap action between a data block and a writing block can be avoided during a random write operation. A storage structure in a nonvolatile memory device are organized to perform the forgoing writing operation.

A wireless device, system and method for receiving and playing multimedia files streamed from a multimedia server over a wireless telecommunications network. A desired multimedia file is selected from one or more multimedia files stored in the multimedia server, which server is operatively connected to the wireless telecommunications network. Successive blocks of data from the desired multimedia file are streamed over the wireless telecommunications network in a digitized and compressed format and received by the wireless telecommunications device. The received blocks of data from the streamed multimedia file are temporarily stored in a buffer in the wireless device, decoded and decompressed, and successively played through an audio and / or video output in the wireless device. The wireless device monitors the blocks of data stored in the buffer and signals the wireless telecommunications network to increase the rate that the data blocks are transmitted over the wireless network in the event that the data stored in the buffer falls below a minimum threshold level.

Described herein are systems and methods for multiplexing pipelined data for backup operations. Various data streams are combined such as by multiplexing by a multiplexing module. The multiplexing module combines the data from the various data streams received by receiver module(s) into a single stream of chunks. The multiplexing module may combine data from multiple archive files into a single chunk. Additional modules perform other operations on the chunks of data to be transported such as encryption, compression, etc. The data chunks are transmitted via a transport channel to a receive pipeline that includes a second receiver module and other modules. The data chunks are then stored in a backup medium. The chunks are later retrieved and separated such as by demultiplexing for restoring to a client or for further storage as auxiliary copies of the separated data streams or archive files.

Network data files are secure through the operation of an infrastructure gateway-based network file access appliance. Network file data, corresponding to network pocket payload data, are further reduced to a sequence of data blocks that are secured through any combination of block encryption, compression, and digital signatures. File meta-data, including encryption, compression and block-level digital signatures are persistently stored with the file data, either in-band in the file as stored or out-of-band key as a separately stored file or file policy record. File meta-data is recovered with accesses of the file data to support bidirectional encryption and compression and to detect tampering with the file data by comparison against block-level digital signatures.

A method for taking snapshots of data. In an embodiment, a first map data structure is obtained that records locations for a plurality of data blocks. A second map data structure is formed that is initially empty of locations, the second map data structure representing the snapshot after the snapshot is started. Writes on data that arrive before the snapshot starts are applied to the first map, and writes that arrive after the snapshot starts are applied to the second map.

Read-only and read-write snapshot copies of a production file in a Unix-based file system are organized as a version set of file inodes and shared file blocks. Version pointers and branch pointers link the inodes. Initially the production file can have all its blocks preallocated or it can be a sparse file having only an inode and its last data block. A protocol is provided for creating read-only and read-write snapshots, deleting snapshots, restoring the production file with a specified snapshot, refreshing a specified snapshot, and naming the snapshots. Block pointers are marked with a flag indicating whether or not the pointed-to block is owned by the parent inode. A non-owner marking is inherited by all of the block's descendants. The block ownership controls the copying of indirect blocks when writing to the production file, and also controls deallocation and passing of blocks when deleting a read-only snapshot.

A system for de-duplicating data includes providing a first volume including at least one pointer to a second volume that corresponds to physical storage space, wherein the first volume is a logical volume. A first set of data is detected as a duplicate of a second set of data stored on the second volume at a first data chunk. A pointer of the first volume associated with the first set of data is modified to point to the first data chunk. After modifying the pointer, no additional physical storage space is allocated for the first set of data.

Described is using flash memory, RAM-based data structures and mechanisms to provide a flash store for caching data items (e.g., key-value pairs) in flash pages. A RAM-based index maps data items to flash pages, and a RAM-based write buffer maintains data items to be written to the flash store, e.g., when a full page can be written. A recycle mechanism makes used pages in the flash store available by destaging a data item to a hard disk or reinserting it into the write buffer, based on its access pattern. The flash store may be used in a data deduplication system, in which the data items comprise chunk-identifier, metadata pairs, in which each chunk-identifier corresponds to a hash of a chunk of data that indicates. The RAM and flash are accessed with the chunk-identifier (e.g., as a key) to determine whether a chunk is a new chunk or a duplicate.

The present invention provides for an enhanced method and system for assuring integrity of deduplicated data objects stored within a storage system. A digital signature of the data object is generated to determine if the data object reassembled from a deduplicated state is identical to its pre-deduplication state. In one embodiment, generating the object signature of a data object before deduplication comprises generating an object signature from intermediate hash values computed from a hash function operating on each data chunk within the data object, the hash function also used to determine duplicate data chunks. In an alternative embodiment, generating the object signature of a data object before deduplication comprises generating an object signature on a portion of each data chunk of the data object.

The present invention provides for a method, system, and computer program for the application of data deduplication according to a policy-based strategy of tiered data. The method operates by defining a plurality of data storage policies for data in a deduplication system, policies which may be arranged in tiers. Data objects are classified according to a selected data storage policy and are split into data chunks. If the selected data storage policy for the data object does not allow deduplication, the data chunks are stored in a deduplication pool. If the selected data storage policy for the data object allows deduplication, deduplication is performed. The data storage policy may specify a maximum number of references to data chunks, facilitating storage of new copies of the data chunks when the maximum number of references is met.

A redundant array of independent disk (RAID) memory storage system comprising data storage blocks arranged in a first plurality of data rows and a second plurality of data columns, wherein parity data is stored in additionally defined parity blocks, and wherein numbers of data blocks in respective columns are different, to accommodate the additional diagonal parity data block that the geometry of the system requires. The system is suitable for an SSD array in which sequential disk readout is not required.

The aim is to provide high-speed data synchronization. To achieve the aim, in data synchronization using a plurality of key databases with respect to same data pieces, a key for one key database, which has been determined in advance, is used for updating the data piece managed under the other key database. This reduces the number of key decryption operations. A key management software 116, which manages a key database A120 and a key database B121 each having a tree structure, determines whether to perform data synchronization when requested by an upper-level application to perform data encryption, and performs synchronization of encrypted data by using a key of the other database which has been determined in advance. This reduces the number of times the encrypted key is loaded onto a cryptographic processing unit 114, and realizes high-speed cryptographic processing on data.

A data backup system for backing up a data file from a source device having a source processor and an input device to a target device having a target processor and a database over a communication network. The source processor determines whether the data file has been modified, and if so, to determines the modified data block(s). The modified data block(s) are transmitted from the source processor to the target processor over the communication network. The target processor receives the modified data block(s) from the source device over the communications network, and uses the modified data blocks to construct the data file, and to store the data file on the database. Concurrently, it is determined if the input device is active at any time during the backup process, and if so, the backup processing is interrupted until the input device has been inactive for a predetermined period of time.

A tile-based arrangement is employed to effect watermarking of plural-bit auxiliary data into still or moving digital images. In a particular embodiment, the plural-bit auxiliary data is represented as a rectangular block of overlay data. This block is repetitively tiled, both vertically and horizontally, across the image, and combined with the image data to effect the encoding.

The present invention provides a method and apparatus of managing a storage array. The method comprises: striping the storage array to form a plurality of stripes; selecting F storage chunks from each stripe as local parity chunks, and selecting another L storage chunks from the storage array as global parity chunks; performing (F+L) fault tolerant erasure coding on all data chunks in a stripe to generate (F+L) groups of parity data, and storing F groups of parity data therein into the F local parity chunks; performing cross-stripe operation on another L groups of parity data to generate L groups of global parity data, and storing them into the L global parity chunks, respectively. The apparatus corresponds to the method. With the invention, a plurality of errors in the storage array can be detected and / or recovered to improve fault tolerance and space utilization of the storage array.

An efficient method to apply an erasure encoding and decoding scheme across dispersed data stores that receive constant updates. A data store is a persistent memory for storing a data block. Such data stores include, without limitation, a group of disks, a group of disk arrays, or the like. An encoding process applies a sequencing method to assign a sequence number to each data and checksum block as they are modified and updated onto their data stores. The method preferably uses the sequence number to identify data set consistency. The sequencing method allows for self-healing of each individual data store, and it maintains data consistency and correctness within a data block and among a group of data blocks. The inventive technique can be applied on many forms of distributed persistent data stores to provide failure resiliency and to maintain data consistency and correctness.

Allocation process that allows erasure coded data to be stored on any of a plurality of disk drives, in a pool of drives, so that the allocation is not tied to a fixed group of drives. Still further, the encoded data can be generated by any of multiple different erasure coding algorithms, where again storage of the encoded data is not restricted to a single group of drives based on the erasure algorithm being utilized to encode the data. In another embodiment, the encoded data can be “stacked” (aligned) on select drives to reduce the number of head seeks required to access the data. As a result of these improvements, the system can dynamically determine which one of multiple erasure coding algorithms to utilize for a given incoming data block, without being tied to one particular algorithm and one particular group of storage devices as in the prior art.

Systems and methods for providing fast and efficient data compression using a combination of content independent data compression and content dependent data compression. In one aspect, a method for compressing data comprises the steps of: analyzing a data block of an input data stream to identify a data type of the data block, the input data stream comprising a plurality of disparate data types; performing content dependent data compression on the data block, if the data type of the data block is identified; performing content independent data compression on the data block, if the data type of the data block is not identified.

A method and system for streaming software applications to a client uses an application server having a library with the application files stored therein. A streaming manager is configured to send the application files to a client as a plurality of streamlets, each streamlet corresponding to a particular data block in a respective application file. A streaming prediction engine is provided to identify at least one streamlet which is predicted to be most appropriate to send to a given client at a particular time in accordance with a prediction model reflecting the manner in which the application files are loaded and used by the application. In the preferred implementation, the application files are preprocessed and stored as a set of compressed streamlets, each of which corresponds to a file data block having a size equal to a code page size, such as 4k, used during file reads by an operating system expected to be present on a client system. In addition, the server is configured to send a startup block to a new streaming client containing a file structure specification of the application files and a set of streamlets comprising at least those streamlets containing the portions of the application required to enable execution of the application to be initiated.

In an asynchronous remote copy system, a primary site keeps a list of data blocks for which changes have been made to a primary file system and transmitted to a secondary file system at a secondary site. When the primary site becomes inoperative, the secondary site begins read / write access to the secondary file system and keeps a snapshot copy of the restart point. Once the primary site becomes operative, the primary file system is restored to the state of the secondary file system at the restart point by using the list of data blocks for obtaining from the snapshot copy the data of the blocks for which changes had been made to the primary file system after the restart point. Then the primary file system is synchronized to the secondary file system, and read / write access is switched back from the secondary file system to the primary file system.

A logging system includes an event receiver and a storage manager. The receiver receives log data, processes it, and outputs a data “chunk.” The manager receives data chunks and stores them so that they can be queried. The receiver includes buffers that store events and a metadata structure that stores metadata about the contents of the buffers. The metadata includes a unique identifier associated with the receiver, the number of events in the buffers, and, for each “field of interest,” a minimum value and a maximum value that reflect the range of values of that field over all of the events in the buffers. A chunk includes the metadata structure and a compressed version of the contents of the buffers. The metadata structure acts as a search index when querying event data. The logging system can be used in conjunction with a security information / event management (SIEM) system.

A data stream, such as a digital motion picture, is encrypted in units of one or more blocks, each block having an assigned encryption key. A plurality of encryption keys is assigned to the complete data stream, with a synchronization index provided to map each individual encryption key to its starting data block. Encryption keys and associated synchronization indices are provided separately from the data stream, using one or more additional data transfer mechanisms. An optional offset, randomly generated, allows variation in intervals between data blocks at which encryption by a specific encryption key can be performed.

In an asynchronous remote copy system, a primary site keeps a list of data blocks for which changes have been made to a primary file system and transmitted to a secondary file system at a secondary site. When the primary site becomes inoperative, the secondary site begins read / write access to the secondary file system and keeps a snapshot copy of the restart point. Once the primary site becomes operative, the primary file system is restored to the state of the secondary file system at the restart point by using the list of data blocks for obtaining from the snapshot copy the data of the blocks for which changes had been made to the primary file system after the restart point. Then the primary file system is synchronized to the secondary file system, and read / write access is switched back from the secondary file system to the primary file system.

Allocation process that allows erasure coded data to be stored on any of a plurality of disk drives, in a pool of drives, so that the allocation is not tied to a fixed group of drives. Still further, the encoded data can be generated by any of multiple different erasure coding algorithms, where again storage of the encoded data is not restricted to a single group of drives based on the erasure algorithm being utilized to encode the data. In another embodiment, the encoded data can be “stacked” (aligned) on select drives to reduce the number of head seeks required to access the data. As a result of these improvements, the system can dynamically determine which one of multiple erasure coding algorithms to utilize for a given incoming data block, without being tied to one particular algorithm and one particular group of storage devices as in the prior art.

A data chunking system divides data into predominantly fixed-sized chunks such that duplicate data may be identified. The data chunking system may be used to reduce the data storage and save network bandwidth by allowing storage or transmission of primarily unique data chunks. The system may also be used to increase reliability in data storage and network transmission, by allowing an error affecting a data chunk to be repaired with an identified duplicate chunk. The data chunking system chunks data by selecting a chunk of fixed size, then moving a window along the data until a match to existing data is found. As the window moves across the data, unique chunks predominantly of fixed size are formed in the data passed over. Several embodiments provide alternate methods of determining whether a selected chunk matches existing data and methods by which the window is moved through the data. To locate duplicate data, the data chunking system remembers data by computing a mathematical function of a data chunk and inserting the computed value into a hash table.

The invention discloses an asset transaction method and a system. The method comprises steps that an acquired first transaction request is signed by a consignment agency, the first transaction request after signature is sent to the block chain network established on the basis of the P2P network, the first transaction response data is constructed by an issuing mechanism on the basis of the first transaction request acquired from the block chain network, the first transaction response data is sent to the block chain network so that validity of the first transaction response data can be confirmed by a block chain node on the block chain network responsible for accounting, the first effective transaction response data is written in a block chain data block, the block chain data block is acquired by the consignment agency through the block chain network, and asset transaction with the issuing mechanism and the consignment agency is accomplished. Through the method, transaction steps between the issuing mechanism and the consignment agency can be reduced in a transaction process, risks of being tampered of the data can be reduced, asset transaction confirmation cost is reduced, and operation cost of the block chain network is further reduced.

A flash memory system, including a flash memory device and a controller, and having improved efficiency error correction coding (ECC), is disclosed. Each page in the flash memory device has the capacity to store multiple sectors' worth of data. However, partial page programming (i.e., followed by a later write to fill the page) is prohibited for reliability reasons. A scratchpad block within the flash memory device is designed, and stores both user data and control data. ECC efficiency is improved by encoding the ECC, or parity, bits over the entire data block corresponding to the user and control data in the page. Retrieval of a particular sector of data requires reading and decoding of the entire page. Especially for codes such as Reed-Solomon and BCH codes, the larger data block including multiple sectors' data improves the error correction capability, and thus enables either fewer redundant memory cells in each page or improved error correction.