6124 results about "Data chunk" patented technology

The invention provides a method for organizing a writing operation to a nonvolatile memory. The method comprises setting a specific cache area, into which a specific data belonging to a specific group of logical blocks is to be written. It is determined whether or not the writing operation is a random write. If the writing operation is the random write, then the following steps are performed: determining whether or not the writing operation is to write a data that is belonging to the specific group of logical blocks; and writing the data into the specific cache area if the data is belonging to the specific group of logical blocks. As a result, a swap action between a data block and a writing block can be avoided during a random write operation. A storage structure in a nonvolatile memory device are organized to perform the forgoing writing operation.

Described herein are systems and methods for multiplexing pipelined data for backup operations. Various data streams are combined such as by multiplexing by a multiplexing module. The multiplexing module combines the data from the various data streams received by receiver module(s) into a single stream of chunks. The multiplexing module may combine data from multiple archive files into a single chunk. Additional modules perform other operations on the chunks of data to be transported such as encryption, compression, etc. The data chunks are transmitted via a transport channel to a receive pipeline that includes a second receiver module and other modules. The data chunks are then stored in a backup medium. The chunks are later retrieved and separated such as by demultiplexing for restoring to a client or for further storage as auxiliary copies of the separated data streams or archive files.

Network data files are secure through the operation of an infrastructure gateway-based network file access appliance. Network file data, corresponding to network pocket payload data, are further reduced to a sequence of data blocks that are secured through any combination of block encryption, compression, and digital signatures. File meta-data, including encryption, compression and block-level digital signatures are persistently stored with the file data, either in-band in the file as stored or out-of-band key as a separately stored file or file policy record. File meta-data is recovered with accesses of the file data to support bidirectional encryption and compression and to detect tampering with the file data by comparison against block-level digital signatures.

A method for taking snapshots of data. In an embodiment, a first map data structure is obtained that records locations for a plurality of data blocks. A second map data structure is formed that is initially empty of locations, the second map data structure representing the snapshot after the snapshot is started. Writes on data that arrive before the snapshot starts are applied to the first map, and writes that arrive after the snapshot starts are applied to the second map.

Read-only and read-write snapshot copies of a production file in a Unix-based file system are organized as a version set of file inodes and shared file blocks. Version pointers and branch pointers link the inodes. Initially the production file can have all its blocks preallocated or it can be a sparse file having only an inode and its last data block. A protocol is provided for creating read-only and read-write snapshots, deleting snapshots, restoring the production file with a specified snapshot, refreshing a specified snapshot, and naming the snapshots. Block pointers are marked with a flag indicating whether or not the pointed-to block is owned by the parent inode. A non-owner marking is inherited by all of the block's descendants. The block ownership controls the copying of indirect blocks when writing to the production file, and also controls deallocation and passing of blocks when deleting a read-only snapshot.

A system for de-duplicating data includes providing a first volume including at least one pointer to a second volume that corresponds to physical storage space, wherein the first volume is a logical volume. A first set of data is detected as a duplicate of a second set of data stored on the second volume at a first data chunk. A pointer of the first volume associated with the first set of data is modified to point to the first data chunk. After modifying the pointer, no additional physical storage space is allocated for the first set of data.

The present invention provides for a method, system, and computer program for the application of data deduplication according to a policy-based strategy of tiered data. The method operates by defining a plurality of data storage policies for data in a deduplication system, policies which may be arranged in tiers. Data objects are classified according to a selected data storage policy and are split into data chunks. If the selected data storage policy for the data object does not allow deduplication, the data chunks are stored in a deduplication pool. If the selected data storage policy for the data object allows deduplication, deduplication is performed. The data storage policy may specify a maximum number of references to data chunks, facilitating storage of new copies of the data chunks when the maximum number of references is met.

A redundant array of independent disk (RAID) memory storage system comprising data storage blocks arranged in a first plurality of data rows and a second plurality of data columns, wherein parity data is stored in additionally defined parity blocks, and wherein numbers of data blocks in respective columns are different, to accommodate the additional diagonal parity data block that the geometry of the system requires. The system is suitable for an SSD array in which sequential disk readout is not required.

A tile-based arrangement is employed to effect watermarking of plural-bit auxiliary data into still or moving digital images. In a particular embodiment, the plural-bit auxiliary data is represented as a rectangular block of overlay data. This block is repetitively tiled, both vertically and horizontally, across the image, and combined with the image data to effect the encoding.

An efficient method to apply an erasure encoding and decoding scheme across dispersed data stores that receive constant updates. A data store is a persistent memory for storing a data block. Such data stores include, without limitation, a group of disks, a group of disk arrays, or the like. An encoding process applies a sequencing method to assign a sequence number to each data and checksum block as they are modified and updated onto their data stores. The method preferably uses the sequence number to identify data set consistency. The sequencing method allows for self-healing of each individual data store, and it maintains data consistency and correctness within a data block and among a group of data blocks. The inventive technique can be applied on many forms of distributed persistent data stores to provide failure resiliency and to maintain data consistency and correctness.

A method and system for streaming software applications to a client uses an application server having a library with the application files stored therein. A streaming manager is configured to send the application files to a client as a plurality of streamlets, each streamlet corresponding to a particular data block in a respective application file. A streaming prediction engine is provided to identify at least one streamlet which is predicted to be most appropriate to send to a given client at a particular time in accordance with a prediction model reflecting the manner in which the application files are loaded and used by the application. In the preferred implementation, the application files are preprocessed and stored as a set of compressed streamlets, each of which corresponds to a file data block having a size equal to a code page size, such as 4k, used during file reads by an operating system expected to be present on a client system. In addition, the server is configured to send a startup block to a new streaming client containing a file structure specification of the application files and a set of streamlets comprising at least those streamlets containing the portions of the application required to enable execution of the application to be initiated.

A logging system includes an event receiver and a storage manager. The receiver receives log data, processes it, and outputs a data “chunk.” The manager receives data chunks and stores them so that they can be queried. The receiver includes buffers that store events and a metadata structure that stores metadata about the contents of the buffers. The metadata includes a unique identifier associated with the receiver, the number of events in the buffers, and, for each “field of interest,” a minimum value and a maximum value that reflect the range of values of that field over all of the events in the buffers. A chunk includes the metadata structure and a compressed version of the contents of the buffers. The metadata structure acts as a search index when querying event data. The logging system can be used in conjunction with a security information/event management (SIEM) system.

In an asynchronous remote copy system, a primary site keeps a list of data blocks for which changes have been made to a primary file system and transmitted to a secondary file system at a secondary site. When the primary site becomes inoperative, the secondary site begins read/write access to the secondary file system and keeps a snapshot copy of the restart point. Once the primary site becomes operative, the primary file system is restored to the state of the secondary file system at the restart point by using the list of data blocks for obtaining from the snapshot copy the data of the blocks for which changes had been made to the primary file system after the restart point. Then the primary file system is synchronized to the secondary file system, and read/write access is switched back from the secondary file system to the primary file system.

Allocation process that allows erasure coded data to be stored on any of a plurality of disk drives, in a pool of drives, so that the allocation is not tied to a fixed group of drives. Still further, the encoded data can be generated by any of multiple different erasure coding algorithms, where again storage of the encoded data is not restricted to a single group of drives based on the erasure algorithm being utilized to encode the data. In another embodiment, the encoded data can be “stacked” (aligned) on select drives to reduce the number of head seeks required to access the data. As a result of these improvements, the system can dynamically determine which one of multiple erasure coding algorithms to utilize for a given incoming data block, without being tied to one particular algorithm and one particular group of storage devices as in the prior art.

A data chunking system divides data into predominantly fixed-sized chunks such that duplicate data may be identified. The data chunking system may be used to reduce the data storage and save network bandwidth by allowing storage or transmission of primarily unique data chunks. The system may also be used to increase reliability in data storage and network transmission, by allowing an error affecting a data chunk to be repaired with an identified duplicate chunk. The data chunking system chunks data by selecting a chunk of fixed size, then moving a window along the data until a match to existing data is found. As the window moves across the data, unique chunks predominantly of fixed size are formed in the data passed over. Several embodiments provide alternate methods of determining whether a selected chunk matches existing data and methods by which the window is moved through the data. To locate duplicate data, the data chunking system remembers data by computing a mathematical function of a data chunk and inserting the computed value into a hash table.

The invention discloses an asset transaction method and a system. The method comprises steps that an acquired first transaction request is signed by a consignment agency, the first transaction request after signature is sent to the block chain network established on the basis of the P2P network, the first transaction response data is constructed by an issuing mechanism on the basis of the first transaction request acquired from the block chain network, the first transaction response data is sent to the block chain network so that validity of the first transaction response data can be confirmed by a block chain node on the block chain network responsible for accounting, the first effective transaction response data is written in a block chain data block, the block chain data block is acquired by the consignment agency through the block chain network, and asset transaction with the issuing mechanism and the consignment agency is accomplished. Through the method, transaction steps between the issuing mechanism and the consignment agency can be reduced in a transaction process, risks of being tampered of the data can be reduced, asset transaction confirmation cost is reduced, and operation cost of the block chain network is further reduced.

A flash memory system, including a flash memory device and a controller, and having improved efficiency error correction coding (ECC), is disclosed. Each page in the flash memory device has the capacity to store multiple sectors' worth of data. However, partial page programming (i.e., followed by a later write to fill the page) is prohibited for reliability reasons. A scratchpad block within the flash memory device is designed, and stores both user data and control data. ECC efficiency is improved by encoding the ECC, or parity, bits over the entire data block corresponding to the user and control data in the page. Retrieval of a particular sector of data requires reading and decoding of the entire page. Especially for codes such as Reed-Solomon and BCH codes, the larger data block including multiple sectors' data improves the error correction capability, and thus enables either fewer redundant memory cells in each page or improved error correction.

In a data processing system, a first processor pre-allocates data blocks for use in a file system at a later time when a second processor needs data blocks for extending the file system. The second processor selectively maps the logical addresses of the pre-allocated blocks so that when the pre-allocated blocks are used in the file system, the layout of the file system on disk is improved to avoid block scatter and enhance I/O performance. The selected mapping can be done at a program layer between a conventional file system manager and a conventional logical volume layer so that there is no need to modify the data block mapping mechanism of the file system manager or the logical volume layer. The data blocks can be pre-allocated adaptively in accordance with the allocation history of the file system.

In accordance with embodiments of the invention, a method is provided for performing a storage operation in a pipeline storage system in which one or more data streams containing data to be stored are written into data chunks. The method includes generating an encryption key associated with a first archive file to be stored when encryption is requested for the storage operation, encrypting the archive data from the data stream using the encryption key to create an encrypted data chunk when a data stream containing the archive file is processed in the pipeline storage system, storing the encrypted data chunk on a storage medium, and storing the encryption key in a manner accessible during a restore operation of the encrypted data chunk.