944 results about "Shard" patented technology

A method of maintaining a solid-state drive so that free space within memory blocks of the drive becomes free usable space to the drive. The drive comprises cells organized in pages that are organized in memory blocks in which at least user files are stored. A defragmentation utility is executed to cause at least some of the memory blocks that are partially filled with data and contain file fragments to be combined or aligned and to cause at least some of the memory blocks that contain only invalid data to be combined or aligned. A block consolidation utility is then executed to eliminate at least some of the partially-filled blocks by consolidating the file fragments into a fewer number of the memory blocks. The consolidation utility also increases the number of memory blocks that contain only invalid memory. All of the memory blocks containing only invalid data are then erased.

A method and system are disclosed for controlling the storage of data in a storage device to reduce fragmentation. The method may include a controller of a storage device receiving data for storage in non-volatile memory, proactively preventing fragmentation by only writing an amount of sequentially addressed logical groups of data into a main storage area of the storage device, such as multi-level cell (MLC) flash memory, and reactively defragmenting data previously written into the MLC memory when a trigger event is reached. The system may include a storage device with a controller configured to perform the method noted above, where the thresholds for minimum sequential writes into MLC, and for scanning the memory for fragmented data and removing fragmentation by re-writing the fragmented data already in MLC into new MLC blocks, may be fixed or variable.

A differential snapshot is established and maintained for a set of files stored on a volume. Copy-on-write operations are avoided for logically insignificant moves of blocks, such as the block rearrangements characteristic of defragmentation utilities. A file system passes a block copy command to lower-level drivers that are to inform the snapshotter that a block move operation is not logically meaningful. When the logically insignificant move is of a block whose data forms part of the data captured in the snapshot virtual volume, and when the move is to a block location that is functioning as logical free space, the snapshotter can simply modify its block bitmap and update translation table entries without needing to perform a copy-on-write.

Methods and systems are described for applying the use of shards within a single memory address space. A database request is processed by providing the request from a client to a processor, the processor then distributing the request to multiple threads within a single process but executing in a shared memory address environment, wherein each thread performs the request on a distinct shard, and aggregating the results of the multiple threads being aggregated and returning a final result to the client. By parallelizing operations in this way, the request response time can be reduced and the total amount of communication overhead can be reduced.

A method and system for creating, reading and writing compressed files for use with a file access storage. The compressed data of a raw file are packed into a plurality of compressed units and stored as compressed files. One or more corresponding compressed units may be read and / or updated with no need for restoring the entire file whilst maintaining de-fragmented structure of the compressed file.

Provided are a method, system, and program for managing data reception processing using offload engines which may be located on a network adaptor. Data packets which become fragmented after encryption can be forwarded to a transport offload engine to be reassembled. The reassembled packets may be fed back to a security offload engine to be decrypted. The decrypted and reassembled packets may be forwarded again to the transport offload engine to extract the data payloads of the packets.

A file shard store includes high performance encoding and compaction circuits. An apparatus and its method of operation avoids duplicate storage of file fragments. A plurality of tables control write operations into a plurality of log segments. Shard keys are transferred to uniquely identify fragments of files which may have been previously stored and associated with one or more of client subscribers. An apparatus comprises a plurality of location / subscription tables, a shard transfer circuit, a storage control circuit, an append-to-log circuit coupled to a large storage array, and a log segment compactor circuit.

A method and system for creating, reading and writing compressed data for use with a block mode access storage. The compressed data are packed into plurality of compressed units and stored in a storage logical unit (LU). One or more corresponding compressed units may be read and / or updated with no need of restoring the entire storage logical unit whilst maintaining de-fragmented structure of the LU.

A method and system for creating, reading and writing compressed files for use with a file access storage. The compressed data of a raw file are packed into a plurality of compressed units and stored as compressed files. One or more corresponding compressed units may be read and / or updated with no need for restoring the entire file whilst maintaining de-fragmented structure of the compressed file.

When a dynamic data structure is used for managing sparsely accessed data stored in memory of a digital computer, pages of the memory are dynamically allocated and de-allocated to respective portions of the dynamic data structure so that the pages become distributed over the portions of the dynamic data structure and the de-allocated pages include free pages that are mapped in the dynamic data structure and free pages that are not mapped in the dynamic data structure. To reduce memory fragmentation and recover memory, upon de-allocating a page of memory from a portion of the data structure, a determination is made whether or not to un-map the de-allocated page from the portion of the dynamic data structure so that un-mapping of the de-allocated page has a probability that is the inverse of a superlinear function of the number of allocated pages in the portion of the dynamic data structure.

Certain example embodiments relate to the concept of controlling the flow of data by providing an intelligent flow controller / manager, and a client-side component for the selection of a communication channel from a pool, and having these components communicate to regulate data flow through gateways to a broker- and / or other-type secondary stage. Data fragmentation and reassembly can be used to increase performance, e.g., through self-regulating behaviors. Advantageously, reliability is improved by enabling in-memory data persistence, rather than resorting to potentially performance-degrading use of disk storage. The delivery mechanism may be used to deliver data to multiple consumers, providing an end-to-end sender-to-consumer solution that self-regulates to optimize the data flow while still being reliable.

Provided are apparatus and method for distributedly storing an object in a plurality of data centers included in a cloud storage system. In the method, an object may be divided into a plurality of sub-objects each having a fragmentation size. Then, the sub-objects may be distributedly stored in at least one data center based on weights of the plurality of data centers.

Media files such as MPEG-4 files are fragmented to allow for media and live media creation and delivery. A MPEG-4 standard description box includes synchronization information, end of file information, and chapter information to provide signaling information for near live playback of fragments. Playback can begin upon receiving a first MPEG-4 file fragment. A second MPEG-4 file fragment can be requested using information included in the first MPEG-4 file fragment.

A defragmentation technique determines the extent to which data blocks of a file are fragmented on disks of a computer and, in response, efficiently relocates those blocks if such relocation improves the on-disk layout of the file. Each indirect block of the file is examined and the current layout of the range of pointers referencing the data blocks is determined. In addition, the number of operations needed to retrieve those data blocks from disks is calculated. A potential new layout is then estimated based on an average fullness of the file system. If the potential new layout improves the fragmentation of the current layout, then the data blocks for that range are relocated, if there is sufficient free space on disk. Otherwise, the blocks are not relocated and the current on-disk layout of the file is maintained.

The invention relates to a big data technology-based space data storage processing middleware framework. The framework can provide a method for rapidly obtaining data content from mixed data of existing multi-source heterogeneous structural data and non-structural data; and distributed storage efficiency is improved by adopting a mainstream big data access tool. The big data technology-based space data storage processing middleware framework disclosed by the invention performs extraction, conversion and loading on the multi-source heterogeneous space data to establish a diversified, fragmented and non-structural data distributed virtualized storage framework, so that directly readable data content is provided for subsequent space big data analysis and mining.

A defragmentation method and system to overcome fragmentation of virtual volumes in storage area networks (SANs). The method includes combining and migrating fragments of data spread over multiple storage areas in order to increase storage space utilization and optimize access time to storage. By defragmenting virtual volumes both performance and storage utilization is significantly improved.

A system and method for rasterizing and rendering graphics data is disclosed. Vertices may be grouped to form primitives such as triangles, which are rasterized using two-dimensional arrays of samples bins. To overcome fragmentation problems, the system's sample evaluation hardware may be configured to over-evaluate samples each clock cycle. Since a number of the samples will typically not survive evaluation because they will be outside the primitive being rendered, the remaining surviving samples may be combined into sets, with one set being forwarded to subsequent pipeline stages each clock cycle in order to attempt to keep the pipeline utilization high.

The invention provides a system combining intelligent knowledge diagnosing and teacher online tutoring. Basic information of a student is acquired first, required information is extracted from a massive resource library according to the information of the student so as to perform intelligent test paper composition, an intelligent diagnosing report is generated after the answers of the student are acquired, and learning resources corresponding to the weak knowledge links of the student are pushed to the student; the student can also transmits the intelligent diagnosing report to an online teacher for one-to-one communication, the teacher analyzes the diagnosing result again to increase the accuracy of the result, and the teacher can also push targeted resources to the student after the diagnosing. The method has the advantages that the convenience of the Internet is fully utilized, the student's time is utilized in a fragment manner, comprehensive intelligent knowledge diagnosing can be provided for the student fast, the student can find the weak knowledge links conveniently, and learning efficiency of the student is increased.

A method and system for analyzing the physical characteristics of database records, such as in IMS databases. The information obtained during this analysis may then be used to perform a more efficient reorganization or restructuring of the database. In one embodiment, the method involves tracing the database retrieval process to collect physical location information for each segment of each database record in an IMS database. The database retrieval process for each database record may begin at the root segment of the database record and traverse the child segments of the database record, preferably in hierarchical order, e.g. top to bottom, left to right. The database retrieval process identifies the segment code causing the first reference to a block and the number of segments retrieved from the block before fetching a new block. The physical location information for each segment of each database record in the IMS database may be analyzed to identify one or more segments which represent a fragmented boundary twin chain. Two calculations may be performed for those segments which are boundary twin chains. If the total number of physical blocks currently used to hold the boundary twin chain exceeds the minimum number of physical blocks needed to hold the boundary twin chain by a pre-determined amount, the boundary twin chain may be determined to be fragmented. Similar calculations may be performed for the database record. A reorganization recommendation list for the database record may be created in response to determining whether the database record is fragmented.

The invention discloses a chain log storage structure, a hash index structure and a data operation method thereof, a server and a medium. The chain log storage structure comprises a plurality of storage segments and an array; a storage unit of each storage segment is a log entry; the log entry includes an initial size field, an index position field, a key length field, a Key field and a Value field, wherein the Value field is used for storing data; the Value field is used for storing data keys; the initial size field is used for storing an initial size of the storage log entry; the index position field is used for storing the position of an index of the log entry in a hash table; and the array is used for storing meta information of each storage segment. The hash index structure comprises a plurality of hash buckets; and a data structure of each hash bucket includes an index number field, a valid mark field, a label field, an offset field and an extension bucket field. According to the chain log storage structure, the hash index structure and the data operation method thereof, the server and the medium, the close continuity of the whole structure memory can be maintained and the fragmentation rate is reduced.

The invention belongs to the technical field of network storage, and discloses a method for creating an allocation-on-demand incremental volume. The volume has a two-layer volume structure, and realizes separation of a physical space and a logical space. The creating method comprises that: a user state interface participates in a new parameter (N) in a user state command lvcreate of a logical volume manger (LVM). The real volume of the bottom layer is a common logical volume, and the virtual volume of the upper layer is a data structure in a Linux kernel. The physical space of the real volume is allocated according to the actual requirement, so the utilization rate of a disk space is improved. Real-time monitoring of the physical space of the real volume and automatic capacity expansion of the actual space effectively avoid overflow of the spaces. A user can randomly designate capacity expansion increment magnitude as required to perform capacity expansion of the virtual volume. The volume processes the write request by adopting a mode of additional write. Moreover, the volume processes garbage fragments by adopting a quick and effective garbage reclamation algorithm.

A nonvolatile storage device can read and write data by receiving a logical address from a host. The nonvolatile storage device includes: a nonvolatile memory writing and reading data based on a physical address; a logical / physical conversion table storing information on correspondence between the logical address and the physical address for each of a plurality of data management units; a duplication table storing information on correspondence between the logical address and the physical address of data arranged over a plurality of areas in a duplicate (redundant) manner in the nonvolatile memory and having a size smaller than a size of a data management unit; and a controller controlling an operation of the nonvolatile storage device. The controller permits duplicate (redundant) writing of data having a same logical address as the data (which has been written in one data management unit in another data management unit), and the controller performs a plurality of write modes for the duplicate (redundant) writing, and selectively switches between the write modes according to situation of a logical address change designated by the host.

The invention relates to the field of a distributed system, in particular to a resource scheduling technology in the distributed system. According to a task allocation method, a share of allocated resources of a user is obtained; a to-be-allocated task is selected from a to-be-allocated task list; based on a highest threshold, the to-be-allocated task is allocated to a first computing node with residual resources capable of meeting the to-be-allocated task; and after the to-be-allocated task is allocated to the first computing node, at least one monitored resource exists in the first computingnode, and the residual quantity of the monitored resources is greater than or equal to the highest threshold corresponding to the monitored resources. Through the scheme provided by the task allocation method and apparatus, resource fragments in the distributed system can be reduced, so that the resource utilization rate of the distributed system is increased.

The invention discloses a method for achieving mobile phone user behavior portraits based on informationized labels and relates to the technical field of communication. Data association integration of different systems is achieved through client mobile phone numbers from the service level by means of data acquired by telecom operators, the informationized labels are pasted for users, and fragmented user data becomes comprehensible valuable data facilitating computer program processing. The informationized labels are pasted for the users, and the fragmented user data becomes the comprehensible valuable data facilitating computer program processing. Through label features of the user portraits, the pairing relation between market resources and user demands is integrated, and industry application aiming at user behavior data is achieved.

A system and process that involve a digital computer storage comprising block locations having physical block addresses and logical block addresses representing a relational database configuration of cells at logical intersections of sequences of rows and columns that specify a sequence of records and a sequence of attributes. A key attribute is a unique identifier that corresponds to the date / time instance of entry of a selected record into the database system. The arrangement is such that a succession of records corresponds to a succession of date / time instances of entry into the database system. This arrangement facilitates selection of a range of electronic records that is outside a range of electronic records that may be subject to hardware or software malfunction or corruption, facilitates the timed periodic storage and destruction of electronic records pursuant to an archive schedule, and reduces storage fragmentation by which seek time delay is reduced.