695 results about "Data partitioning" patented technology

A system for automating data partitioning in a parallel database includes plural nodes connected in parallel. Each node includes a database server and two databases connected thereto. Each database server includes a query optimizer. Moreover, a partitioning advisor communicates with the database server and the query optimizer. The query optimizer and the partitioning advisor include a program for recommending and evaluating data table partitions that are useful for processing a workload of query statements. The data table partitions are recommended and evaluated without requiring the data tables to be physically repartitioned.

The system and methods described herein may reduce read/write fence latencies and cache pressure related to STM metadata accesses. These techniques may leverage locality information (as reflected by the value of a respective locale guard) associated with each of a plurality of data partitions (locales) in a shared memory to elide various operations in transactional read/write fences when transactions access data in locales owned by their threads. The locale state may be disabled, free, exclusive, or shared. For a given memory access operation of an atomic transaction targeting an object in the shared memory, the system may implement the memory access operation using a contention mediation mechanism selected based on the value of the locale guard associated with the locale in which the target object resides. For example, a traditional read/write fence may be employed in some memory access operations, while other access operations may employ an optimized read/write fence.

The invention discloses a data partitioning method for a distributed parallel database system. The method comprises the following steps of: establishing a fact table and a dimension table according to the constructed distributed parallel database system; inserting records of the dimension table and the fact table on different nodes according to a partitioning rule; copying the records of the dimension table to the nodes of the fact table; and deleting and updating the data. When a data set or data stream is imported or inserted into the distributed database system in a partitioning way, the relation between tables defined by a database schema can be met on each node, particularly the primary key-foreign key restrictive condition, so the data on each node has local completeness of the data. For the query processing on the connection between the tables by using the primary key-foreign key restrictive condition, the data of each node has the local completeness on the query, so dynamic repartitioning of data between the nodes is not needed; and thus the method has the advantages of preventing time-consuming network transmission of the data, shortening the query response time and improving the query efficiency.

Systems and methods are provided for knowledge discovery in spatial data as well as to systems and methods for optimizing recipes used in spatial environments such as may be found in precision agriculture. A spatial data analysis and modeling module is provided which allows users to interactively and flexibly analyze and mine spatial data. The spatial data analysis and modeling module applies spatial data mining algorithms through a number of steps. The data loading and generation module obtains or generates spatial data and allows for basic partitioning. The inspection module provides basic statistical analysis. The preprocessing module smoothes and cleans the data and allows for basic manipulation of the data. The partitioning module provides for more advanced data partitioning. The prediction module applies regression and classification algorithms on the spatial data. The integration module enhances prediction methods by combining and integrating models. The recommendation module provides the user with site-specific recommendations as to how to optimize a recipe for a spatial environment such as a fertilizer recipe for an agricultural field.

System and method for tracking statistics at the subfile level and transparently placing or migrating inactive or less active blocks of data to other storage devices. Embodiments may provide mechanisms to track statistics at the subfile level of files including, but not limited to, database files, and to transparently place or migrate inactive or less active blocks of data of the files from higher-performing, typically more expensive, storage to lower-performing, typically less expensive, storage, while placing or migrating active blocks of data of the files to higher-performing storage, based on the subfile-level statistics rather than on file-level timestamps. In some embodiments, knowledge of file structure (e.g., database file structure), for example knowledge of database partitions with header blocks and data blocks in databases using data partitioning, may be used to separate more active and less active blocks of data of files onto storage with different performance and/or other characteristics.

Version indicators within an existing range can be associated with a data partition in a distributed data store. A partition reconfiguration can be associated with one of multiple partitions in the data store, and a new version indicator that is outside the existing range can be assigned to the reconfigured partition. Additionally, a broadcast message can be sent to multiple nodes, which can include storage nodes and/or client nodes that are configured to communicate with storage nodes to access data in a distributed data store. The broadcast message can include updated location information for data in the data store. In addition, a response message can be sent to a requesting node of the multiple nodes in response to receiving from that node a message that requests updated location information for the data. The response message can include the requested updated location information.

The invention discloses a method for processing data in online business processing, which comprises the following steps: a data partitioning field of a target data table is confirmed according to a business characteristic; and the target data table is divided into a plurality of subtables according to the data partitioning field. The invention uses the method that the core data table in an online business processing system is divided into a plurality of subtables to enable the data amount of the subtables after division to be lower than that of the original core data table. When DDL operation is carried out to the subtables, the occupied space of the temporary table is small, and blocked talks are few, thus the pressure born by the system is greatly reduced, and the maintainability and the manageability of the core data table of the database are improved.

A system and method are disclosed that provide a simple and efficient layered video coding technique using a backward adaptive rate-distortion optimized data partitioning (RD-DP) of DCT coefficients. The video coding system may include an rate-distortion optimized data partitioning encoder and decoder. The RD-DP encoder adapts the partition point block-by-block which greatly improves the coding efficiency of the base layer bit stream without explicit transmission thereby saving the bandwidth significantly. The RD-DP decoder can also find the partition location in backward-fashion from the decoded data.

An embedded electronic device and a firmware updating method thereof are provided. The embedded electronic device includes a first storage unit and a second storage unit with a profile partition, a data partition and a system partition. In the method, a firmware package with a setup information file and a plurality of firmware code files is first downloaded into the first storage unit. The setup information file and the firmware code files are written in the profile partition and the data partition, respectively. The embedded electronic device is re-started, and a boot loader in the system partition is executed to sequentially read each firmware code file from the data partition for writing in the system partition. A write-in progress in the profile partition is updated upon completion of writing of each firmware code file.

Systems and methods are disclosed for partitioning data for storage in a non-volatile memory (“NVM”), such as flash memory. In some embodiments, a priority may be assigned to data being stored, and the data may be logically partitioned based on the priority. For example, a file system may identify a logical address within a first predetermined range for higher priority data and within a second predetermined range for lower priority data, such using a union file system. Using the logical address, a NVM driver can determine the priority of data being stored and can process (e.g., encode) the data based on the priority. The NVM driver can store an identifier in the NVM along with the data, and the identifier can indicate the processing techniques used on the associated data.

A system and method for separating the data needed for inter-layer prediction and data unneeded for inter-layer prediction in the bitstream. For the coded data of a picture, the decoding of the data needed for inter-layer prediction is performed independent of the data not needed for inter-layer prediction, and it is identified whether the data is needed for inter-layer prediction.

A cell-specific dictionary is applied adaptively to adequate cells, where the cell-specific dictionary subsequently optimizes the handling of frequency-partitioned multi-dimensional data. This includes improved data partitioning with super cells or adjusting resulting cells by sub-dividing very large cells and merging multiple small cells, both of which avoid the highly skewed data distribution in cells and improve the query processing. In addition, more efficient encoding is taught within a cell in case the distinct values that actually appear in that cell are much smaller than the size of the column dictionary.

A method and system for generating a decision-tree classifier from a training set of records, independent of the system memory size. The method includes the steps of: generating an attribute list for each attribute of the records, sorting the attribute lists for numeric attributes, and generating a decision tree by repeatedly partitioning the records using the attribute lists. For each node, split points are evaluated to determine the best split test for partitioning the records at the node. Preferably, a gini index and class histograms are used in determining the best splits. The gini index indicates how well a split point separates the records while the class histograms reflect the class distribution of the records at the node. Also, a hash table is built as the attribute list of the split attribute is divided among the child nodes, which is then used for splitting the remaining attribute lists of the node. The method reduces I/O read time by combining the read for partitioning the records at a node with the read required for determining the best split test for the child nodes. Further, it requires writes of the records only at one out of n levels of the decision tree where n>/=2. Finally, a novel data layout on disk minimizes disk seek time. The I/O optimizations work in a general environment for hierarchical data partitioning. They also work in a multi-processor environment. After the generation of the decision tree, any prior art pruning methods may be used for pruning the tree.

The invention discloses a method and a device for data partitioning in a key-value database. The method comprises the following steps of: resolving partitioning number information and partitioning algorithm information from a pre-stored partitioning information basic list; and running a corresponding partitioning algorithm by a partitioning callback function according to the partitioning algorithm information, wherein the corresponding partitioning algorithm determines information of partitioned sub databases of key-value data to be partitioned according to the partitioning number information and a calculation value; and the calculation value is a value relevant to the key-value data to be partitioned. The partitioning number information and the partitioning algorithm information are recorded into the partitioning information basic list; when the key-value data are subjected to operation such as storage and query, the partitioned sub databases are determined for the key-value data according to contents recorded in the partitioning information basic list; and therefore, the partitioning mode can be flexibly changed by modifying the contents in the partitioning information basic list.

A fault tolerant method transforms physically contiguous data in-place on a disk by partitioning the physically contiguous data into an empty region physically adjacent to data regions including a first data region and a last data region, the first and last data regions at opposing ends of the physically contiguous data regions. The physically contiguous data are transformed in an order beginning with the first data region and ending with the last data region. The transforming step perform first locking and reading the first data region, second, transforming the first data region, third, writing and unlocking the transformed first data region to the empty region, and fourth, declaring the first data region as the empty region while declaring the empty region as the first region. The first through fourth steps are repeated for each data region, until completion, to transform the physically contiguous data in-place on the disk.

The invention discloses a method for achieving dynamic partitioning and load balancing of a data-partitioning distributed environment. According to the invention, the method can achieve the local load balancing by basic data unit dynamic combination, namely dynamic partitioning, so as to make the partition granularity meet the system operation requirement. Therefore, the method solves the problemof load balancing time limitation resulting from over-large granularity of a fixed partitioning method. The method provided by the invention has wide applicability by using a partition request queue and a scarce resource queue model to analyze node loads. A local monitor provided in the invention can dynamically adjust the number of partitions after the system achieves load balance so as to achieve self-adaptive adjustment of the partitions inside nodes, thereby improving the resource utilization rate inside the nodes and further improving the work efficiency of the entire system.

This invention generally relates to methods, systems, data structures and computer program code for managing spatial data, in particular very large volumes of data such as map data for a region or country. Thus we describe a method of partitioning a database of spatial data, the database including a plurality of spatial feature objections, the method comprising: reading data from said database grid cell-by-grid cell, a said grid cell comprising a cell of a grid spatially subdividing a region of spatial coverage of said database, each said grid cell including spatial feature objects; and determining a set of spatial partitions for said database, each grid cell being allocated to a said partition, responsive to a number of said spatial feature objects in each said grid cell.

The present invention provides a flexible, dynamic database partition method and system. The method includes the steps of acquiring a data partition rule, where the data partition rule is used to identify a first relationship between a data partition condition and a database partition; establishing a second relationship between the data partition condition and a data partition key based on the data partition rule and a third relationship between the database partition and the data partition key; adding the data partition key to a data item where the data item is stored in the database based on the second relationship between the data partition condition and the data partition key; and storing the data item in the database partition based on the data partition key of the data item.

An embodiment of the invention provides a video data partitioning method and device. The method comprises steps as follows: separating audio data and video image data from video data; performing video feature detection on the audio data and the video image data to acquire one or more candidate partition points; selecting one or more target partition points from the one or more candidate partition points; partitioning the video data according to the one or more target partition points to acquire multiple pieces of video sub-data. With the adoption of the video data partitioning method and device, automatic partitioning of the video data is realized, the manual interference operation is greatly reduced, the partitioning time is greatly shortened, and the labor cost is greatly reduced.

The invention relates to a safety cloud storage method for use in a multi-cloud environment and solves the safety storage problem of cloud data. The method includes data storage and data reading; data storage includes the steps of 1, data partitioning, 2, data deduplication, 3, data encoding, 4, data encrypting, 5, trans-cloud transmission management; data reading includes the steps of 1, metadata reading, 2, reading optimizing, 3, data reading, 4, data decrypting, 5, data decoding and 6, data assembling.

Packets of real-time information are sent with a source rate greater than zero kilobits per second, and a time or path or combined time/path diversity rate initially being zero kilobits per second. This results in a quality of service QoS, optionally measured at the sender or the receiver. When the QoS is on an unacceptable side of a threshold of acceptability, the sender sends diversity packets at an increased rate. Increasing the diversity rate while either reducing or maintaining the overall transmission rate is new. CELP-based multiple-description data partitioning sends the base or important information plus a subset of fixed excitation in one packet and sends the base or important information plus the complementary subset of fixed excitation in another packet. Reconstruction produces acceptable quality when only one of the two packets is received and better quality when both packets are received. Reconstruction provides for single and multiple lost packets.