Non-Relational Database Chunk Auto-Balancing for Parallel Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In big-data storage systems, iterating through entire partitions in non-relational databases is inefficient, consuming computation resources and hindering parallel processing due to the need to fetch individual records or fixed-size batches, which is not intuitive and does not support quick pagination from arbitrary offsets.
Innovation Solution
A method and system for assigning records to chunks within a partition, using a partition key, chunk key, and record key to enable auto-balancing, allowing records to be split into smaller chunks without iterating through the entire partition, thereby optimizing resource usage and enabling parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If iteration through entire partition is used to process data, then data processing can be performed, but computation resources are wasted and parallel processing is hindered
Solution Approach 1:
The patent divides a partition into multiple chunks, each containing a subset of records. Instead of iterating through all records in a partition sequentially, the system can process multiple chunks in parallel, significantly improving data processing efficiency while reducing computation resource consumption per chunk.
2Ease of operation
If fixed-size batch fetching is used for pagination, then pagination can be implemented, but it requires holding database connections and creating records in memory, wasting resources
Solution Approach 1:
The system pre-divides data into chunks before pagination is needed. Each chunk represents a manageable unit that can be processed independently, eliminating the need to hold entire partitions in memory or maintain long-lived database connections during pagination operations.
3Speed
If traditional B-tree storage is used, then sequential access and sorting are efficient, but it is not suitable for distributed big-data storage systems
Solution Approach 1:
The patent segments the partition into multiple chunks that can be distributed across different nodes in a distributed storage system. Each chunk maintains internal ordering similar to B-tree structures, allowing efficient sequential access within chunks while enabling parallel processing and distributed storage capabilities.
4Productivity
If equal sizing of pages is enforced for parallel processing, then load balancing is improved, but it requires complex pagination logic and does not work well with arbitrary offsets
Solution Approach 1:
The system uses dynamic chunk sizing where each chunk contains a reasonable number of records rather than enforcing strict equal sizing. This approach simplifies pagination logic while still enabling effective parallel processing, as chunks can be processed independently without requiring complex load balancing algorithms.
Data Source
AI summary
A method and system are provided for assigning a particular record into a chunk of a partition within a non-relational database system. When a number of records in a particular candidate chunk is greater than a particular threshold number, an application performs an auto-balancing operation to split the particular candidate chunk such that records originally assigned to the particular candidate chunk are divided among the particular candidate chunk and a new chunk. Some of the number of records that were originally part of the particular candidate chunk are assigned to a new chunk and the other remaining ones of the number of records that were originally part of the particular candidate chunk remain assigned to the particular candidate chunk.