Data Partitioning via Coding Schemes for Parallel Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face limitations in processing speed due to hardware constraints and inefficient data storage and retrieval methods, particularly in large-scale data processing environments.
Innovation Solution
The implementation of a parallelized data processing system that divides data into partitions, segments, and slabs, using a desired encoding scheme to sort and store data across multiple computing devices, enabling efficient query processing and response through a distributed network architecture.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored using traditional single-node methods, then storage simplicity is maintained, but processing speed and query response time are limited
Solution Approach 1:
The patent divides data into partitions that are distributed across multiple computing nodes in a cluster. Each partition can be independently processed, enabling parallel query execution and significantly improving processing speed while maintaining manageable system complexity through automated partition management
Solution Approach 2:
The patent introduces a distributed storage architecture that adds a network dimension to traditional single-node storage. By organizing data across multiple nodes with defined communication protocols, the system achieves horizontal scalability and improved performance without proportionally increasing operational complexity
2Productivity
If data is stored in traditional formats, then storage simplicity is maintained, but retrieval efficiency and query performance deteriorate
Solution Approach 1:
The patent segments data into partitions that can be independently retrieved and processed. This segmentation enables selective data retrieval based on query requirements, improving retrieval efficiency by fetching only necessary data partitions rather than scanning entire datasets
Solution Approach 2:
The patent implements pre-processing and indexing of data partitions to optimize query performance. By organizing data in advance according to access patterns and query types, the system reduces retrieval time without requiring complex real-time processing during queries
3Speed
If hardware resources are increased to improve processing speed, then query performance improves, but system cost and resource consumption increase
Solution Approach 1:
The patent distributes computational workload across multiple nodes, allowing each node to process only its local data partitions. This segmentation reduces the computational burden on individual hardware components, enabling faster processing without requiring proportionally more expensive or power-intensive hardware
Solution Approach 2:
The patent enables continuous parallel processing of multiple queries across distributed nodes, maximizing hardware utilization. By keeping all nodes actively engaged in data processing operations, the system achieves high throughput and processing speed without idle resource consumption
Data Source
AI summary
A method for execution by at least one computing entity of a database system, the method includes obtaining a plurality of data partitions of a data set for storage in the database system, where the data set is organized in rows and columns, and the rows correspond to data records and the columns correspond to fields of the data records. The method further includes dividing a first partition of a plurality of data partitions to produce a first number of first raw data segments for storage in the database system, where the first number is based on a first data storage coding scheme. The method further includes dividing a second partition of the plurality of data partitions to produce a second number of second raw data segments for storage in the database system, where the second number is based on a second data storage coding scheme.