Large-Scale Database Architecture with Key-Based Data Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, which hinder efficient execution of complex database functions.
Innovation Solution
A parallelized database system architecture that includes sub-systems for data input, storage, retrieval, and processing, utilizing a network of computing devices to divide data into segments, optimize query plans, and distribute processing across multiple nodes for concurrent execution, thereby enhancing processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data processing is performed using traditional sequential methods, then system complexity is low, but processing speed is slow
Solution Approach 1:
The patent divides the database system into multiple independent processing nodes, each capable of handling specific data segments. The data is partitioned across these nodes, allowing parallel processing of different portions of the database simultaneously, thereby increasing processing speed while maintaining manageable complexity through modular architecture
Solution Approach 2:
The patent transitions from sequential single-threaded processing to parallel multi-threaded processing across multiple nodes. By adding the spatial dimension of distributed computing nodes and the temporal dimension of concurrent execution, the system achieves exponential speedup without proportionally increasing operational complexity
2Productivity
If hardware resources are increased to improve processing speed, then processing capability increases, but hardware cost and complexity increase
Solution Approach 1:
Each processing node in the distributed system is designed to be multi-functional, capable of performing data storage, query processing, and computation tasks. This universality allows the system to achieve high processing capability through software configuration rather than specialized hardware, reducing hardware complexity and cost
Solution Approach 2:
The patent creates multiple copies of the database across different nodes (data replication) and multiple copies of processing logic. This allows the system to handle increased workloads by distributing queries across multiple identical processing units rather than requiring increasingly complex single hardware systems
3Speed
If data is stored in traditional formats, then storage simplicity is maintained, but query processing speed is limited
Solution Approach 1:
The patent implements columnar storage format where data is organized by columns rather than rows, and further partitions data into data slabs based on query patterns. This local optimization allows query processing to access only relevant data segments (specific columns and slabs) rather than scanning entire tables, dramatically improving query speed while keeping storage management systematic
Solution Approach 2:
The system pre-processes and organizes data into optimized storage structures (columnar format, data slabs, inverted indexes) during data loading and maintenance operations. This preliminary organization enables much faster query execution without requiring complex real-time data reorganization during query processing
Data Source
AI summary
A data input sub-system of a large scale application specific computing system receives a data set that includes a plurality of records, each with a plurality of data fields, and divides the data set into a plurality of data segments. The data input sub-system further restructures records of data segments based on a key field of the plurality of data fields to produce restructured data segments and generates storage instructions for storing the restructured data segments. A data storage and processing sub-system of the computing system interprets the storage instructions to determine resources to engage and stores the restructured data segments using engaged resources. A query and results sub-system of the computing system generates an initial query plan based on a data processing request, optimizes the initial query plan to produce an optimized query plan, and sends the optimized query plan to the data storage and processing sub-system for execution.