Parallel Data Ingestion Architecture for Faster Database Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems are limited by hardware, data storage methods, and co-processing options, leading to inefficiencies in processing speed and execution time.
Innovation Solution
A parallelized data input, store, retrieve, and process sub-system architecture that divides data into partitions, segments, and slabs, using a 4-of-5 encoding scheme, and employs a distributed query and response system with independent processing core resources to optimize query plans and execute queries in parallel.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a traditional database system architecture is used, then the system structure is simple, but the processing speed and execution time are limited by hardware and storage methods
Solution Approach 1:
The database system is divided into multiple independent computing nodes, each capable of autonomous data processing. Data is partitioned across these nodes, enabling parallel processing operations that significantly improve processing speed while distributing system complexity across modular components rather than concentrating it in a single bottlenecked architecture.
2Productivity
If data is stored in a traditional manner, then the storage structure is simple, but the data retrieval and processing efficiency is limited
Solution Approach 1:
Data is segmented into partitions distributed across multiple computing nodes, with each node maintaining local data structures optimized for parallel access. This segmentation enables simultaneous data retrieval and processing operations across different nodes, dramatically improving productivity while the complexity of data management is distributed and handled autonomously by each node.
Solution Approach 2:
The system transitions from traditional single-dimension sequential storage to a multi-dimensional distributed storage architecture where data exists across spatial (different nodes) and logical (partitions, segments) dimensions. This enables parallel access paths and improves processing efficiency by allowing simultaneous operations on different data portions stored across the distributed architecture.
3Speed
If co-processing options are restricted, then the system configuration is simple, but the execution speed is limited
Solution Approach 1:
Processing tasks are segmented and distributed to multiple computing nodes that execute operations in parallel. Each node independently processes assigned data partitions, enabling concurrent execution of multiple operations simultaneously. This segmentation of processing functions improves execution speed by utilizing multiple processing units rather than a single sequential processor.
Data Source
AI summary
A parallelized data input sub-system of a database system includes a first set of loader nodes of pluralities of computing nodes of a plurality of computing device clusters. The first set of loader nodes includes a plurality of memory devices and a plurality of processing modules. The first set of loader node is operable to ingest at least a portion of a dataset (“data”), a set of the memory devices stores the data, and a set of the processing modules determines whether the data is regarding a query. When it is regarding the query, the set of processing modules provides the data to a query and response sub-system. When it is not regarding the query, the set of processing modules determines long term storage parameters, processes the data in accordance with the parameters to produce formatted data, and provides the formatted data to a store and compute sub-system.